Hi Burak, It's a great question. Tajo controls the size of input data for each task according to given parameters. Let me explain it in more detail. There are two cases where Tajo decides data size that will be processed.
The first case is when the dataset resides in HDFS. In this case, like MapReduce, QueryMaster creates a number of tasks and assigns them to TaskRunners. Each TaskRunner processes one task at a time. Naturally, TaskRunner deals with the input data corresponding to one HDFS block at a time. Basically, Tajo allocates 2~3 GB to each TaskRunner, whereas the HDFS block size is 64 ~ 256MB. In this case, most physical operators work with main memory. The second case is when tasks consume the hash or range repartitioned data. During query processing, Tajo collects the statistic information from tasks and aggregates them. According to this statistic information (including intermediate data size and the number of partition keys) and the type of repartition, tajo.master.Repartitioner determines the desired input data size for each task. For example, given the hash repartition for join, Repartitioner determines that 64MB is the desired input data size for each task. In the current implementation, the predefined desired sizes are given in TajoConf. This approach is somewhat straightforward. There is still room for improvement. Does 'swap operation' mean choosing an alternative physical operator? In the current implementation, PhysicalPlannerImpl determines physical operators to be executed according to the input data size. For example, for join operator, if the input data size is larger than a given parameter, PhysicalPlannerImpl chooses the external sort algorithm and the merge join algorithm. Otherwise, PhysicalPlannerImpl chooses only in-memory hash join. Besides, I would like explain the overall planning steps. TajoMaster transforms a SQL statement to an optimized logical plan. Then, TajoMaster breaks the logical plan to multiple logical plans, each of which become a subquery (a execution block in PPT). The global plan represents a DAG of these subqueries. The subquery is a base unit to be distributed across a number of nodes. Please note that each subquery contains a logical plan. Therefore, PhysicalPlannerImpl in TaskRunner determines most of physical operators to be executed. If you have any question, feel free to ask me anything! Thanks, Hyunsik On Mar 27, 2013, at 7:26 PM, burakkk <[email protected]> wrote: > Hello, > How does Tajo decide the data size that will be processed fit in the memory > or when does it use the swap operation? > > Thanks > Best regards... > > -- > > *BURAK ISIKLI** *| *http://burakisikli.wordpress.com* > * > *
