----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20149/#review39979 -----------------------------------------------------------
Ship it! +1 This patch changes the resource allocation policy for leaf stages. In leaf stage, the resource allocation will be based on disk resource. In non-leaf stage, it will be based on memory resource. I verified unit tests, and just tested it on a real Tajo cluster. It works well! - Hyunsik Choi On April 9, 2014, 4:22 p.m., Jinho Kim wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/20149/ > ----------------------------------------------------------- > > (Updated April 9, 2014, 4:22 p.m.) > > > Review request for Tajo. > > > Bugs: TAJO-743 > https://issues.apache.org/jira/browse/TAJO-743 > > > Repository: tajo > > > Description > ------- > > Currently, resource allocation is calculated by memory base. If a machine > have a large memory, in default settings, heavy disk IO per disk is usually > caused by high task concurrency. However, it is likely to seem to be > problematic. > > When i tested the leaf task scan by 2(concurrency of SATA disk), the > performance was better. if you have SAS Storage or SSD, you can increase the > disk concurrency. This patch changes the default resource allocation policy > to use disk resource. > > The following configs have been available so far: > * tajo.worker.resource.disks - available disk resource of each worker > * tajo.task.disk-slot.default - how many disk resource is consumed per task > > Below config is newly introduced in this patch > * tajo.worker.resource.dfs-dir-aware - it can be true/false. If it is true, > each worker uses the number of HDFS datanodes > > > Diffs > ----- > > tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java > db96eb829a567e34f07c168e5d75e01545a680ef > > tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java > bcf10ddb4f41bfe68816d5659680a0a00b4f8a77 > > tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/WorkerHeartbeatService.java > 3843b8f303b6138fa979314bc9002755e0f2082d > tajo-storage/src/main/java/org/apache/tajo/storage/v2/DiskUtil.java > bb90c3917bc577f701ec05b44ae54584a1bb0d1a > > Diff: https://reviews.apache.org/r/20149/diff/ > > > Testing > ------- > > mvn clean install > > > Thanks, > > Jinho Kim > >
