abstractdog commented on PR #301: URL: https://github.com/apache/tez/pull/301#issuecomment-1623121365
finally a working version for this patch, unit tests are included, tested on cluster as show below: 1. ran a dag with resources available for 2 tasks containers (after the first 2 allocation/launch, the rest 6 attempts were all reuse as expected) ``` INFO : org.apache.tez.common.counters.DAGCounter: INFO : NUM_SUCCEEDED_TASKS: 8 INFO : TOTAL_LAUNCHED_TASKS: 8 INFO : OTHER_LOCAL_TASKS: 5 INFO : AM_CPU_MILLISECONDS: 5130 INFO : AM_GC_TIME_MILLIS: 141 INFO : INITIAL_HELD_CONTAINERS: 0 INFO : TOTAL_CONTAINERS_USED: 2 INFO : TOTAL_CONTAINER_ALLOCATION_COUNT: 2 INFO : TOTAL_CONTAINER_LAUNCH_COUNT: 2 INFO : TOTAL_CONTAINER_REUSE_COUNT: 6 ``` 2. ran the dag quickly again: the same amount of tasks, but already had 2 containers, so all 8 attempts were running on reused containers (on the same 2 containers as shown in TOTAL_CONTAINERS_USED) ``` INFO : org.apache.tez.common.counters.DAGCounter: INFO : NUM_SUCCEEDED_TASKS: 8 INFO : TOTAL_LAUNCHED_TASKS: 8 INFO : OTHER_LOCAL_TASKS: 5 INFO : AM_CPU_MILLISECONDS: 1130 INFO : AM_GC_TIME_MILLIS: 83 INFO : INITIAL_HELD_CONTAINERS: 2 INFO : TOTAL_CONTAINERS_USED: 2 INFO : TOTAL_CONTAINER_REUSE_COUNT: 8 ``` 3. killed another yarn application to make resources, this is the same as 1) just with 3 containers: ``` INFO : org.apache.tez.common.counters.DAGCounter: INFO : NUM_SUCCEEDED_TASKS: 8 INFO : TOTAL_LAUNCHED_TASKS: 8 INFO : OTHER_LOCAL_TASKS: 5 INFO : AM_CPU_MILLISECONDS: 5790 INFO : AM_GC_TIME_MILLIS: 184 INFO : INITIAL_HELD_CONTAINERS: 0 INFO : TOTAL_CONTAINERS_USED: 3 INFO : TOTAL_CONTAINER_ALLOCATION_COUNT: 3 INFO : TOTAL_CONTAINER_LAUNCH_COUNT: 3 INFO : TOTAL_CONTAINER_REUSE_COUNT: 5 ``` 4. ran dag again, the same as 2) but with 3 containers: ``` INFO : org.apache.tez.common.counters.DAGCounter: INFO : NUM_SUCCEEDED_TASKS: 8 INFO : TOTAL_LAUNCHED_TASKS: 8 INFO : OTHER_LOCAL_TASKS: 5 INFO : AM_CPU_MILLISECONDS: 1360 INFO : AM_GC_TIME_MILLIS: 77 INFO : INITIAL_HELD_CONTAINERS: 3 INFO : TOTAL_CONTAINERS_USED: 3 INFO : TOTAL_CONTAINER_REUSE_COUNT: 8 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
