Hi all,
while reading the log from the join example (2 source vertexes and a
sink vertex) I noticed the following:
2014-09-26 21:20:27,021 INFO [TaskSchedulerEventHandlerThread]
org.apache.tez.dag.app.rm.YarnTaskSchedulerService: Allocation *request*
for task: attempt_1411734050933_0003_1_00_000000_0 with request:
Capability[<memory:1024, vCores:1>]*Priority[2]* host: node02 rack: null
...
2014-09-26 21:20:28,237 INFO [DelayedContainerManager]
org.apache.tez.dag.app.rm.YarnTaskSchedulerService: Assigning container
to task, container=*Container*: [ContainerId:
container_1411734050933_0003_01_000002, NodeId: node02:34192,
NodeHttpAddress: node02:8042, Resource: <memory:1024, vCores:1>,
*Priority: 2*, Token: Token { kind: ContainerToken, service:
192.168.56.102:34192 }, ],
task=attempt_1411734050933_0003_1_00_000000_0, containerHost=node02,
localityMatchType=NodeLocal, matchedLocation=node02,
honorLocalityFlags=true, reusedContainer=false, delayedContainers=1,
containerResourceMemory=1024, containerResourceVCores=1
And something similar for the other "parent" vertex, nothing strange
here. But this is about the "joiner" vertex:
2014-09-26 21:21:24,292 INFO [TaskSchedulerEventHandlerThread]
org.apache.tez.dag.app.rm.YarnTaskSchedulerService: Allocation *request*
for task: attempt_1411734050933_0003_1_02_000000_0 with request:
Capability[<memory:1024, vCores:1>]*Priority[4]* host: null rack: null
...
2014-09-26 21:21:24,318 INFO [DelayedContainerManager]
org.apache.tez.dag.app.rm.YarnTaskSchedulerService: Assigning container
to task, container=*Container*: [ContainerId:
container_1411734050933_0003_01_000003, NodeId: node02:34192,
NodeHttpAddress: node02:8042, Resource: <memory:1024, vCores:1>,
*Priority: 2*, Token: Token { kind: ContainerToken, service:
192.168.56.102:34192 }, ],
task=attempt_1411734050933_0003_1_02_000000_0, containerHost=node02,
localityMatchType=NodeLocal, matchedLocation=node02,
honorLocalityFlags=true, reusedContainer=true, delayedContainers=2,
containerResourceMemory=1024, containerResourceVCores=1
Here the priority of the obtained container is still 2, but I was
expecting to find the same priority of the request (4). So what is the
priority of the obtained container, since it seems to be 2 regardless of
the request? Is it used by Tez? How?
Another question I would like to ask is: I see the priority is
calculated as /(vertexDistanceFromRoot + 1) * 2/, where
vertexDistanceFromRoot is (I think) the distance from the vertex which
got its input from a file, or at least not from another vertex. But I
haven't been able to understand how this value is set, especially in
case two (or more) branches converging in a common vertex X have not the
same "depth"... in other words: what happens if X has two parents, one
with priority 4 and one 6? Which will be its priority?
Thanks in advance
Fabio
- Vertex/task priority and container request Fabio
-