[
https://issues.apache.org/jira/browse/TEZ-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated TEZ-3270:
-------------------------
Description: One of the scheduling factors is data locality. For a given
completed task A, it is better to have its depending tasks in B run on the same
host/container to reduce the network data transfer between the two. In
addition, it might be better to pick larger partition task over smaller
partition task. For example, in the above fair routing diagram, after task A1
has completed, task B1 and/or task B2 can be scheduled on the same
host/container as task A1; and B2 has higher priority than B1 given P2 is
larger than P1. (was: One of the scheduling factors is data locality. For a
given completed task A, it is better to have its depending tasks in B run on
the same host/container to reduce the network data transfer between the two. In
addition, it might be better to pick larger partition task over smaller
partition task. For example, in the above fair routing diagram, after task A1
has completed, task B1
and/or task B2 can be scheduled on the same host/container as task A1; and B2
has higher priority than B1 given P2 is larger than P1.)
> Data locality based scheduling policy in fair routing
> -----------------------------------------------------
>
> Key: TEZ-3270
> URL: https://issues.apache.org/jira/browse/TEZ-3270
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Ming Ma
>
> One of the scheduling factors is data locality. For a given completed task A,
> it is better to have its depending tasks in B run on the same host/container
> to reduce the network data transfer between the two. In addition, it might be
> better to pick larger partition task over smaller partition task. For
> example, in the above fair routing diagram, after task A1 has completed, task
> B1 and/or task B2 can be scheduled on the same host/container as task A1; and
> B2 has higher priority than B1 given P2 is larger than P1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
