[
https://issues.apache.org/jira/browse/TEZ-3770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065756#comment-16065756
]
Bikas Saha commented on TEZ-3770:
---------------------------------
Just clarifying that the original scheduler was not made dag aware by design.
It was an attempt to prevent leaky features where code changed across the
scheduler and the dag state machine. Like it happened in MR code where logic
was spread all over. The DAG core logic and VertexManager user logic could
determine the dependencies and priorities of tasks and the scheduler would
allocate resources based on priority. So other schedulers could be easily
written since they dont need to understand complex relationships.
However not all of those design assumptions have been validated since we dont
have many schedulers written :P
> DAG-aware YARN task scheduler
> -----------------------------
>
> Key: TEZ-3770
> URL: https://issues.apache.org/jira/browse/TEZ-3770
> Project: Apache Tez
> Issue Type: New Feature
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: TEZ-3770.001.patch
>
>
> There are cases where priority alone does not convey the relationship between
> tasks, and this can cause problems when scheduling or preempting tasks. If
> the YARN task scheduler was aware of the relationship between tasks then it
> could make smarter decisions when trying to assign tasks to containers or
> preempt running tasks to schedule pending tasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
