[
https://issues.apache.org/jira/browse/TEZ-3770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16293274#comment-16293274
]
Eric Wohlstadter commented on TEZ-3770:
---------------------------------------
Here is some data for consideration.
Running TPC-DS Q64 10TB. At least with my setup:
* DagAwareYarnTaskScheduler, job completes in 17 mins.
* default scheduler, job continues to fail:
{code}
2017-12-15 14:53:58,441 [INFO] [AMRM Callback Handler Thread]
|rm.YarnTaskSchedulerService|: Trying to service 7 out of total 62 pending
requests at pri: 440 by preempting from 75 running tasks at priority: 1160
{code}
and things just continue deteriorating from there.
Could this be explained by the use of the DagAwareYarnTaskScheduler?
> DAG-aware YARN task scheduler
> -----------------------------
>
> Key: TEZ-3770
> URL: https://issues.apache.org/jira/browse/TEZ-3770
> Project: Apache Tez
> Issue Type: New Feature
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: TEZ-3770.001.patch
>
>
> There are cases where priority alone does not convey the relationship between
> tasks, and this can cause problems when scheduling or preempting tasks. If
> the YARN task scheduler was aware of the relationship between tasks then it
> could make smarter decisions when trying to assign tasks to containers or
> preempt running tasks to schedule pending tasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
