[
https://issues.apache.org/jira/browse/TEZ-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14203590#comment-14203590
]
Siddharth Seth edited comment on TEZ-1750 at 11/8/14 8:52 PM:
--------------------------------------------------------------
Updated with expert level setting.
There's comments in their on some future enhancements - considering cluster
capacity to allow scheduling downstream even if upstream is not scheduled,
generic slow start.
Not enabling this by default in 0.5.3, because I think it's not a good change
to have in place on a minor version. On 0.6.0, based on how this performs, it
could be enabled by default.
was (Author: sseth):
Updated with experimental.
There's comments in their on some future enhancements - considering cluster
capacity to allow scheduling downstream even if upstream is not scheduled,
generic slow start.
Not enabling this by default in 0.5.3, because I think it's not a good change
to have in place on a minor version. On 0.6.0, based on how this performs, it
could be enabled by default.
> Add a DAGScheduler which schedules tasks only when sources have been scheduled
> ------------------------------------------------------------------------------
>
> Key: TEZ-1750
> URL: https://issues.apache.org/jira/browse/TEZ-1750
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: TEZ-1750.1.txt, TEZ-1750.2.txt, TEZ-1750.3.txt
>
>
> Splitting out the patch on TEZ-1522 into a separate jira.
> There's several scenarios in which we end up scheduling downstream tasks
> before their sources have been scheduled - and then get into a situation
> where the sources are starved. Currently, anywhere a ShuffleVertexManager is
> used can cause such behaviour - since it starts scheduling it's tasks after a
> certain number of sources are complete, but subsequen non-shuffle
> VertexManagers will scheduled immediately.
> Disabling slow-start is one option to achieve this (or setting slow start on
> all vertices), but it doesn't work for the situation where dynamic reducer
> parallelism kicks in - since it has to wait for source tasks to complete.
> The intent here is to add a DAGScheduler, which affectively negates the slow
> start, and in case of dynamic parallelism determination, waits for upstream
> tasks to be scheduled before scheduling downstream tasks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
