[
https://issues.apache.org/jira/browse/TEZ-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001681#comment-16001681
]
Eric Badger commented on TEZ-3274:
----------------------------------
[~sseth], in our offline discussion last Thursday, we agreed to remove the
ImmediateStartVertexManager dependency from the RootInputVertexManager and then
implement a new SlowStartVertexManager that the RootInputVertexManager would
extend from. This would solve the MRInput and BROADCAST input problem since
that case uses the RootInputVertexManager. In the interest of not creating
redundant code, I think that it would be a good idea to take the relevant slow
start code out of the ShuffleVertexManager/ShuffleVertexManagerBase and have
ShuffleVertexManagerBase also extend this new SlowStartVertexManager. However,
this would then cause some different functionality for the
ShuffleVertexManagerBase, since the ShuffleVertexManagerBase currently only
uses SCATTER_GATHER edges when computing slow start and the new
SlowStartVertexManager would need to handle both SCATTER_GATHER and BROADCAST.
So I have 2 questions:
1. Should I remove the relevant slow start code from
ShuffleVertexManager/ShuffleVertexManagerBase and have ShuffleVertexManagerBase
extend SlowStartVertexManager
2. If yes, should I compute slow start separately for both SCATTER_GATHER and
BROADCAST or should I just treat all edge types as the same in the computation?
cc [~jeagles]
> Vertex with MRInput and broadcast input does not respect slow start
> -------------------------------------------------------------------
>
> Key: TEZ-3274
> URL: https://issues.apache.org/jira/browse/TEZ-3274
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jonathan Eagles
> Assignee: Eric Badger
> Attachments: TEZ-3274.001.patch, TEZ-3274.002.patch,
> TEZ-3274.003.patch
>
>
> Vertices with shuffle input and MRInput choose RootInputVertexManager (and
> not ShuffleVertexManager) and start containers and tasks immediately. In this
> scenario, resources can be wasted since they do not respect
> tez.shuffle-vertex-manager.min-src-fraction
> tez.shuffle-vertex-manager.max-src-fraction.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
