[
https://issues.apache.org/jira/browse/TEZ-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated TEZ-3274:
-----------------------------
Attachment: TEZ-3274.002.patch
Adding a new patch that moves the functionality out of
{{VertexManagerPluginContext}} and into {{VertexManagerPlugin}}. Since
{{VertexManagerPlugin}} is an abstract class, I am able to write default
implementations for the slow start methods, while allowing the underlying
plugins that extend it to override these methods if they choose. It makes
{{VertexManagerPlugin}} a little bit more bloated, but I'm not sure how else to
do it since an abstract class can only extend a single abstract class IIUC.
I moved the configs into TezConfiguration since the slow start functionality is
now in {{VertexManagerPlugin}}. Additionally, I changed the
{{VertexManagerPluginContext}} API back so that everything goes through
{{scheduleTasks()}}. The new approach has the plugins calling
{{tasksToScheduleFilter()}} on the list of tasks that it wants to schedule, and
{{tasksToScheduleFilter()}} prunes that list depending on the different
modifiers that are implemented. Currently the only modifier is slow start.
However, in the future many more could be added easily.
[~sseth], any thoughts on this new approach? Hopefully this is more in line
with what you were envisioning.
> Vertex with MRInput and shuffle input does not respect slow start
> -----------------------------------------------------------------
>
> Key: TEZ-3274
> URL: https://issues.apache.org/jira/browse/TEZ-3274
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jonathan Eagles
> Assignee: Eric Badger
> Attachments: TEZ-3274.001.patch, TEZ-3274.002.patch
>
>
> Vertices with shuffle input and MRInput choose RootInputVertexManager (and
> not ShuffleVertexManager) and start containers and tasks immediately. In this
> scenario, resources can be wasted since they do not respect
> tez.shuffle-vertex-manager.min-src-fraction
> tez.shuffle-vertex-manager.max-src-fraction.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
