[
https://issues.apache.org/jira/browse/TEZ-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yingda Chen updated TEZ-3999:
-----------------------------
External issue ID: TEZ-3997
> Extend VertexManagerPlugin interface to allow for relevant events notification
> ------------------------------------------------------------------------------
>
> Key: TEZ-3999
> URL: https://issues.apache.org/jira/browse/TEZ-3999
> Project: Apache Tez
> Issue Type: Task
> Reporter: Yingda Chen
> Priority: Major
>
> This is an umbrella task of TEZ-3997
> **For concurrent connection, the downstream and upstream vertices would be
> running concurrently, and in some cases, they would be scheduled at the same
> time as well, such as (sub-graph) gang scheduling. However, *this is not
> always true*. In the example in Fig. 1, tasks in PS vertex should be running
> before tasks in W vertex should be scheduled. Since otherwise if the resource
> requests for PS cannot be fulfilled first, W will be spinning in vain. In
> other examples, as long as part of tasks in upstream vertex are running, we
> can start scheduling downstream tasks.
>
> In other words, if we put this into the context of existing
> interface/implementation of VertexMangerPlugin, we can see strong duality of
> “OnSourceTaskRunning” for concurrent connection vs the
> “OnSourceTaskCompleted” for (existing) sequential connection. Therefore, we
> propose an addition of “_onConcurrentSourceTaskRunning(TaskAttemptIdentifer
> attempt)_” interface to the VertexManager Plugin, with default implementation
> being not supported.
> This change will also include the logic to add source task running event and
> to send such events to downstream vertices. To reduce unnecessary event
> traffic, we will limit the sending of such events to CONCURRENT edge, and
> when the ConcurrentSchedulingType is specified to be SOURCE_TASK_STARTED .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
