[
https://issues.apache.org/jira/browse/TEZ-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14300772#comment-14300772
]
Bikas Saha commented on TEZ-2020:
---------------------------------
[~rajesh.balamohan] Please review as a follow up to TEZ-2011.
The patch builds on top of TEZ-2011 to
1) Have input ready manager inform the vertex that it plan to reconfigure the
vertex (similar to shufflevertexmanager). This prevents the early configured
signal from being sent by the vertex.
2) When the input ready manager configures itself, then it checks if the final
1-1 edge source task counts match the vertex task count. If they dont then it
updates the vertex parallelism to match that. This takes care of both cases
when current vertex parallelism is -1 (not set) or >0 (when set to some value
but needs to update to new value - e.g. auto-reduce parallelism on the parent).
This also removes the special case handling of this action of the 1-1 edge from
the VertexImpl code. That dead code is now removed.
3) Existing tests for these scenarios pass and tests have been updated to
verify new flow.
> For 1-1 edge vertex configured event may be sent incorrectly
> ------------------------------------------------------------
>
> Key: TEZ-2020
> URL: https://issues.apache.org/jira/browse/TEZ-2020
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: TEZ-2020.1.patch
>
>
> For a fully configured 1-1 vertex configured event will be sent after init
> because the vertex manager does not signal intent to reconfigure. However a
> change in parallelism upstream on the 1-1 edge can cause this vertex's
> parallelism to be updated accordingly. This breaks the guarantee that
> configured should be sent after vertex is fully configured. This is because
> update of parallelism on 1-1 edge has until now been hard-coded in the Vertex
> code. This can be removed now with the InputReadyVertexManager handling the
> 1-1 edge and the addition of vertex state updates. Not only will this fix the
> issue but also remove the special handling for 1-1 edges in the AM which
> should not be there for cleanliness.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
