[
https://issues.apache.org/jira/browse/TEZ-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306585#comment-14306585
]
Bikas Saha commented on TEZ-2020:
---------------------------------
Yes. User does need to be aware since vertex managers are user code applying
user logic. For the same reason, the framework cannot detect or disable this by
itself. For the framework, ShuffleVertexManager is a black box that is applied
to a vertex. What is does and whether auto-reduce is enabled or not, is
something the framework does not know. Even the edge constraint of 1-1 edge is
user logic in theory (and framework code to support that is being removed in
this patch - VertexOneToOneSplitTransition). This constraint is being checked
by the InputReadyVertexManager thought it should also be checked by the
OneToOneEdgeManager. Will open a jira for that.
Thanks for the reviews and additional testing.
> For 1-1 edge vertex configured event may be sent incorrectly
> ------------------------------------------------------------
>
> Key: TEZ-2020
> URL: https://issues.apache.org/jira/browse/TEZ-2020
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: TEZ-2020.1.patch, synthetic_job.png
>
>
> For a fully configured 1-1 vertex configured event will be sent after init
> because the vertex manager does not signal intent to reconfigure. However a
> change in parallelism upstream on the 1-1 edge can cause this vertex's
> parallelism to be updated accordingly. This breaks the guarantee that
> configured should be sent after vertex is fully configured. This is because
> update of parallelism on 1-1 edge has until now been hard-coded in the Vertex
> code. This can be removed now with the InputReadyVertexManager handling the
> 1-1 edge and the addition of vertex state updates. Not only will this fix the
> issue but also remove the special handling for 1-1 edges in the AM which
> should not be there for cleanliness.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
