[
https://issues.apache.org/jira/browse/TEZ-800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16342498#comment-16342498
]
Hitesh Sharma commented on TEZ-800:
-----------------------------------
I have a case where two 1:1 edges are coming into a vertex. One of the source
vertex has parallelism specified (say it is set to 'n') while other doesn't (it
is set to -1). Thus DAG::checkAndInferOneToOneParallelism infers the
parallelism of the vertex to be 'n'.
The code then below fails to verify the DAG.
{code:java}
// check for inconsistency and errors
for (Edge e : edges) {
Vertex inputVertex = e.getInputVertex();
Vertex outputVertex = e.getOutputVertex();
if (e.getEdgeProperty().getDataMovementType() == DataMovementType.ONE_TO_ONE)
{
if (inputVertex.getParallelism() != outputVertex.getParallelism()) {
// both should be equal or equal to -1.
if (outputVertex.getParallelism() != -1) {
throw new TezUncheckedException(
"1-1 Edge. Destination vertex parallelism must match source vertex.
"
+ "Vertex: " + inputVertex.getName() + " does not match vertex: "
+ outputVertex.getName());
}
}
}
}
{code}
It seems to me that we should be checking for the sourceVertex parallelism to
be -1 as well over {color:#59afe1}here{color}.
Will that be okay or some other feature may break?
> One-one edge with parallelism -1 fails if source vertex parallelism is not -1
> as well
> -------------------------------------------------------------------------------------
>
> Key: TEZ-800
> URL: https://issues.apache.org/jira/browse/TEZ-800
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.3.0
> Reporter: Rohini Palaniswamy
> Assignee: Bikas Saha
> Priority: Major
> Fix For: 0.5.0
>
> Attachments: TEZ-800.1.patch, TEZ-800.2.patch
>
>
> // no input initializers. At this moment, only other case is 1-1 edge
> // with uninitialized sources
> boolean hasOneToOneUninitedSource = false;
> for (Map.Entry<Vertex, Edge> entry :
> vertex.sourceVertices.entrySet()) {
> if (entry.getValue().getEdgeProperty().getDataMovementType() ==
> DataMovementType.ONE_TO_ONE) {
> if (entry.getKey().getTotalTasks() == -1) {
> hasOneToOneUninitedSource = true;
> break;
> }
> }
> }
> This checks for the source vertex which has the 1-1 edge to also have -1
> parallelism.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
