[
https://issues.apache.org/jira/browse/TEZ-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14144089#comment-14144089
]
Bikas Saha commented on TEZ-1612:
---------------------------------
The problem thats happening here is that the custom edge set by the
shufflevertexmanager hard codes its value of numSourceTasks instead of using
the actual value. So if that value is picked before the source changed
parallelism then it will use that parallelism instead of the new one. The
custom edge should use the actual value from context(). This fix will not fix
the general case of downstream tasks starting before upstream tasks but is
definitely a fix needed by itself and may solve a bunch of cases for Pig unit
tests.
> Pig on tez unit test intermittent hang
> --------------------------------------
>
> Key: TEZ-1612
> URL: https://issues.apache.org/jira/browse/TEZ-1612
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.5.0
> Reporter: Daniel Dai
> Assignee: Bikas Saha
> Attachments: DAG1.png, syslog_dag_1411413615885_0001_1
>
>
> Several Pig unit tests hang intermittently. For example,
> TestNewPlanImplicitSplit.testImplicitSplitInCoGroup, which is a DAG of 4
> nodes:
> !DAG1.png!
> It uses auto-parallelism, vertex 106 change parallelism from 2->1, and vertex
> 107 from 21->1.
> Log attached.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
