[
https://issues.apache.org/jira/browse/PIG-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-5444:
------------------------------
Attaching the patch which basically does two things.
> [~rohini] , should I add a predecessor check to prevent this merge ?
>
(1) Added a check to make sure there are no overlap on predecessors among all
the "tentativeSuccessors".
(2) Introduce a new walker which considers the dependency order of nodes'
successors.
For the example from this jira,
{noformat}
B5(input)
/ \
---------6 7
/ /
/ A1(input) /
\ / \ /
2 3------
\ /
4(shuffle->out)
{noformat}
2 and 3 are Joins. Now, with the previous ReverseDependencyOrderWalker, we do
not consider the dependency order of nodes' successors, therefore visit order
of A1 and B5 are indeterministic. With the new
ReverseSuccessorsDependencyOrderWalker, A1 will always get visited before B5.
> TestFRJoin.testFRJoinOut7 and testFRJoinOut8 failing with Edge already
> defined error on Tez
> -------------------------------------------------------------------------------------------
>
> Key: PIG-5444
> URL: https://issues.apache.org/jira/browse/PIG-5444
> Project: Pig
> Issue Type: Bug
> Components: tez
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Major
> Attachments: pig-5444-v01.patch
>
>
> With Tez, when testing individual tests (TestFRJoin.testFRJoinOut7 and
> testFRJoinOut8) separately, they pass the tests. But when entire TestFRJoin
> is run, these two tests on Tez are failing with
> {noformat}
> Unable to open iterator for alias E
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
> open iterator for alias E
> at org.apache.pig.PigServer.openIterator(PigServer.java:1024)
> at org.apache.pig.test.TestFRJoin.testFRJoinOut7(TestFRJoin.java:409)
> Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias E
> at org.apache.pig.PigServer.storeEx(PigServer.java:1127)
> at org.apache.pig.PigServer.store(PigServer.java:1086)
> at org.apache.pig.PigServer.openIterator(PigServer.java:999)
> Caused by:
> org.apache.pig.backend.hadoop.executionengine.JobCreationException: ERROR
> 2017: Internal error creating job configuration.
> at
> org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.getJob(TezJobCompiler.java:153)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.compile(TezJobCompiler.java:81)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:200)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1479)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464)
> at org.apache.pig.PigServer.storeEx(PigServer.java:1123)
> Caused by: java.lang.IllegalArgumentException: Edge [scope-632 :
> org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor] ->
> [scope-628 :
> org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor] ({
> BROADCAST : org.apache.tez.runtime.library.input.UnorderedKVInput >>
> PERSISTED >> org.apache.tez.runtime.library.output.UnorderedKVOutput >>
> NullEdgeManager }) already defined!
> at org.apache.tez.dag.api.DAG.addEdge(DAG.java:296)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder.visitTezOp(TezDagBuilder.java:410)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:265)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:56)
> at
> org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:87)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.buildDAG(TezJobCompiler.java:69)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.getJob(TezJobCompiler.java:120)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
