[ 
https://issues.apache.org/jira/browse/PIG-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744494#comment-16744494
 ] 

Koji Noguchi edited comment on PIG-5375 at 1/16/19 10:08 PM:
-------------------------------------------------------------

For my reference (and maybe for the reviewer) attaching how the plan evolved 
with vertexgroup by the UnionOptimizer before my patch.  (using the script from 
the description)

https://issues.apache.org/jira/secure/attachment/12955153/pig-5375.png

Note the bottom of the vertex group output
{panel}
Tez vertex group scope-87 <- [{color:#FF0000}scope-56{color}, scope-53, 
{color:#FF0000}scope-56{color}, scope-54, scope-55] -> scope-81
{panel}
where the scope-56 no longer exist. What my patch does is, instead of replacing 
one scope-56 with scope-54&scope-55, it replaces all three scope-56 with 
scope-54&scope-55.
  

 


was (Author: knoguchi):
For my reference (and maybe for the reviewer) attaching how the plan evolved 
with vertexgroup by the UnionOptimizer before my patch.  (using the script from 
the description)

!hadooppf-5375.png|thumbnail! 

Note the bottom of the vertex group output
{panel}
Tez vertex group scope-87 <- [{color:#FF0000}scope-56{color}, scope-53, 
{color:#FF0000}scope-56{color}, scope-54, scope-55] -> scope-81
{panel}
where the scope-56 no longer exist. What my patch does is, instead of replacing 
one scope-56 with scope-54&scope-55, it replaces all three scope-56 with 
scope-54&scope-55.
  

 

> NullPointerException for multi-level self unions with Tez UnionOptimizer
> ------------------------------------------------------------------------
>
>                 Key: PIG-5375
>                 URL: https://issues.apache.org/jira/browse/PIG-5375
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Koji Noguchi
>            Assignee: Koji Noguchi
>            Priority: Major
>         Attachments: pig-5375-v1.patch, pig-5375.png
>
>
> {code}
> A = load 'input.txt' as (a0:int, a1: chararray, a2:int);
> B = load 'input.txt' as (a0:int, a1: chararray, a2:int);
> C = load 'input.txt' as (a0:int, a1: chararray, a2:int);
> A_and_B = UNION A, B;
> SPLIT A_and_B INTO A_and_B2 IF a0 > 10, A_and_B3 OTHERWISE;
> A_and_B_and_C = UNION ONSCHEMA C, A_and_B;
> X = UNION ONSCHEMA A_and_B_and_C, A_and_B2, A_and_B3;
> X2 = GROUP X ALL ;
> dump X2;
> {code}
> This fails _on Tez_ with 
> {noformat}
> Pig Stack Trace
> ---------------
> ERROR 1002: Unable to store alias X2
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to 
> open iterator for alias X2
>         at org.apache.pig.PigServer.openIterator(PigServer.java:1024)
>         at 
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:790)
>         at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:383)
>         at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:231)
>         at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206)
>         at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
>         at org.apache.pig.Main.run(Main.java:630)
>         at org.apache.pig.Main.main(Main.java:175)
> Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias X2
>         at org.apache.pig.PigServer.storeEx(PigServer.java:1127)
>         at org.apache.pig.PigServer.store(PigServer.java:1086)
>         at org.apache.pig.PigServer.openIterator(PigServer.java:999)
>         ... 7 more
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: 
> java.lang.NullPointerException
>         at 
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:296)
>         at org.apache.pig.PigServer.launchPlan(PigServer.java:1479)
>         at 
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464)
>         at org.apache.pig.PigServer.storeEx(PigServer.java:1123)
>         ... 9 more
> Caused by: java.lang.NullPointerException
>         at 
> org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.patchPackage(TezPOPackageAnnotator.java:97)
>         at 
> org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.handlePackage(TezPOPackageAnnotator.java:78)
>         at 
> org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.visitTezOp(TezPOPackageAnnotator.java:61)
>         at 
> org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:265)
>         at 
> org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:56)
>         at 
> org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
>         at 
> org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71)
>         at 
> org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71)
>         at 
> org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:52)
>         at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46)
>         at 
> org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:197)
>         at 
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290)
>         ... 12 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to