[jira] [Updated] (PIG-5375) NullPointerException for multi-level self unions with Tez UnionOptimizer
[ https://issues.apache.org/jira/browse/PIG-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5375: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 0.18.0 Status: Resolved (was: Patch Available) Thanks for the review Rohini! Committed to trunk(0.18). > NullPointerException for multi-level self unions with Tez UnionOptimizer > > > Key: PIG-5375 > URL: https://issues.apache.org/jira/browse/PIG-5375 > Project: Pig > Issue Type: Bug >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Fix For: 0.18.0 > > Attachments: pig-5375-v1.patch, pig-5375.png > > > {code} > A = load 'input.txt' as (a0:int, a1: chararray, a2:int); > B = load 'input.txt' as (a0:int, a1: chararray, a2:int); > C = load 'input.txt' as (a0:int, a1: chararray, a2:int); > A_and_B = UNION A, B; > SPLIT A_and_B INTO A_and_B2 IF a0 > 10, A_and_B3 OTHERWISE; > A_and_B_and_C = UNION ONSCHEMA C, A_and_B; > X = UNION ONSCHEMA A_and_B_and_C, A_and_B2, A_and_B3; > X2 = GROUP X ALL ; > dump X2; > {code} > This fails _on Tez_ with > {noformat} > Pig Stack Trace > --- > ERROR 1002: Unable to store alias X2 > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to > open iterator for alias X2 > at org.apache.pig.PigServer.openIterator(PigServer.java:1024) > at > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:790) > at > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:383) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:231) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206) > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) > at org.apache.pig.Main.run(Main.java:630) > at org.apache.pig.Main.main(Main.java:175) > Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias X2 > at org.apache.pig.PigServer.storeEx(PigServer.java:1127) > at org.apache.pig.PigServer.store(PigServer.java:1086) > at org.apache.pig.PigServer.openIterator(PigServer.java:999) > ... 7 more > Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: > java.lang.NullPointerException > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:296) > at org.apache.pig.PigServer.launchPlan(PigServer.java:1479) > at > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464) > at org.apache.pig.PigServer.storeEx(PigServer.java:1123) > ... 9 more > Caused by: java.lang.NullPointerException > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.patchPackage(TezPOPackageAnnotator.java:97) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.handlePackage(TezPOPackageAnnotator.java:78) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.visitTezOp(TezPOPackageAnnotator.java:61) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:265) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:56) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71) > at > org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:52) > at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:197) > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290) > ... 12 more > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PIG-5375) NullPointerException for multi-level self unions with Tez UnionOptimizer
[ https://issues.apache.org/jira/browse/PIG-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated PIG-5375: Status: Patch Available (was: Open) > NullPointerException for multi-level self unions with Tez UnionOptimizer > > > Key: PIG-5375 > URL: https://issues.apache.org/jira/browse/PIG-5375 > Project: Pig > Issue Type: Bug >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Attachments: pig-5375-v1.patch, pig-5375.png > > > {code} > A = load 'input.txt' as (a0:int, a1: chararray, a2:int); > B = load 'input.txt' as (a0:int, a1: chararray, a2:int); > C = load 'input.txt' as (a0:int, a1: chararray, a2:int); > A_and_B = UNION A, B; > SPLIT A_and_B INTO A_and_B2 IF a0 > 10, A_and_B3 OTHERWISE; > A_and_B_and_C = UNION ONSCHEMA C, A_and_B; > X = UNION ONSCHEMA A_and_B_and_C, A_and_B2, A_and_B3; > X2 = GROUP X ALL ; > dump X2; > {code} > This fails _on Tez_ with > {noformat} > Pig Stack Trace > --- > ERROR 1002: Unable to store alias X2 > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to > open iterator for alias X2 > at org.apache.pig.PigServer.openIterator(PigServer.java:1024) > at > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:790) > at > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:383) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:231) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206) > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) > at org.apache.pig.Main.run(Main.java:630) > at org.apache.pig.Main.main(Main.java:175) > Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias X2 > at org.apache.pig.PigServer.storeEx(PigServer.java:1127) > at org.apache.pig.PigServer.store(PigServer.java:1086) > at org.apache.pig.PigServer.openIterator(PigServer.java:999) > ... 7 more > Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: > java.lang.NullPointerException > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:296) > at org.apache.pig.PigServer.launchPlan(PigServer.java:1479) > at > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464) > at org.apache.pig.PigServer.storeEx(PigServer.java:1123) > ... 9 more > Caused by: java.lang.NullPointerException > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.patchPackage(TezPOPackageAnnotator.java:97) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.handlePackage(TezPOPackageAnnotator.java:78) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.visitTezOp(TezPOPackageAnnotator.java:61) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:265) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:56) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71) > at > org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:52) > at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:197) > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290) > ... 12 more > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PIG-5375) NullPointerException for multi-level self unions with Tez UnionOptimizer
[ https://issues.apache.org/jira/browse/PIG-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5375: -- Attachment: pig-5375.png > NullPointerException for multi-level self unions with Tez UnionOptimizer > > > Key: PIG-5375 > URL: https://issues.apache.org/jira/browse/PIG-5375 > Project: Pig > Issue Type: Bug >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Attachments: pig-5375-v1.patch, pig-5375.png > > > {code} > A = load 'input.txt' as (a0:int, a1: chararray, a2:int); > B = load 'input.txt' as (a0:int, a1: chararray, a2:int); > C = load 'input.txt' as (a0:int, a1: chararray, a2:int); > A_and_B = UNION A, B; > SPLIT A_and_B INTO A_and_B2 IF a0 > 10, A_and_B3 OTHERWISE; > A_and_B_and_C = UNION ONSCHEMA C, A_and_B; > X = UNION ONSCHEMA A_and_B_and_C, A_and_B2, A_and_B3; > X2 = GROUP X ALL ; > dump X2; > {code} > This fails _on Tez_ with > {noformat} > Pig Stack Trace > --- > ERROR 1002: Unable to store alias X2 > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to > open iterator for alias X2 > at org.apache.pig.PigServer.openIterator(PigServer.java:1024) > at > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:790) > at > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:383) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:231) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206) > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) > at org.apache.pig.Main.run(Main.java:630) > at org.apache.pig.Main.main(Main.java:175) > Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias X2 > at org.apache.pig.PigServer.storeEx(PigServer.java:1127) > at org.apache.pig.PigServer.store(PigServer.java:1086) > at org.apache.pig.PigServer.openIterator(PigServer.java:999) > ... 7 more > Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: > java.lang.NullPointerException > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:296) > at org.apache.pig.PigServer.launchPlan(PigServer.java:1479) > at > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464) > at org.apache.pig.PigServer.storeEx(PigServer.java:1123) > ... 9 more > Caused by: java.lang.NullPointerException > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.patchPackage(TezPOPackageAnnotator.java:97) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.handlePackage(TezPOPackageAnnotator.java:78) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.visitTezOp(TezPOPackageAnnotator.java:61) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:265) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:56) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71) > at > org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:52) > at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:197) > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290) > ... 12 more > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PIG-5375) NullPointerException for multi-level self unions with Tez UnionOptimizer
[ https://issues.apache.org/jira/browse/PIG-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5375: -- Attachment: (was: hadooppf-5375.pdf) > NullPointerException for multi-level self unions with Tez UnionOptimizer > > > Key: PIG-5375 > URL: https://issues.apache.org/jira/browse/PIG-5375 > Project: Pig > Issue Type: Bug >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Attachments: pig-5375-v1.patch > > > {code} > A = load 'input.txt' as (a0:int, a1: chararray, a2:int); > B = load 'input.txt' as (a0:int, a1: chararray, a2:int); > C = load 'input.txt' as (a0:int, a1: chararray, a2:int); > A_and_B = UNION A, B; > SPLIT A_and_B INTO A_and_B2 IF a0 > 10, A_and_B3 OTHERWISE; > A_and_B_and_C = UNION ONSCHEMA C, A_and_B; > X = UNION ONSCHEMA A_and_B_and_C, A_and_B2, A_and_B3; > X2 = GROUP X ALL ; > dump X2; > {code} > This fails _on Tez_ with > {noformat} > Pig Stack Trace > --- > ERROR 1002: Unable to store alias X2 > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to > open iterator for alias X2 > at org.apache.pig.PigServer.openIterator(PigServer.java:1024) > at > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:790) > at > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:383) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:231) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206) > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) > at org.apache.pig.Main.run(Main.java:630) > at org.apache.pig.Main.main(Main.java:175) > Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias X2 > at org.apache.pig.PigServer.storeEx(PigServer.java:1127) > at org.apache.pig.PigServer.store(PigServer.java:1086) > at org.apache.pig.PigServer.openIterator(PigServer.java:999) > ... 7 more > Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: > java.lang.NullPointerException > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:296) > at org.apache.pig.PigServer.launchPlan(PigServer.java:1479) > at > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464) > at org.apache.pig.PigServer.storeEx(PigServer.java:1123) > ... 9 more > Caused by: java.lang.NullPointerException > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.patchPackage(TezPOPackageAnnotator.java:97) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.handlePackage(TezPOPackageAnnotator.java:78) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.visitTezOp(TezPOPackageAnnotator.java:61) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:265) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:56) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71) > at > org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:52) > at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:197) > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290) > ... 12 more > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PIG-5375) NullPointerException for multi-level self unions with Tez UnionOptimizer
[ https://issues.apache.org/jira/browse/PIG-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5375: -- Attachment: (was: hadooppf-5375.png) > NullPointerException for multi-level self unions with Tez UnionOptimizer > > > Key: PIG-5375 > URL: https://issues.apache.org/jira/browse/PIG-5375 > Project: Pig > Issue Type: Bug >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Attachments: pig-5375-v1.patch > > > {code} > A = load 'input.txt' as (a0:int, a1: chararray, a2:int); > B = load 'input.txt' as (a0:int, a1: chararray, a2:int); > C = load 'input.txt' as (a0:int, a1: chararray, a2:int); > A_and_B = UNION A, B; > SPLIT A_and_B INTO A_and_B2 IF a0 > 10, A_and_B3 OTHERWISE; > A_and_B_and_C = UNION ONSCHEMA C, A_and_B; > X = UNION ONSCHEMA A_and_B_and_C, A_and_B2, A_and_B3; > X2 = GROUP X ALL ; > dump X2; > {code} > This fails _on Tez_ with > {noformat} > Pig Stack Trace > --- > ERROR 1002: Unable to store alias X2 > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to > open iterator for alias X2 > at org.apache.pig.PigServer.openIterator(PigServer.java:1024) > at > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:790) > at > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:383) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:231) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206) > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) > at org.apache.pig.Main.run(Main.java:630) > at org.apache.pig.Main.main(Main.java:175) > Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias X2 > at org.apache.pig.PigServer.storeEx(PigServer.java:1127) > at org.apache.pig.PigServer.store(PigServer.java:1086) > at org.apache.pig.PigServer.openIterator(PigServer.java:999) > ... 7 more > Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: > java.lang.NullPointerException > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:296) > at org.apache.pig.PigServer.launchPlan(PigServer.java:1479) > at > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464) > at org.apache.pig.PigServer.storeEx(PigServer.java:1123) > ... 9 more > Caused by: java.lang.NullPointerException > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.patchPackage(TezPOPackageAnnotator.java:97) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.handlePackage(TezPOPackageAnnotator.java:78) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.visitTezOp(TezPOPackageAnnotator.java:61) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:265) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:56) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71) > at > org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:52) > at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:197) > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290) > ... 12 more > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PIG-5375) NullPointerException for multi-level self unions with Tez UnionOptimizer
[ https://issues.apache.org/jira/browse/PIG-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5375: -- Attachment: hadooppf-5375.png > NullPointerException for multi-level self unions with Tez UnionOptimizer > > > Key: PIG-5375 > URL: https://issues.apache.org/jira/browse/PIG-5375 > Project: Pig > Issue Type: Bug >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Attachments: hadooppf-5375.pdf, hadooppf-5375.png, pig-5375-v1.patch > > > {code} > A = load 'input.txt' as (a0:int, a1: chararray, a2:int); > B = load 'input.txt' as (a0:int, a1: chararray, a2:int); > C = load 'input.txt' as (a0:int, a1: chararray, a2:int); > A_and_B = UNION A, B; > SPLIT A_and_B INTO A_and_B2 IF a0 > 10, A_and_B3 OTHERWISE; > A_and_B_and_C = UNION ONSCHEMA C, A_and_B; > X = UNION ONSCHEMA A_and_B_and_C, A_and_B2, A_and_B3; > X2 = GROUP X ALL ; > dump X2; > {code} > This fails _on Tez_ with > {noformat} > Pig Stack Trace > --- > ERROR 1002: Unable to store alias X2 > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to > open iterator for alias X2 > at org.apache.pig.PigServer.openIterator(PigServer.java:1024) > at > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:790) > at > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:383) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:231) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206) > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) > at org.apache.pig.Main.run(Main.java:630) > at org.apache.pig.Main.main(Main.java:175) > Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias X2 > at org.apache.pig.PigServer.storeEx(PigServer.java:1127) > at org.apache.pig.PigServer.store(PigServer.java:1086) > at org.apache.pig.PigServer.openIterator(PigServer.java:999) > ... 7 more > Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: > java.lang.NullPointerException > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:296) > at org.apache.pig.PigServer.launchPlan(PigServer.java:1479) > at > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464) > at org.apache.pig.PigServer.storeEx(PigServer.java:1123) > ... 9 more > Caused by: java.lang.NullPointerException > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.patchPackage(TezPOPackageAnnotator.java:97) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.handlePackage(TezPOPackageAnnotator.java:78) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.visitTezOp(TezPOPackageAnnotator.java:61) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:265) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:56) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71) > at > org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:52) > at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:197) > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290) > ... 12 more > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PIG-5375) NullPointerException for multi-level self unions with Tez UnionOptimizer
[ https://issues.apache.org/jira/browse/PIG-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5375: -- Attachment: hadooppf-5375.pdf For my reference (and maybe for the reviewer) attaching how the plan evolved with vertexgroup by the UnionOptimizer before my patch. (using the script from the description) !hadooppf-5375.pdf|thumbnail! Note the bottom of the vertex group output {panel} Tez vertex group scope-87 <- [{color:#FF}scope-56{color}, scope-53, {color:#FF}scope-56{color}, scope-54, scope-55] -> scope-81 {panel} where the scope-56 no longer exist. What my patch does is, instead of replacing one scope-56 with scope-54&scope-55, it replaces all three scope-56 with scope-54&scope-55. > NullPointerException for multi-level self unions with Tez UnionOptimizer > > > Key: PIG-5375 > URL: https://issues.apache.org/jira/browse/PIG-5375 > Project: Pig > Issue Type: Bug >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Attachments: hadooppf-5375.pdf, pig-5375-v1.patch > > > {code} > A = load 'input.txt' as (a0:int, a1: chararray, a2:int); > B = load 'input.txt' as (a0:int, a1: chararray, a2:int); > C = load 'input.txt' as (a0:int, a1: chararray, a2:int); > A_and_B = UNION A, B; > SPLIT A_and_B INTO A_and_B2 IF a0 > 10, A_and_B3 OTHERWISE; > A_and_B_and_C = UNION ONSCHEMA C, A_and_B; > X = UNION ONSCHEMA A_and_B_and_C, A_and_B2, A_and_B3; > X2 = GROUP X ALL ; > dump X2; > {code} > This fails _on Tez_ with > {noformat} > Pig Stack Trace > --- > ERROR 1002: Unable to store alias X2 > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to > open iterator for alias X2 > at org.apache.pig.PigServer.openIterator(PigServer.java:1024) > at > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:790) > at > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:383) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:231) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206) > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) > at org.apache.pig.Main.run(Main.java:630) > at org.apache.pig.Main.main(Main.java:175) > Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias X2 > at org.apache.pig.PigServer.storeEx(PigServer.java:1127) > at org.apache.pig.PigServer.store(PigServer.java:1086) > at org.apache.pig.PigServer.openIterator(PigServer.java:999) > ... 7 more > Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: > java.lang.NullPointerException > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:296) > at org.apache.pig.PigServer.launchPlan(PigServer.java:1479) > at > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464) > at org.apache.pig.PigServer.storeEx(PigServer.java:1123) > ... 9 more > Caused by: java.lang.NullPointerException > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.patchPackage(TezPOPackageAnnotator.java:97) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.handlePackage(TezPOPackageAnnotator.java:78) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.visitTezOp(TezPOPackageAnnotator.java:61) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:265) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:56) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71) > at > org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:52) > at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:197) > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290) > ... 12 more > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PIG-5375) NullPointerException for multi-level self unions with Tez UnionOptimizer
[ https://issues.apache.org/jira/browse/PIG-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5375: -- Attachment: pig-5375-v1.patch Script in the description works when last union only has store-outputs. (Doing {{dump X}} instead of {{dump X2}}.) This is because Tez's UnionOptimizer.java has slightly different logic for store-outputs compared to handling of non-store-outputs. Attaching a patch that changes the logic of non-store-outputs and making it similar to how store-outputs finds existingVertexGroup at the beginning. > NullPointerException for multi-level self unions with Tez UnionOptimizer > > > Key: PIG-5375 > URL: https://issues.apache.org/jira/browse/PIG-5375 > Project: Pig > Issue Type: Bug >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Attachments: pig-5375-v1.patch > > > {code} > A = load 'input.txt' as (a0:int, a1: chararray, a2:int); > B = load 'input.txt' as (a0:int, a1: chararray, a2:int); > C = load 'input.txt' as (a0:int, a1: chararray, a2:int); > A_and_B = UNION A, B; > SPLIT A_and_B INTO A_and_B2 IF a0 > 10, A_and_B3 OTHERWISE; > A_and_B_and_C = UNION ONSCHEMA C, A_and_B; > X = UNION ONSCHEMA A_and_B_and_C, A_and_B2, A_and_B3; > X2 = GROUP X ALL ; > dump X2; > {code} > This fails _on Tez_ with > {noformat} > Pig Stack Trace > --- > ERROR 1002: Unable to store alias X2 > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to > open iterator for alias X2 > at org.apache.pig.PigServer.openIterator(PigServer.java:1024) > at > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:790) > at > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:383) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:231) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206) > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) > at org.apache.pig.Main.run(Main.java:630) > at org.apache.pig.Main.main(Main.java:175) > Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias X2 > at org.apache.pig.PigServer.storeEx(PigServer.java:1127) > at org.apache.pig.PigServer.store(PigServer.java:1086) > at org.apache.pig.PigServer.openIterator(PigServer.java:999) > ... 7 more > Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: > java.lang.NullPointerException > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:296) > at org.apache.pig.PigServer.launchPlan(PigServer.java:1479) > at > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464) > at org.apache.pig.PigServer.storeEx(PigServer.java:1123) > ... 9 more > Caused by: java.lang.NullPointerException > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.patchPackage(TezPOPackageAnnotator.java:97) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.handlePackage(TezPOPackageAnnotator.java:78) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPOPackageAnnotator.visitTezOp(TezPOPackageAnnotator.java:61) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:265) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:56) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:71) > at > org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:52) > at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:197) > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290) > ... 12 more > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)