[
https://issues.apache.org/jira/browse/PIG-512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh Srinivasan updated PIG-512:
------------------------------------
Attachment: PIG-512.patch
Atached patch (PIG-512.patch) includes the following:
1. Logical Plan cloning which in turn includes logical operator cloning.
Caveat: Only logical plan cloning is allowed and LogicalPlanCloner is the
supported mechanism for cloning logical plans. The following operators do not
support cloning:
i. LOLoad
ii. LOStore
iii. LOStream
2. A visitor to remove redundant project( * ) operators that occur between two
relational operators or between two expression operators.
3. Unit tests for 1 and 2
All unit tests pass.
> Expressions in foreach lead to errors
> -------------------------------------
>
> Key: PIG-512
> URL: https://issues.apache.org/jira/browse/PIG-512
> Project: Pig
> Issue Type: Bug
> Affects Versions: types_branch
> Reporter: Santhosh Srinivasan
> Assignee: Santhosh Srinivasan
> Fix For: types_branch
>
> Attachments: PIG-512.patch
>
>
> Use of expressions that use the same sub-expressions in foreach lead to
> translation errors. This issue is caused due to sharing operators across
> nested plans. To remedy this issue, logical operators should be cloned and
> not shared across plans.
> {code}
> grunt> a = load 'a' as (x, y, z);
> grunt> b = foreach a {
> >> exp1 = x + y;
> >> exp2 = exp1 + x;
> >> generate exp1, exp2;
> >> }
> grunt> explain b;
> 2008-10-30 15:38:40,257 [main] WARN org.apache.pig.PigServer - bytearray is
> implicitly casted to double under LOAdd Operator
> 2008-10-30 15:38:40,258 [main] WARN org.apache.pig.PigServer - bytearray is
> implicitly casted to double under LOAdd Operator
> 2008-10-30 15:38:40,258 [main] WARN org.apache.pig.PigServer - bytearray is
> implicitly casted to double under LOAdd Operator
> Logical Plan:
> Store sms-Thu Oct 30 11:27:27 PDT 2008-2609 Schema: {double,double} Type:
> Unknown
> |
> |---ForEach sms-Thu Oct 30 11:27:27 PDT 2008-2605 Schema: {double,double}
> Type: bag
> | |
> | Add sms-Thu Oct 30 11:27:27 PDT 2008-2600 FieldSchema: double Type:
> double
> | |
> | |---Cast sms-Thu Oct 30 11:27:27 PDT 2008-2606 FieldSchema: double
> Type: double
> | | |
> | | |---Project sms-Thu Oct 30 11:27:27 PDT 2008-2598 Projections:
> [0] Overloaded: false FieldSchema: x: bytearray Type: bytearray
> | | Input: Load sms-Thu Oct 30 11:27:27 PDT 2008-2597
> | |
> | |---Cast sms-Thu Oct 30 11:27:27 PDT 2008-2607 FieldSchema: double
> Type: double
> | |
> | |---Project sms-Thu Oct 30 11:27:27 PDT 2008-2599 Projections:
> [1] Overloaded: false FieldSchema: y: bytearray Type: bytearray
> | Input: Load sms-Thu Oct 30 11:27:27 PDT 2008-2597
> | |
> | Add sms-Thu Oct 30 11:27:27 PDT 2008-2603 FieldSchema: double Type:
> double
> | |
> | |---Project sms-Thu Oct 30 11:27:27 PDT 2008-2601 Projections: [*]
> Overloaded: false FieldSchema: double Type: double
> | | Input: Add sms-Thu Oct 30 11:27:27 PDT 2008-2600|
> | | |---Add sms-Thu Oct 30 11:27:27 PDT 2008-2600 FieldSchema: double
> Type: double
> | | |
> | | |---Project sms-Thu Oct 30 11:27:27 PDT 2008-2598
> Projections: [0] Overloaded: false FieldSchema: x: bytearray Type: bytearray
> | | | Input: Load sms-Thu Oct 30 11:27:27 PDT 2008-2597
> | | |
> | | |---Project sms-Thu Oct 30 11:27:27 PDT 2008-2599
> Projections: [1] Overloaded: false FieldSchema: y: bytearray Type: bytearray
> | | Input: Load sms-Thu Oct 30 11:27:27 PDT 2008-2597
> | |
> | |---Cast sms-Thu Oct 30 11:27:27 PDT 2008-2608 FieldSchema: double
> Type: double
> | |
> | |---Project sms-Thu Oct 30 11:27:27 PDT 2008-2602 Projections:
> [0] Overloaded: false FieldSchema: x: bytearray Type: bytearray
> | Input: Load sms-Thu Oct 30 11:27:27 PDT 2008-2597
> |
> |---Load sms-Thu Oct 30 11:27:27 PDT 2008-2597 Schema: {x: bytearray,y:
> bytearray,z: bytearray} Type: bag
> 2008-10-30 15:38:40,272 [main] ERROR org.apache.pig.impl.plan.OperatorPlan -
> Attempt to give operator of type
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject
> multiple outputs. This operator does not support multiple outputs.
> 2008-10-30 15:38:40,272 [main] ERROR
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.LogToPhyTranslationVisitor
> - Invalid physical operators in the physical planAttempt to give operator of
> type
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject
> multiple outputs. This operator does not support multiple outputs.
> 2008-10-30 15:38:40,273 [main] ERROR org.apache.pig.tools.grunt.GruntParser -
> java.io.IOException: Unable to explain alias b
> [org.apache.pig.impl.plan.VisitorException]
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:235)
> at org.apache.pig.PigServer.compilePp(PigServer.java:731)
> at org.apache.pig.PigServer.explain(PigServer.java:495)
> at
> org.apache.pig.tools.grunt.GruntParser.processExplain(GruntParser.java:155)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:193)
> at
> org.apache.pig.tools.grunt.GruntParser.parseContOnError(GruntParser.java:94)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:58)
> at org.apache.pig.Main.main(Main.java:282)
> Caused by: org.apache.pig.backend.executionengine.ExecException:
> org.apache.pig.impl.plan.VisitorException
> ... 8 more
> Caused by: org.apache.pig.impl.plan.VisitorException
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.LogToPhyTranslationVisitor.visit(LogToPhyTranslationVisitor.java:324)
> at org.apache.pig.impl.logicalLayer.LOAdd.visit(LOAdd.java:69)
> at org.apache.pig.impl.logicalLayer.LOAdd.visit(LOAdd.java:29)
> at
> org.apache.pig.impl.plan.DependencyOrderWalkerWOSeenChk.walk(DependencyOrderWalkerWOSeenChk.java:68)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.LogToPhyTranslationVisitor.visit(LogToPhyTranslationVisitor.java:805)
> at
> org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:121)
> at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:40)
> at
> org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:232)
> ... 7 more
> Caused by: org.apache.pig.impl.plan.PlanException: Attempt to give operator
> of type
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject
> multiple outputs. This operator does not support multiple outputs.
> at
> org.apache.pig.impl.plan.OperatorPlan.connect(OperatorPlan.java:158)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.plans.PhysicalPlan.connect(PhysicalPlan.java:89)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.LogToPhyTranslationVisitor.visit(LogToPhyTranslationVisitor.java:320)
> ... 16 more
> 2008-10-30 15:38:40,274 [main] ERROR org.apache.pig.tools.grunt.GruntParser -
> Unable to explain alias b [org.apache.pig.impl.plan.VisitorException]
> 2008-10-30 15:38:40,274 [main] ERROR org.apache.pig.tools.grunt.GruntParser -
> java.io.IOException: Unable to explain alias b
> [org.apache.pig.impl.plan.VisitorException]
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
