[
https://issues.apache.org/jira/browse/PIG-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934628#action_12934628
]
Xuefu Zhang commented on PIG-1741:
----------------------------------
+1
Patch looks fine as a temporary fix, as we are moving to the new logical plan,
where lineage is done different. Per the issue, the right fix should be to find
why the canonical map is empty and create the map, instead of checking if the
map is empty as done in the fix.
> Lineage fail when flatten a bag
> -------------------------------
>
> Key: PIG-1741
> URL: https://issues.apache.org/jira/browse/PIG-1741
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.8.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: PIG-1741-0.patch, PIG-1741-1.patch
>
>
> The following script fail:
> {code}
> a = load '1.txt' as (a0);
> b = foreach a generate flatten((bag{tuple(map[])})a0) as b0:map[];
> c = foreach b generate (long)b0#'key1';
> dump c;
> {code}
> 1.txt:
> {([key1#1])}
> Error message:
> org.apache.pig.backend.executionengine.ExecException: ERROR 1075: Received a
> bytearray from the UDF. Cannot determine how to convert the bytearray to long.
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:286)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:361)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:291)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:236)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> In logical plan:
> {code}
> fake: Store 1-30 Schema: {long} Type: Unknown
> |
> |---c: ForEach 1-29 Schema: {long} Type: bag
> | |
> | Cast 1-28--null FieldSchema: long Type: long
> | |
> | |---MapLookup 1-26 FieldSchema: bytearray Type: bytearray
> | |
> | |---Project 1-27 Projections: [0] Overloaded: false FieldSchema:
> b0: map Type: map
> | Input: b: ForEach 1-21
> |
> |---b: ForEach 1-21 Schema: {b0: map[ ]} Type: bag
> | |
> | Cast 1-20--org.apache.pig.builtin.PigStorage FieldSchema: a0:
> bag({(map[ ])}) Type: bag
> | |
> | |---Project 1-19 Projections: [0] Overloaded: false FieldSchema:
> a0: bytearray Type: bytearray
> | Input: a: Load 1-15
> |
> |---a: Load 1-15 Schema: {a0: bytearray} Type: bag
> {code}
> Cast 1-28 get null caster. This error cannot be fixed by
> -Dusenewlogicalplan=false flag.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
