Split -> distinct or order -> cogroup fails
-------------------------------------------
Key: PIG-425
URL: https://issues.apache.org/jira/browse/PIG-425
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: types_branch
Reporter: Alan Gates
Assignee: Alan Gates
Fix For: types_branch
A script like:
{code}
\a = load 'myfile' as (name:chararray, age:int, gpa:double);
split a into a1 if age > 50, a2 if name < 'm';
b2 = distinct a2;
b1 = order a1 by name;
c = cogroup b2 by name, b1 by name;
d = foreach c generate flatten(group), COUNT($1), COUNT($2);
store d into 'OUTPATH';
{code}
Will abort with the error:
{code}
08/09/09 11:46:50 ERROR mapReduceLayer.Launcher: Error message from task (map)
tip_200809080906_0185_m_000000java.lang.ClassCastException:
org.apache.pig.data.DefaultTuple cannot be cast to
org.apache.pig.data.IndexedTuple
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:81)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:135)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:75)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
{code}
The issue is that the RearrangeAdjuster in MRCompiler is not properly seeing
this as a cogroup and moving the localrearrnge out of the reduce and into the
map.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
