[
https://issues.apache.org/jira/browse/PIG-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995049#comment-12995049
]
Laukik Chitnis commented on PIG-1491:
-------------------------------------
This is the script I executed:
{noformat}
A = load '$input1' as (a, b, c, d, x, m, n, u, v);
B = load '$input2' as (a, b, c, d, x, m, n, u, v);
DATA_G = COGROUP A by (a, b, c, d) OUTER, B by (a, b, c, d) OUTER;
DATA = FOREACH DATA_G {
a_items = DISTINCT A.x;
b_items = DISTINCT B.x;
GENERATE
FLATTEN(group) as (a,b,c,d),
SUM(A.m) as m, SUM(A.n) as n,
COUNT(a_items) as a_item_count,
(long)(SUM(B.u) + (double)0.5) as u,
(long)(SUM(B.v) + (double)0.5) as v,
COUNT(b_items) as b_item_count;
}
store DATA into '$output';
{noformat}
My sample inputs:
{noformat}
1 1 1 1 10 10 10 10 10
2 2 2 2 20 20 20 20 20
1 1 1 1 10 10 10 10 10
{noformat}
{noformat}
1 1 1 1 100 100 100 100 100
2 2 2 2 200 200 200 200 200
2 2 2 2 202 202 202 202 202
1 1 1 1 100 100 100 100 100
{noformat}
And the output:
{noformat}
1 1 1 1 20.0 20.0 1 200 200 1
2 2 2 2 20.0 20.0 1 402 402 2
{noformat}
This was tested on svn versions 1068769 (recent trunk) and 966485 (Pig version
0.7.0 compiled on Jul 22 2010)
@Scott
Can you try the latest trunk, and close the issue if it cannot be reproduced?
Thanks!
> Failure planning nested FOREACH with DISTINCT, POLoad cannot be cast to
> POLocalRearrange
> ----------------------------------------------------------------------------------------
>
> Key: PIG-1491
> URL: https://issues.apache.org/jira/browse/PIG-1491
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Scott Carey
> Assignee: Laukik Chitnis
> Fix For: 0.9.0
>
>
> I have a failure that occurs during planning while using DISTINCT in a nested
> FOREACH.
> Caused by: java.lang.ClassCastException:
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad
> cannot be cast to
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizer.visitMROp(SecondaryKeyOptimizer.java:352)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:218)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:40)
> at
> org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
