Pradeep Kamath updated PIG-846:

    Status: Patch Available  (was: Open)

The root cause is that MultiQueryOptimizer as part of merging MROpers updates 
the keyInfo Hashmap present in POPackage to reflect the new "index" (originally 
the index is 0 since multiquery optimization only happens with single input 
MROpers). MultiQueryOptimization uses indices in the ranges 0x80 to 0x8f by 
ORing the index with a multiQuery bitmask (0x80). These indices are assigned by 
flattenning out all POLocalRearranges across all merged map plans (including 
those nested in any POSplit opreator in a map plan). Now the corresponding 
POPackages need to have their keyInfo HashMaps updated to reflect these new 
indexes. MultiQueryOptimizer does this update for POPackage operators that 
occur in the top level POMultiQueryPackage. However if merging of MROpers 
happens recursively, we could have POMultiQueryPackage operators in the list of 
packages maintained by the topLevel POMultiQueryPackage. These were not getting 

The fix updates (recursively) the packages in any POMultiQueryPackage present 
in the top level POMultiQueryPackage.

> MultiQuery optimization in some cases has an issue when there is a split in 
> the map plan 
> -----------------------------------------------------------------------------------------
>                 Key: PIG-846
>                 URL: https://issues.apache.org/jira/browse/PIG-846
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.2.1
>            Reporter: Pradeep Kamath
>            Assignee: Pradeep Kamath
>             Fix For: 0.3.0
>         Attachments: PIG-846.patch
> The following script produces the error that follows:
> {noformat}
> A = LOAD 'input.txt' as (f0, f1, f2, f3, f4, f5, f6, f7, f8); 
> B = FOREACH A GENERATE f0, f1, f2, f3, f4;
> B1 = foreach B generate f0, f1, f2;
> C = GROUP B1 BY (f1, f2);
> STORE C into 'foo1';
> B2 = FOREACH B GENERATE f0, f3, f4;
> E = GROUP B2 BY (f3, f4);
> STORE E into 'foo2';
> F = FOREACH A GENERATE f0, f5, f6, f7, f8;
> F1 = FOREACH F GENERATE f0, f5,f6;
> G = GROUP F1 BY (f5, f6);
> STORE G into 'foo3';
> F2  = FOREACH F GENERATE f0, f7, f8;
> I = GROUP F2 BY (f7, f8);
> STORE I into 'foo4';
> {noformat}
> Exception encountered during execution:
> {noformat}
> java.lang.NullPointerException
>       at 
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackage.getValueTuple(POPackage.java:262)
>       at 
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackage.getNext(POPackage.java:209)
>       at 
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMultiQueryPackage.getNext(POMultiQueryPackage.java:186)
>       at 
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMultiQueryPackage.getNext(POMultiQueryPackage.java:186)
>       at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.processOnePackageOutput(PigMapReduce.java:277)
>       at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:268)
>       at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:142)
>       at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:318)
>       at 
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
> {noformat}

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

Reply via email to