This is Pig 9.1.
Script is like :
register /home/ayon/udfs.jar;
a = load '$input' using PigStorage() as (a:chararray, b:chararray, c:int,
d:chararray);
g = group a by (a, b);
g = foreach g {
x = order $1 by c;
generate group.a, group.b, x;
};
u = foreach g generate myudf($1) as triplet; -- the triplet is a Bag of tuples
of (chararray, chararray, int)
describe u;
z = foreach u generate flatten(triplet);
z = foreach z generate $0 as p, $1 as q, $2 as r;
dump z;
-Ayon
See My Photos on Flickr
Also check out my Blog for answers to commonly asked questions.
________________________________
From: Daniel Dai <[email protected]>
To: [email protected]; Ayon Sinha <[email protected]>
Sent: Monday, December 5, 2011 6:39 PM
Subject: Re: Pig 0.9.1's not so helpful error message about ColumnMapKeyPrune
Hi, Ayon,
Which version of Pig are you using? Can you share your script?
Thanks,
Daniel
On Mon, Dec 5, 2011 at 6:10 PM, Ayon Sinha <[email protected]> wrote:
> Thanks Jonathan, that works but now I want to know why. I'm guessing I'm
> loosing some optimization. The thing is the previous version of the script
> was working fine without the -t ColumnMapKeyPrune. All I did was added a
> new column to my tuple that needs to get flattened.
>
> Anyone know what's going on inside?
>
> -Ayon
> See My Photos on Flickr
> Also check out my Blog for answers to commonly asked questions.
>
>
>
> ________________________________
> From: Jonathan Coveney <[email protected]>
> To: [email protected]; Ayon Sinha <[email protected]>
> Sent: Monday, December 5, 2011 5:58 PM
> Subject: Re: Pig 0.9.1's not so helpful error message
>
> There are some known issues with ColumnMapKeyPrune AFAIK, so instead of
> doing:
>
> pig script.pig
>
> do
>
> pig -t ColumnMapKeyPrune script.pig
>
> does that work?
>
> 2011/12/5 Ayon Sinha <[email protected]>
>
> > This is all I got during the run. What does it mean? Now I have to debug
> > line by line. Any hint is greatly appreciated.
> >
> > 2011-12-05 17:28:01,854 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> > ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t
> > ColumnMapKeyPrune
> > Details at logfile: /home/ayon/pig_1323134879161.log
> > $ cat pig_1323134879161.log
> > Pig Stack Trace
> > ---------------
> > ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t
> > ColumnMapKeyPrune
> >
> > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error
> > processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
> > at
> >
> org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
> > at
> >
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:281)
> > at org.apache.pig.PigServer.compilePp(PigServer.java:1360)
> > at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1297)
> > at org.apache.pig.PigServer.execute(PigServer.java:1286)
> > at org.apache.pig.PigServer.executeBatch(PigServer.java:360)
> > at
> > org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> > at
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> > at
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> > at org.apache.pig.Main.run(Main.java:553)
> > at org.apache.pig.Main.main(Main.java:108)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> > Caused by: java.util.ConcurrentModificationException
> > at
> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> > at java.util.AbstractList$Itr.next(AbstractList.java:343)
> > at
> >
> org.apache.pig.newplan.logical.rules.ColumnPruneVisitor.removeSubTree(ColumnPruneVisitor.java:441)
> > at
> >
> org.apache.pig.newplan.logical.rules.ColumnPruneVisitor.visit(ColumnPruneVisitor.java:421)
> > at
> >
> org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> > at
> >
> org.apache.pig.newplan.ReverseDependencyOrderWalker.walk(ReverseDependencyOrderWalker.java:70)
> > at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> > at
> >
> org.apache.pig.newplan.logical.rules.ColumnMapKeyPrune$ColumnMapKeyPruneTransformer.transform(ColumnMapKeyPrune.java:141)
> > at
> >
> org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
> > ... 16 more
> >
> >
> ================================================================================
> >
> > -Ayon
> > See My Photos on Flickr
> > Also check out my Blog for answers to commonly asked questions.
> >
