After a bunch of fiddling around (including some pretty heavy use of the
secretDebugCmd--thanks), I finally got the LzoTokenizedStorage working,
but now I'm having problems with the LzoTokenizedLoader.
I'm still using pig 0.8.0-CDH3B4-SNAPSHOT, and for storage, have only
seemed to have luck with the hirohanin jarfile in hadoop-gpl; registered
all the other jarfiles from that package (i.e. google-collect-1.0.jar,
hadoop-lzo-0.4.8.jar, protobuf-java-2.3.0.jar, slf4j-api-1.5.8.jar,
slf4j-log4j12-1.5.10.jar, and yamlbeans-0.9.3.jar) and am now getting
the error ans stack trace:
ERROR 2997: Unable to recreate exception from backed error:
java.io.IOException: No codec for file
hdfs://localhost/rawfiles/cf938c112909470baac12a1655e4d705/1295791200/16173945/pst/part-m-00000.lzo
not found, cannot run
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable
to open iterator for alias test. Backend error : Unable to recreate
exception from backed error: java.io.IOException: No codec for file
hdfs://localhost/rawfiles/cf938c112909470baac12a1655e4d705/1295791200/16173945/pst/part-m-00000.lzo
not found, cannot run
at org.apache.pig.PigServer.openIterator(PigServer.java:742)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
at org.apache.pig.Main.run(Main.java:465)
at org.apache.pig.Main.main(Main.java:107)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR
2997: Unable to recreate exception from backed error:
java.io.IOException: No codec for file
hdfs://localhost/rawfiles/cf938c112909470baac12a1655e4d705/1295791200/16173945/pst/part-m-00000.lzo
not found, cannot run
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:221)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:151)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:337)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198)
at org.apache.pig.PigServer.storeEx(PigServer.java:874)
at org.apache.pig.PigServer.store(PigServer.java:816)
at org.apache.pig.PigServer.openIterator(PigServer.java:728)
... 7 more
I'd think that with all the jarfiles I've loaded in (and the fact that
LzoTokenizedStorage worked fine), that there should be a codec kicking
around, so I'm kinda confused here..
Any help would be appreciated.
Thanks,
Kris
On Sun, Feb 13, 2011 at 03:00:13PM -0800, Dmitriy Ryaboy wrote:
> You are still getting an IncompatibleClassChangeError?
> That usually indicates there's something wrong with the classpath,
> like perhaps your version of Pig is not the one your UDFs (in this
> case, EB) was compiled against.
>
> Can you check your classpath? You can print out exactly what Pig is
> running by passing the "-secretDebugCmd" flag to bin/pig.
>
> D
>
> On Sun, Feb 13, 2011 at 11:00 AM, Kris Coward <[email protected]> wrote:
> >
> > Upgrading to pig 0.8.0-CDH3B4-SNAPSHOT, hasn't solved the problem, even
> > trying with each of of the 0.8.0 jar files (the hadoopgpl package used
> > provides 3 of them, one tagged as hirohanin's, another as gerritjvv's
> > and another as yours). I'm going to keep banging away at this, but would
> > be happy to hear any other ideas that might help.
> >
> > Thanks,
> > Kris
> >
> > On Fri, Feb 11, 2011 at 11:15:56AM -0800, Dmitriy Ryaboy wrote:
> >> The 0.7 branch of EB is not a real thing, it's hirohanin's first
> >> attempt at porting that has a billion bugs.
> >> Most things in it do not, in fact, work.
> >> Use the 0.8 branch and Pig 0.8...
> >>
> >> D
> >>
> >> On Fri, Feb 11, 2011 at 10:47 AM, Kris Coward <[email protected]> wrote:
> >> >
> >> > So in the interest of being a little less i/o bound, and saving a whole
> >> > mess of disk, I've started using
> >> > com.twitter.elephantbird.pig.store.LzoTokenizedStorage for storage... or
> >> > more accurately, will be using it as soon as I stop getting the
> >> > following error (with stack trace):
> >> >
> >> > ERROR 2998: Unhandled internal error. Implementing class
> >> >
> >> > java.lang.IncompatibleClassChangeError: Implementing class
> >> > at java.lang.ClassLoader.defineClass1(Native Method)
> >> > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> >> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> >> > at
> >> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> >> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> >> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> >> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> >> > at java.security.AccessController.doPrivileged(Native Method)
> >> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> >> > at java.lang.ClassLoader.defineClass1(Native Method)
> >> > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> >> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> >> > at
> >> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> >> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> >> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> >> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> >> > at java.security.AccessController.doPrivileged(Native Method)
> >> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> >> > at java.lang.Class.forName0(Native Method)
> >> > at java.lang.Class.forName(Class.java:247)
> >> > at
> >> > org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:422)
> >> > at
> >> > org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:452)
> >> > at
> >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.NonEvalFuncSpec(QueryParser.java:5087)
> >> > at
> >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.StoreClause(QueryParser.java:3568)
> >> > at
> >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1369)
> >> > at
> >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:911)
> >> > at
> >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:724)
> >> > at
> >> > org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
> >> > at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1336)
> >> > at
> >> > org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1286)
> >> > at org.apache.pig.PigServer.registerQuery(PigServer.java:460)
> >> > at
> >> > org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:738)
> >> > at
> >> > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:324)
> >> > at
> >> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:163)
> >> > at
> >> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:139)
> >> > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89)
> >> > at org.apache.pig.Main.main(Main.java:414)
> >> >
> >> > Anyhow, the host machine is running CentOS, with the Cloudera
> >> > distribution of hadoop and pig (pig version: Apache Pig version
> >> > 0.7.0+16),
> >> > and elephant-bird from the hadoop-gpl-packing RPM. The tweaks described
> >> > at http://code.google.com/p/hadoop-gpl-packing/#Using_in_Pig have been
> >> > applied, and the class still seems to be failing to load.
> >> >
> >> > Anyone have any idea what the problem might be (or how to solve it)?
> >> >
> >> > Thanks,
> >> > Kris
> >> >
> >> >
> >> > --
> >> > Kris Coward http://unripe.melon.org/
> >> > GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3
> >> >
> >
> > --
> > Kris Coward http://unripe.melon.org/
> > GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3
> >
--
Kris Coward http://unripe.melon.org/
GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3
