Hey folks,
i still try to setup elephant bird in pig. I am using the pig-08 branch of
dvryaboy.
i managed to create my example loader using the pig8.util.ThriftToPig
my pig code looks like this..
raw_data = load '/tmp/thrift/vi_base64.txt.lzo' using
com.twitter.elephantbird.pig8.load.LzoThriftB64LinePigLoader('com.example.thrift.VectorSequence');
When i run it, i get the following exception after the map/reduce phase:
elephantbird.thrift.class.for.com.twitter.elephantbird.mapreduce.input.LzoThriftB64LineInputFormat
is not set
Seems like i miss a classpath again? But of course the elephantbird libs are
included, otherwise the script would fail much earlier withe the
LzoThriftB64LinePigLoader missing itself?
May the data be corrupted?
See the stack trace attached, i hope you have some idea.
Best regards,
Torben
Backend error message
---------------------
java.lang.RuntimeException: java.lang.RuntimeException:
elephantbird.thrift.class.for.com.twitter.elephantbird.mapreduce.input.LzoThriftB64LineInputFormat
is not set
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.initNextRecordReader(PigRecordReader.java:236)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.<init>(PigRecordReader.java:109)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:118)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
at org.apache.hadoop.mapred.Child$4.run(Child.java:240)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.hadoop.mapred.Child.main(Child.java:234)
Caused by: java.lang.RuntimeException:
elephantbird.thrift.class.for.com.twitter.elephantbird.mapreduce.input.LzoThriftB64LineInputFormat
is not set
at com.twitter.elephantbird.util.ThriftUtils.getTypeRef(Unknown Source)
at
com.twitter.elephantbird.mapreduce.input.LzoThriftB64LineInputFormat.createRecordReader(Unknown
Source)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.initNextRecordReader(PigRecordReader.java:227)
... 9 more
Pig Stack Trace
---------------
ERROR 2997: Unable to recreate exception from backed error:
java.lang.RuntimeException: java.lang.RuntimeException:
elephantbird.thrift.class.for.com.twitter.elephantbird.mapreduce.input.LzoThriftB64LineInputFormat
is not set
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
open iterator for alias raw_data. Backend error : Unable to recreate
exception from backed error: java.lang.RuntimeException:
java.lang.RuntimeException:
elephantbird.thrift.class.for.com.twitter.elephantbird.mapreduce.input.LzoThriftB64LineInputFormat
is not set
at org.apache.pig.PigServer.openIterator(PigServer.java:742)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:406)
at org.apache.pig.Main.main(Main.java:107)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2997:
Unable to recreate exception from backed error: java.lang.RuntimeException:
java.lang.RuntimeException:
elephantbird.thrift.class.for.com.twitter.elephantbird.mapreduce.input.LzoThriftB64LineInputFormat
is not set
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:221)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:151)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:337)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198)
at org.apache.pig.PigServer.storeEx(PigServer.java:874)
at org.apache.pig.PigServer.store(PigServer.java:816)
at org.apache.pig.PigServer.openIterator(PigServer.java:728)
... 7 more
================================================================================
