That is confusing. Can you capture the classpath of an actual task process, not just the TT? They shouldn't differ much, but it is worth checking.
Jackson 1.3 (or was it 1.2?) and above have all been backwards compatible with each other I believe. And the error you are getting is definitely caused by accessing the enable() methods that were added after 1.0.1. I can change the Avro dependency on Jackson to 1.5.5, 1.7.1, or 1.3, and unit tests pass. If I change it to 1.2, 1.1, or 1.0.1 they break. -Scott On 1/24/11 2:40 PM, "felix gao" <[email protected]<mailto:[email protected]>> wrote: sorry for the spamming, but I forgot to include the TaskTracker's process information. root 20368 1 0 17:31 pts/0 00:00:00 su mapred -s /usr/java/default/bin/java -- -Dproc_tasktracker -Xmx1000m -Dhadoop.log.dir=/usr/lib/hadoop-0.20/logs -Dhadoop.log.file=hadoop-hadoop-tasktracker-ip-10-212-86-214.log -Dhadoop.home.dir=/usr/lib/hadoop-0.20 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,DRFA -Djava.library.path=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64:/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64 -Dhadoop.policy.file=hadoop-policy.xml -classpath /etc/hadoop-0.20/conf:/usr/java/default/lib/tools.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/hadoop-core-0.20.2+737.jar:/usr/lib/hadoop-0.20/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/hadoop-fairscheduler-0.20.2+737.jar:/usr/lib/hadoop-0.20/lib/hadoop-lzo-0.4.6.jar:/usr/lib/hadoop-0.20/lib/hadoop-thriftfs-0.20.2+737.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jackson-core-asl-1.5.5.jar:/usr/lib/hadoop-0.20/lib/jackson-mapper-asl-1.5.5.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar:/usr/lib/hadoop-0.20/lib/junit-4.5.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar:/usr/lib/hadoop-0.20/lib/hadoop-lzo.0.4.4.jar org.apache.hadoop.mapred.TaskTracker as you can see the correct version of jackson library is in the classpath. On Mon, Jan 24, 2011 at 2:09 PM, felix gao <[email protected]<mailto:[email protected]>> wrote: Chase, I tried to ran it on my local box with a standalone version of hadoop installed and I still got the same error. /usr/local/pig-0.7.0/bin/pig avro.pig 11/01/24 14:01:29 INFO pig.Main: Logging error messages to: /Users/felix/Documents/pig/pig_1295906489023.log 2011-01-24 14:01:29,328 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:/// 2011-01-24 14:01:30,277 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId= 2011-01-24 14:01:30,537 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. org.codehaus.jackson.JsonFactory.enable(Lorg/codehaus/jackson/JsonParser$Feature;)Lorg/codehaus/jackson/JsonFactory; Any ideas why? Thanks, Felix On Mon, Jan 24, 2011 at 1:18 PM, Chase Bradford <[email protected]<mailto:[email protected]>> wrote: Yes, you will need to restart them. The child tasks inherit the TT's classpath, which will list only the 1.0.1 jars until you restart the daemon. On Mon, Jan 24, 2011 at 1:11 PM, felix gao <[email protected]<mailto:[email protected]>> wrote: > Hi Guys, > > I am testing out AVRO in our cluster and having java.lang.NoSuchMethodError: > org.codehaus.jackson.JsonFactory.enable(Lorg/codehaus/jackson/JsonParser$Feature;)Lorg/codehaus/jackson/JsonFactory; > when running a simple pig script. > after took a look at AVRO-493 I uploaded the new jackson jar to replace the > CDH2's jackson 1.0.1 jars. > > > The pigscipt looks like below > REGISTER /home/pig/jars/avro-1.4.1.jar > REGISTER /home/pig/jars/json_simple-1.1.jar > REGISTER /home/pig/jars/piggybank.jar > REGISTER /usr/lib/hadoop/lib/jackson-core-asl-1.5.5.jar > REGISTER /usr/lib/hadoop/lib/jackson-mapper-asl-1.5.5.jar > > log_load = LOAD '/user/felix/avro_input/*.avro' USING > org.apache.pig.piggybank.storage.avro.AvroStorage() ; > > dump log_load > > > jackson jars are copied to each slaves and master > > The full stacktrace: > ERROR 2998: Unhandled internal error. > org.codehaus.jackson.JsonFactory.enable(Lorg/codehaus/jackson/JsonParser$Feature;)Lorg/codehaus/jackson/JsonFactory; > > java.lang.NoSuchMethodError: > org.codehaus.jackson.JsonFactory.enable(Lorg/codehaus/jackson/JsonParser$Feature;)Lorg/codehaus/jackson/JsonFactory; > at org.apache.avro.Schema.<clinit>(Schema.java:82) > at > org.apache.pig.piggybank.storage.avro.ASCommons.<clinit>(ASCommons.java:44) > at > org.apache.pig.piggybank.storage.avro.AvroStorage.getSchema(AvroStorage.java:177) > at > org.apache.pig.piggybank.storage.avro.AvroStorage.getAvroSchema(AvroStorage.java:133) > at > org.apache.pig.piggybank.storage.avro.AvroStorage.getAvroSchema(AvroStorage.java:108) > at > org.apache.pig.piggybank.storage.avro.AvroStorage.getSchema(AvroStorage.java:233) > at > org.apache.pig.impl.logicalLayer.LOLoad.determineSchema(LOLoad.java:169) > at > org.apache.pig.impl.logicalLayer.LOLoad.getSchema(LOLoad.java:150) > at > org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:843) > at > org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63) > at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1164) > at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1114) > at org.apache.pig.PigServer.registerQuery(PigServer.java:425) > at > org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:737) > at > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:324) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:162) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:138) > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:75) > at org.apache.pig.Main.main(Main.java:357) > > I am wondering if I need to restart tasktracker and jobtracker in order for > the jackson jars to be picked up? > > > > > > > > -- Chase Bradford “If in physics there's something you don't understand, you can always hide behind the uncharted depths of nature. But if your program doesn't work, there is no obstinate nature. If it doesn't work, you've messed up.” - Edsger Dijkstra
