Hi EdwardKing, I had a similar issue come up last week, though mine was because I was trying to use Avro 1.7.5 instead of 1.7.4. After some Googling I found a MapReduce job config parameter you can change which solved my problem.
It turns out that Hadoop places jars it ships with BEFORE user defined jars on the classpath by default. This means that if you specify Avro and Hadoop specifies Avro, its version (1.7.4) goes first. I needed MY jars to come first so I could use updated versions of Avro and other libraries. The parameter I changed is this: mapreduce.job.user.classpath.first = true Please see these two links for more details: http://www.kiji.org/2013/10/08/fixing-classpath-ordering-issues-in-hadoop/+ https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/kiWv2PFjT1s Good luck, Gary On Mon, Feb 24, 2014 at 2:45 AM, EdwardKing <[email protected]> wrote: > I use hadoop 2.2.0 to run avro, I know hadoop contains > avro-1.7.4.jar,like follows: > [hadoop@master lib]$ pwd > /home/software/hadoop-2.2.0/share/hadoop/common/lib > > [hadoop@master lib]$ ls av* > avro-1.7.4.jar > > Then I put avro-1.7.4.jar into classpath > $export > CLASSPATH=.:/home/software/hadoop-2.2.0/share/hadoop/common/lib/avro-1.7.4.jar:${CLASSPATH} > > My code is follows: > > public int run(String[] args) throws Exception{ > JobConf conf=new JobConf(getConf(),getClass()); > conf.setJobName("UFO count"); > String[] otherArgs=new > GenericOptionsParser(conf,args).getRemainingArgs(); > if(otherArgs.length!=2){ > System.err.println("Usage: avro UFO counter <in><out>"); > System.exit(2); > } > FileInputFormat.addInputPath(conf,new Path(otherArgs[0])); > Path outputPath=new Path(otherArgs[1]); > FileOutputFormat.setOutputPath(conf,outputPath); > outputPath.getFileSystem(conf).delete(outputPath); > Schema > input_schema=Schema.parse(getClass().getResourceAsStream("ufo.avsc")); > AvroJob.setInputSchema(conf,input_schema); > > AvroJob.setMapOutputSchema(conf,Pair.getPairSchema(Schema.create(Schema.Type.STRING),Schema.create(Schema.Type.LONG))); > AvroJob.setOutputSchema(conf,OUTPUT_SCHEMA); > AvroJob.setMapperClass(conf,AvroRecordMapper.class); > AvroJob.setReducerClass(conf,AvroRecordReducer.class); > conf.setInputFormat(AvroInputFormat.class); > JobClient.runJob(conf); > /*----------------AvroMR.java:41----------------*/ > > > Then I run avro > > [hadoop@master ~]$ hadoop jar avroufo.jar AvroMR avroin avroout > 14/02/24 00:06:54 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 14/02/24 00:06:56 INFO client.RMProxy: Connecting to ResourceManager at > master/172.11.12.6:8993 > 14/02/24 00:06:56 INFO client.RMProxy: Connecting to ResourceManager at > master/172.11.12.6:8993 > 14/02/24 00:07:00 INFO mapred.FileInputFormat: Total input paths to > process : 1 > 14/02/24 00:07:00 INFO mapreduce.JobSubmitter: number of splits:2 > 14/02/24 00:07:00 INFO Configuration.deprecation: user.name is > deprecated. Instead, use mapreduce.job.user.name > 14/02/24 00:07:00 INFO Configuration.deprecation: mapred.jar is > deprecated. Instead, use mapreduce.job.jar > 14/02/24 00:07:00 INFO Configuration.deprecation: > mapred.output.key.comparator.class is deprecated. Instead, use > mapreduce.job.output.key.comparator.class > 14/02/24 00:07:00 INFO Configuration.deprecation: > mapred.mapoutput.value.class is deprecated. Instead, use > mapreduce.map.output.value.class > 14/02/24 00:07:00 INFO Configuration.deprecation: > mapred.used.genericoptionsparser is deprecated. Instead, use > mapreduce.client.genericoptionsparser.used > 14/02/24 00:07:00 INFO Configuration.deprecation: mapred.job.name is > deprecated. Instead, use mapreduce.job.name > 14/02/24 00:07:00 INFO Configuration.deprecation: mapred.input.dir is > deprecated. Instead, use mapreduce.input.fileinputformat.inputdir > 14/02/24 00:07:00 INFO Configuration.deprecation: mapred.output.dir is > deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir > 14/02/24 00:07:00 INFO Configuration.deprecation: mapred.map.tasks is > deprecated. Instead, use mapreduce.job.maps > 14/02/24 00:07:00 INFO Configuration.deprecation: mapred.output.key.class > is deprecated. Instead, use mapreduce.job.output.key.class > 14/02/24 00:07:00 INFO Configuration.deprecation: > mapred.mapoutput.key.class is deprecated. Instead, use > mapreduce.map.output.key.class > 14/02/24 00:07:00 INFO Configuration.deprecation: mapred.working.dir is > deprecated. Instead, use mapreduce.job.working.dir > 14/02/24 00:07:01 INFO mapreduce.JobSubmitter: Submitting tokens for job: > job_1393229044702_0001 > 14/02/24 00:07:04 INFO impl.YarnClientImpl: Submitted application > application_1393229044702_0001 to ResourceManager at master/ > 172.11.12.6:8993 > 14/02/24 00:07:04 INFO mapreduce.Job: The url to track the job: > http://master:8088/proxy/application_1393229044702_0001/ > 14/02/24 00:07:04 INFO mapreduce.Job: Running job: job_1393229044702_0001 > 14/02/24 00:07:59 INFO mapreduce.Job: Job job_1393229044702_0001 running > in uber mode : false > 14/02/24 00:07:59 INFO mapreduce.Job: map 0% reduce 0% > 14/02/24 00:12:12 INFO mapreduce.Job: map 50% reduce 0% > 14/02/24 00:12:13 INFO mapreduce.Job: Task Id : > attempt_1393229044702_0001_m_000001_0, Status : FAILED > Error: > org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter; > 14/02/24 00:12:13 INFO mapreduce.Job: Task Id : > attempt_1393229044702_0001_m_000000_0, Status : FAILED > Error: > org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter; > 14/02/24 00:12:14 INFO mapreduce.Job: map 0% reduce 0% > 14/02/24 00:12:30 INFO mapreduce.Job: Task Id : > attempt_1393229044702_0001_m_000001_1, Status : FAILED > Error: > org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter; > 14/02/24 00:12:31 INFO mapreduce.Job: Task Id : > attempt_1393229044702_0001_m_000000_1, Status : FAILED > Error: > org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter; > 14/02/24 00:12:43 INFO mapreduce.Job: Task Id : > attempt_1393229044702_0001_m_000001_2, Status : FAILED > Error: > org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter; > 14/02/24 00:12:44 INFO mapreduce.Job: Task Id : > attempt_1393229044702_0001_m_000000_2, Status : FAILED > Error: > org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter; > 14/02/24 00:12:56 INFO mapreduce.Job: map 50% reduce 0% > 14/02/24 00:12:57 INFO mapreduce.Job: map 100% reduce 100% > 14/02/24 00:12:57 INFO mapreduce.Job: Job job_1393229044702_0001 failed > with state FAILED due to: Task failed task_1393229044702_0001_m_000001 > Job failed as tasks failed. failedMaps:1 failedReduces:0 > > 14/02/24 00:12:57 INFO mapreduce.Job: Counters: 10 > Job Counters > Failed map tasks=7 > Killed map tasks=1 > Launched map tasks=8 > Other local map tasks=6 > Data-local map tasks=2 > Total time spent by all maps in occupied slots (ms)=571664 > Total time spent by all reduces in occupied slots (ms)=0 > Map-Reduce Framework > CPU time spent (ms)=0 > Physical memory (bytes) snapshot=0 > Virtual memory (bytes) snapshot=0 > Exception in thread "main" java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) > at AvroMR.run(AvroMR.java:41) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at AvroMR.main(AvroMR.java:67) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > [hadoop@master ~]$ > > I view the history of Track UI,I find following information: > 2014-02-24 00:12:12,624 FATAL [main] org.apache.hadoop.mapred.YarnChild: > Error running child : java.lang.NoSuchMethodError: > org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter; > at > org.apache.avro.mapred.AvroSerialization.getSerializer(AvroSerialization.java:107) > > I know avro-1.7.4.jar contains > org.apache.avro.generic.GenericData.createDatumWriter,why it still raise > the error "java.lang.NoSuchMethodError"? How to correct it? Thanks. > > > > > > --------------------------------------------------------------------------------------------------- > Confidentiality Notice: The information contained in this e-mail and any > accompanying attachment(s) > is intended only for the use of the intended recipient and may be > confidential and/or privileged of > Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader > of this communication is > not the intended recipient, unauthorized use, forwarding, printing, > storing, disclosure or copying > is strictly prohibited, and may be unlawful.If you have received this > communication in error,please > immediately notify the sender by return e-mail, and delete the original > message and all copies from > your system. Thank you. > > --------------------------------------------------------------------------------------------------- >
