Hi, you cannot add jar like this way.
please look at DistributeCache in the Hadoop Java Doc. please call DistributeCache.addArchive() in your main Class before submit the MR job. On Sat, Feb 22, 2014 at 9:30 AM, Gaurav Gupta <gau...@datatorrent.com>wrote: > Jonathan, > > > > You have to make sure that the jar is available on the nodes where the map > reduce job is running. Setting the HADOOP_CLASSPATH on the single node > doesn't work. > > You can use -libjars to the hadoop command line. > > > > Thanks > > Gaurav > > > > *From:* Jonathan Poon [mailto:jkp...@ucdavis.edu] > *Sent:* Friday, February 21, 2014 5:12 PM > *To:* user@hadoop.apache.org > *Subject:* Having trouble adding external JAR to MapReduce Program > > > > Hi Everyone, > > I'm running into trouble adding the Avro JAR into my MapReduce program. I > do the following to try to add the Avro JAR: > > export > HADOOP_CLASSPATH="/tmp/singleEvent.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar:/usr/local/hadoop/hadoop-core-1.2.1.jar" > > export > LIBJARS="/tmp/singleEvent.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar,/usr/local/hadoop/hadoop-core-1.2.1.jar" > > hadoop jar AvroReader.jar org.avro.AvroReader -libjars ${LIBJARS} > /user/jonathanpoon/avro /user/jonathanpoon/output > > However, I get the following error: > > 14/02/21 17:01:17 INFO mapred.JobClient: Task Id : > attempt_201402191318_0014_m_000001_2, Status : FAILED > java.lang.RuntimeException: java.lang.ClassNotFoundException: > org.apache.avro.mapreduce.AvroKeyInputFormat > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857) > at > org.apache.hadoop.mapreduce.JobContext.getInputFormatClass(JobContext.java:187) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: java.lang.ClassNotFoundException: > org.apache.avro.mapreduce.AvroKeyInputFormat > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:270) > at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810) > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855) > ... 8 more > > > Am I placing the Avro JAR files in the improper place? > > Thanks for your help! > > Jonathan > > >
