Hey Som, I didn't quite parse this; you're running a Crunch EMR job (which has an API based on Hadoop 1.0.3), but compiling the jar against 2.0.0-mr1-cdh4.2.0? I don't see how that would get very far b/c the APIs are different; you should compile your local job against Hadoop 1.0.3.
>From the look of the error, I suspect that the Avro jars (avro-1.7.4.jar and avro-mapred-1.7.4.jar) aren't included in the -libjars for the job submission. J On Mon, Oct 21, 2013 at 7:57 PM, Som Satpathy <[email protected]> wrote: > Hi all, > > I am trying to run a crunch job that does a parallelDo who's return PType > is a Avros.specifics() on EMR 1.0.3. But job always throws the below error: > > > Exception in thread "main" java.lang.NoClassDefFoundError: > com/thoughtworks/paranamer/Paranamer > at > org.apache.avro.reflect.ReflectDatumReader.<init>(ReflectDatumReader.java:49) > at > org.apache.crunch.types.avro.AvroCapabilities.canDecodeSpecificSchemaWithReflectDatumReader(AvroCapabilities.java:81) > at org.apache.crunch.types.avro.Avros.<clinit>(Avros.java:84) > at com.research.parser.Explorer.run(Explorer.java:110) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at com.research.parser. Explorer.main(Explorer.java:82) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.main(RunJar.java:187) > Caused by: java.lang.ClassNotFoundException: > com.thoughtworks.paranamer.Paranamer > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 11 more > > I made sure that my project is using hadoop version 2.0.0-mr1-cdh4.2.0, > crunch version 0.7.0, and avro 1.7.4. However, i'm still getting the above > error while executing my job. > > I would really appreciate any pointers to help fix this problem. > > Thanks, > Som > -- Director of Data Science Cloudera <http://www.cloudera.com> Twitter: @josh_wills <http://twitter.com/josh_wills>
