Issue with Crunch + avros on EMR 1.0.3

[Som Satpathy]

Hi all,

I am trying to run a crunch job that does a parallelDo who's return PType
is a Avros.specifics() on EMR 1.0.3. But job always throws the below error:


Exception in thread "main" java.lang.NoClassDefFoundError:
com/thoughtworks/paranamer/Paranamer
        at
org.apache.avro.reflect.ReflectDatumReader.<init>(ReflectDatumReader.java:49)
        at
org.apache.crunch.types.avro.AvroCapabilities.canDecodeSpecificSchemaWithReflectDatumReader(AvroCapabilities.java:81)
        at org.apache.crunch.types.avro.Avros.<clinit>(Avros.java:84)
        at com.research.parser.Explorer.run(Explorer.java:110)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at com.research.parser. Explorer.main(Explorer.java:82)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:187)
Caused by: java.lang.ClassNotFoundException:
com.thoughtworks.paranamer.Paranamer
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 11 more

I made sure that my project is using hadoop version 2.0.0-mr1-cdh4.2.0,
crunch version 0.7.0, and avro 1.7.4. However, i'm still getting the above
error while executing my job.

I would really appreciate any pointers to help fix this problem.

Thanks,
Som