Thanks for sharing the info Dave. Som
On Tue, Oct 22, 2013 at 1:09 AM, Dave Beech <[email protected]> wrote: > Hi Som > > As well as the avro jars as Josh said, paranamer needs to be included in > the fat jar too. > > Cheers > Dave > > > On Tuesday, October 22, 2013, Som Satpathy wrote: > >> Hi Josh, >> >> Thanks for the pointer. Yes, the job fat jar is complied against >> 2.0.0-mr1-cdh4.2.0. >> I was under the assumption that 2.0.0-mr1-cdh4.2.0 is based on hadoop >> 1.x APIs. I will try again, after having it compiled against 1.0.3 >> hadoop-core. >> >> The job fat jar includes avro-1.7.4 and avro-mapred. So, I found it weird >> as to why EMR was complaining about it. >> >> Thanks, >> Som >> >> >> On Mon, Oct 21, 2013 at 8:50 PM, Josh Wills <[email protected]> wrote: >> >>> Hey Som, >>> >>> I didn't quite parse this; you're running a Crunch EMR job (which has an >>> API based on Hadoop 1.0.3), but compiling the jar against >>> 2.0.0-mr1-cdh4.2.0? I don't see how that would get very far b/c the APIs >>> are different; you should compile your local job against Hadoop 1.0.3. >>> >>> From the look of the error, I suspect that the Avro jars (avro-1.7.4.jar >>> and avro-mapred-1.7.4.jar) aren't included in the -libjars for the job >>> submission. >>> >>> J >>> >>> >>> On Mon, Oct 21, 2013 at 7:57 PM, Som Satpathy <[email protected]>wrote: >>> >>>> Hi all, >>>> >>>> I am trying to run a crunch job that does a parallelDo who's return >>>> PType is a Avros.specifics() on EMR 1.0.3. But job always throws the below >>>> error: >>>> >>>> >>>> Exception in thread "main" java.lang.NoClassDefFoundError: >>>> com/thoughtworks/paranamer/Paranamer >>>> at >>>> org.apache.avro.reflect.ReflectDatumReader.<init>(ReflectDatumReader.java:49) >>>> at >>>> org.apache.crunch.types.avro.AvroCapabilities.canDecodeSpecificSchemaWithReflectDatumReader(AvroCapabilities.java:81) >>>> at org.apache.crunch.types.avro.Avros.<clinit>(Avros.java:84) >>>> at com.research.parser.Explorer.run(Explorer.java:110) >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >>>> at com.research.parser. Explorer.main(Explorer.java:82) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:187) >>>> Caused by: java.lang.ClassNotFoundException: >>>> com.thoughtworks.paranamer.Paranamer >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>> ... 11 more >>>> >>>> I made sure that my project is using hadoop version 2.0.0-mr1-cdh4.2.0, >>>> crunch version 0.7.0, and avro 1.7.4. However, i'm still getting the above >>>> error while executing my job. >>>> >>>> I would really appreciate any pointers to help fix this problem. >>>> >>>> Thanks, >>>> Som >>>> >>> >>> >>> >>> -- >>> Director of Data Science >>> Cloudera <http://www.cloudera.com> >>> Twitter: @josh_wills <http://twitter.com/josh_wills> >>> >> >>
