This is sometime tricky and needs careful debugging.I think it could be related
to different versions of same jar. or conflicting with other jar.
you can try to set "ooze.launcher.mapreduce.job.classloader" in shell action's
configuration.
Please let us know how it goes. One important observation is : it is not
ClassNotFoundException rather NoClassDefFoundError.
Regards,Mohammad
On Thursday, May 14, 2015 8:53 PM, Gautam <[email protected]> wrote:
Hello,
I'm trying to submit a hadoop jar job via Oozie workflow. In
particular, trying to run it using the shell action by invoking the
/usr/bin/hadoop jar <jar_file> <MRClass>..
I tested this job separately from the command line and it works when I just
submit using "hadoop jar <jar_file> <class_name> " . But the same job fails
with the following error from within Oozie flow step with this ..
Exception in thread "main" java.lang.NoClassDefFoundError: Could not
initialize class org.apache.log4j.LogManager
at org.apache.log4j.Logger.getLogger(Logger.java:117)
at com.linkedin.camus.etl.kafka.CamusJob.<init>(CamusJob.java:128)
at com.linkedin.camus.etl.kafka.CamusJob.<init>(CamusJob.java:124)
at com.upsight.valhalla.kafka.dumper.KafkaDumper.main(KafkaDumper.java:19)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
I'v made sure there's a log4j.xml and the LogManager impl in my uber jar.
I'd imagine others have tried submitting a MR job with "hadoop jar "
runtime. Although, I'm also aware of the <map-reduce> action but it's too
cumbersome for me since i set a lot of params, counters in my job client
class (which implements the Tool abstract class).
Here's my workflow step config:
<action name="phShimDump">
<shell xmlns="uri:oozie:shell-action:0.3">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<exec>/usr/bin/hadoop</exec>
<argument>jar</argument>
<argument>kafka-hdfs-etl/jar/kafka-hdfs-etl.jar</argument>
<argument>com.upsight.valhalla.kafka.dumper.KafkaDumper</argument>
<!--<argument>kafka-hdfs-etl/script/kafka-etl-console</argument>-->
<argument>-P</argument>
<argument>kafka-hdfs-etl/script/oozie/${phShimDumpConfig}</argument>
<argument>-D</argument>
<argument>camus.job.name=ph_up_shim_data</argument>
<argument>-D</argument>
<argument>zookeeper.hosts=${zookeeperHosts}</argument>
<argument>-D</argument>
<argument>kafka.brokers=${kafkaBrokers}</argument>
<archive>/opt/kafka-hdfs-etl/kafka-hdfs-etl.tar.gz#kafka-hdfs-etl</archive>
<capture-output/>
</shell>
<ok to="end" />
<error to="kill" />
</action>
thanks and regards,
-Gautam.
