Hello,
I'm trying to submit a hadoop jar job via Oozie workflow. In
particular, trying to run it using the shell action by invoking the
/usr/bin/hadoop jar <jar_file> <MRClass>..
I tested this job separately from the command line and it works when I just
submit using "hadoop jar <jar_file> <class_name> " . But the same job fails
with the following error from within Oozie flow step with this ..
Exception in thread "main" java.lang.NoClassDefFoundError: Could not
initialize class org.apache.log4j.LogManager
at org.apache.log4j.Logger.getLogger(Logger.java:117)
at com.linkedin.camus.etl.kafka.CamusJob.<init>(CamusJob.java:128)
at com.linkedin.camus.etl.kafka.CamusJob.<init>(CamusJob.java:124)
at
com.upsight.valhalla.kafka.dumper.KafkaDumper.main(KafkaDumper.java:19)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
I'v made sure there's a log4j.xml and the LogManager impl in my uber jar.
I'd imagine others have tried submitting a MR job with "hadoop jar "
runtime. Although, I'm also aware of the <map-reduce> action but it's too
cumbersome for me since i set a lot of params, counters in my job client
class (which implements the Tool abstract class).
Here's my workflow step config:
<action name="phShimDump">
<shell xmlns="uri:oozie:shell-action:0.3">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<exec>/usr/bin/hadoop</exec>
<argument>jar</argument>
<argument>kafka-hdfs-etl/jar/kafka-hdfs-etl.jar</argument>
<argument>com.upsight.valhalla.kafka.dumper.KafkaDumper</argument>
<!--<argument>kafka-hdfs-etl/script/kafka-etl-console</argument>-->
<argument>-P</argument>
<argument>kafka-hdfs-etl/script/oozie/${phShimDumpConfig}</argument>
<argument>-D</argument>
<argument>camus.job.name=ph_up_shim_data</argument>
<argument>-D</argument>
<argument>zookeeper.hosts=${zookeeperHosts}</argument>
<argument>-D</argument>
<argument>kafka.brokers=${kafkaBrokers}</argument>
<archive>/opt/kafka-hdfs-etl/kafka-hdfs-etl.tar.gz#kafka-hdfs-etl</archive>
<capture-output/>
</shell>
<ok to="end" />
<error to="kill" />
</action>
thanks and regards,
-Gautam.