Sorry for the delay in responding. I set HADOOP_CLASSPATH to point to a dir containing a valid log4j.xml
Yes, the HADOOP_USER_NAME fix worked for me. Cheers, -Gautam. On Fri, May 15, 2015 at 9:47 AM, Mohammad Islam <[email protected]> wrote: > Hi Gautam,What did you set in HADOOP_CLASSPATH?It appears you are using > Hadoop 1.x or earlier version in non-keberized world. > You need to set "<env-var>HADOOP_USER_NAME=${wf:user()}</env-var>" in your > shell action.This should solve the "marred" user issue. > Regards,Mohammad > > > > On Friday, May 15, 2015 12:49 AM, Gautam <[email protected]> > wrote: > > > Got past the log4j problem. Made a bash script that sets the > HADOOP_CLASSPATH and runs hadoop jar from within. > > > Next problem, the job is getting submitted as 'mapred' user, for some > reason, instead of the user that submitted the oozie job. Fails when > creating the staging dir.. > > Exception in thread "main" java.lang.RuntimeException: java.io.IOException: > The ownership on the staging directory > hdfs://{namenode}:8020/user/wand/.staging is not as expected. It is owned > by wand. The directory must be owned by the submitter mapred or by mapred > at com.upsight.valhalla.kafka.dumper.KafkaDumper.main(KafkaDumper.java:27) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) at > org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: > java.io.IOException: The ownership on the staging directory hdfs:// > master22.stg.kontagent.com:8020/user/wand/.staging is not as expected. It > is owned by wand. The directory must be owned by the submitter mapred or by > mapred at > > org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:113) > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:982) at > org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:976) at > java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:415) at > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) > at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:976) > > > > > On Thu, May 14, 2015 at 11:16 PM, Mohammad Islam > <[email protected] > > wrote: > > > This is sometime tricky and needs careful debugging.I think it could be > > related to different versions of same jar. or conflicting with other jar. > > you can try to set "ooze.launcher.mapreduce.job.classloader" in shell > > action's configuration. > > Please let us know how it goes. One important observation is : it is not > > ClassNotFoundException rather NoClassDefFoundError. > > Regards,Mohammad > > > > > > > > On Thursday, May 14, 2015 8:53 PM, Gautam <[email protected]> > > wrote: > > > > > > Hello, > > I'm trying to submit a hadoop jar job via Oozie workflow. In > > particular, trying to run it using the shell action by invoking the > > /usr/bin/hadoop jar <jar_file> <MRClass>.. > > > > > > I tested this job separately from the command line and it works when I > just > > submit using "hadoop jar <jar_file> <class_name> " . But the same job > fails > > with the following error from within Oozie flow step with this .. > > > > Exception in thread "main" java.lang.NoClassDefFoundError: Could not > > initialize class org.apache.log4j.LogManager > > at org.apache.log4j.Logger.getLogger(Logger.java:117) > > at com.linkedin.camus.etl.kafka.CamusJob.<init>(CamusJob.java:128) > > at com.linkedin.camus.etl.kafka.CamusJob.<init>(CamusJob.java:124) > > at > > com.upsight.valhalla.kafka.dumper.KafkaDumper.main(KafkaDumper.java:19) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > > > > > > > > I'v made sure there's a log4j.xml and the LogManager impl in my uber jar. > > I'd imagine others have tried submitting a MR job with "hadoop jar " > > runtime. Although, I'm also aware of the <map-reduce> action but it's too > > cumbersome for me since i set a lot of params, counters in my job client > > class (which implements the Tool abstract class). > > > > > > > > Here's my workflow step config: > > > > <action name="phShimDump"> > > <shell xmlns="uri:oozie:shell-action:0.3"> > > <job-tracker>${jobTracker}</job-tracker> > > <name-node>${nameNode}</name-node> > > <exec>/usr/bin/hadoop</exec> > > <argument>jar</argument> > > <argument>kafka-hdfs-etl/jar/kafka-hdfs-etl.jar</argument> > > > > <argument>com.upsight.valhalla.kafka.dumper.KafkaDumper</argument> > > > > <!--<argument>kafka-hdfs-etl/script/kafka-etl-console</argument>--> > > <argument>-P</argument> > > > > <argument>kafka-hdfs-etl/script/oozie/${phShimDumpConfig}</argument> > > <argument>-D</argument> > > <argument>camus.job.name=ph_up_shim_data</argument> > > <argument>-D</argument> > > <argument>zookeeper.hosts=${zookeeperHosts}</argument> > > <argument>-D</argument> > > <argument>kafka.brokers=${kafkaBrokers}</argument> > > > > > <archive>/opt/kafka-hdfs-etl/kafka-hdfs-etl.tar.gz#kafka-hdfs-etl</archive> > > <capture-output/> > > </shell> > > <ok to="end" /> > > <error to="kill" /> > > </action> > > > > > > thanks and regards, > > -Gautam. > > > > > > > > > > > > -- > "If you really want something in this life, you have to work for it. Now, > quiet! They're about to announce the lottery numbers..." > > > > -- "If you really want something in this life, you have to work for it. Now, quiet! They're about to announce the lottery numbers..."
