Hello Robert, I have used maprR to install my Hadoop stack (I prefer Cloudera but CDH doesn't support yet OOzie 4.2) so I believed that the sharelib installation was correct.
Before I change it manually, I got the java example running so I guess that the sharelib is properly built? Best, -- Axel Oehmichen Research Assistant Data Science Institute +44 (0) 7 842 734 702 [email protected] -----Original Message----- From: Robert Kanter [mailto:[email protected]] Sent: 11 November 2015 18:03 To: [email protected] Subject: Re: Spark action using python file as JAR Hi Axel, The sharelib is not properly installed. With Oozie 4.x, there's now an extra directory. Instead of /oozie/share/lib/spark it should be /oozie/share/lib/lib_<timestamp>/spark. If you use the oozie-setup script, it will properly install the sharelib for you. http://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Server_Setup - Robert On Wed, Nov 11, 2015 at 6:51 AM, Oehmichen, Axel < [email protected]> wrote: > Tried again and it yielded the same error. > > Many thanks. > > Best, > -- > Axel Oehmichen > Research Assistant > Data Science Institute > +44 (0) 7 842 734 702 > [email protected] > > > -----Original Message----- > From: Oussama Chougna [mailto:[email protected]] > Sent: 11 November 2015 14:23 > To: [email protected] > Subject: RE: Spark action using python file as JAR > > OK, > Now in your job.properties include: > oozie.use.system.libpath=true > > This tells Oozie to use that sharelib. > Cheers, > Oussama Chougna > > > From: [email protected] > > To: [email protected] > > Subject: RE: Spark action using python file as JAR > > Date: Wed, 11 Nov 2015 14:18:45 +0000 > > > > Hello Oussama, > > > > Thanks for the response. The sharelib folder does exists on HDFS > > under > /oozie/share/lib/spark > > > > Best, > > Axel > > > > -----Original Message----- > > From: Oussama Chougna [mailto:[email protected]] > > Sent: 11 November 2015 13:30 > > To: [email protected] > > Subject: RE: Spark action using python file as JAR > > > > Hi Axel, > > Did you also install the Oozie sharelib? Sounds like you missing the > sharelib, it is installed on HDFS. See the Oozie docs/MapR for a howto. > > Cheers, > > > > Oussama Chougna > > > > > From: [email protected] > > > To: [email protected] > > > Subject: Spark action using python file as JAR > > > Date: Wed, 11 Nov 2015 11:09:32 +0000 > > > > > > Hello, > > > > > > I am trying to use OOzie to get some python workflows running. I > > > have > installed OOzie and Spark using mapR 5.0 which comes with OOzie 4.2 > and Spark 1.4.1. > > > No matter what I do, I get this error message: > "java.lang.ClassNotFoundException: Class > org.apache.oozie.action.hadoop.SparkMain not found" Error code JA018. > > > > > > I was able to reproduce using the wordcount.py example. > > > (https://github.com/apache/spark/blob/master/examples/src/main/pyt > > > hon/ > > > wordcount.py) (The idea of running wordcount comes from Nitin > > > Kumar > > > message) > > > > > > The command I run is "$ /opt/mapr/oozie/oozie-4.2.0/bin/oozie job > -oozie="http://localhost:11000/oozie"; -config job.properties -run " > > > I have tried through the Java API as well and I end up with the > > > same > result. > > > > > > My job.properties contains: > > > nameNode=maprfs:/// > > > jobTracker=spark-master:8032 > > > oozie.wf.application.path=maprfs:/user/mapr/ > > > > > > my workflow.xml: > > > > > > <workflow-app xmlns='uri:oozie:workflow:0.5' name='Test'> > > > <start to='spark-node' /> > > > > > > <action name='spark-node'> > > > <spark xmlns="uri:oozie:spark-action:0.1"> > > > <job-tracker>${jobTracker}</job-tracker> > > > <name-node>${nameNode}</name-node> > > > <master>yarn-client</master> > > > <mode>client</mode> > > > <name>wordcount</name> > > > <jar>wordcount.py</jar> > > > <spark-opts>--num-executors 2 --driver-memory 1024m > --executor-memory 512m --executor-cores 1</spark-opts> > > > </spark> > > > <ok to="end" /> > > > <error to="fail" /> > > > </action> > > > > > > <kill name="fail"> > > > <message>Workflow failed, error > > > message[${wf:errorMessage(wf:lastErrorNode())}] > > > </message> > > > </kill> > > > <end name='end' /> > > > </workflow-app> > > > > > > I have tried to change the oozie.wf.application.path, specify > explicitly the jar path, remove ord add different fields in the xml, > put the wordcount a little bit everywhere and some other stuff but > nothing changed... > > > > > > I welcome any suggestion or point out any error I made. > > > > > > Many thanks. > > > > > > Axel > > > > > > >
