Hi, I can give you some ideas. Hope to be helpful :
Add spark.job.queuename or smthg like this in your spark program or in oozie configuration node (like mapred.job.queuename) you can add some spark libs in your shell action with the # to put all of them in the same container. Good luck! Le jeudi 4 juin 2015, Muqtafi Akhmad <[email protected]> a écrit : > dear all, > Currently I am trying to submit Spark application using spark-submit (with > Spark application copy exists in all YARN nodes) by writing it down in > Oozie shell action, here is the workflow.xml : > http://pastebin.com/kHVtmNex > > When submitting this job, I kept getting > Error code : JA018 > Error message : Main class [org.apache.oozie.action.hadoop.ShellMain], exit > code [1] > > What do those messages mean? > Thank you, > > > > On Mon, Jun 1, 2015 at 7:45 PM, Jaydeep Vishwakarma < > [email protected] <javascript:;>> wrote: > > > I am not sure what to look. There is a detail log hyperlink option in > > Launcher log on ui. You can try to search the exception over there. > > > > You can add any library as part of sharelib. You just need to push the > lib > > directory inside the sharelib directory of hdfs and update the sharelibs > by > > "oozie admin -sharelibupdate" command. > > > > Spark libs have to present in hdfs, It is independent to modes. > > > > > > On Mon, Jun 1, 2015 at 3:24 PM, Muqtafi Akhmad <[email protected] > <javascript:;>> > > wrote: > > > > > Hello Jaydeep, > > > Logs from > > > > > > > oozie job -info ... > > > > > > and > > > > > > > yarn logs -applicationId ... > > > > > > give the same explanation like the one I've got from Oozie web app. Is > > > there any meaning for > > > > > > > Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [1] > > > > > > error? > > > > > > It is good that Spark action will be supported by Oozie, but given our > > > current Oozie version, can developer add Spark (or other 'common') > > > libraries to Oozie's shared libs? When you mention > > > > > > > It is only working as local mode. > > > > > > does it mean that we don't need to copy Spark libs and apps to HDFS? > > > Thank you, > > > > > > On Mon, Jun 1, 2015 at 3:17 PM, Jaydeep Vishwakarma < > > > [email protected] <javascript:;>> wrote: > > > > > > > You can get more details for this error on Launcher logs by accessing > > the > > > > console url. > > > > Spark action has been recently introduced in oozie. It will come in > > > 4.2.0 , > > > > But there are some issue with spark action, It is only working as > local > > > > mode. Here is the jira for same. > > > > https://issues.apache.org/jira/browse/OOZIE-2143. > > > > > > > > Regards, > > > > Jaydeep > > > > > > > > > > > > > > > > > > > > On Mon, Jun 1, 2015 at 12:43 PM, Muqtafi Akhmad < > [email protected] <javascript:;> > > > > > > > wrote: > > > > > > > > > Hello everyone, > > > > > I'm new to Oozie, currently I'm trying to use Oozie to manage Spark > > > > > applications. I've tried to submit Spark application as Java > action ( > > > as > > > > > described in > > > > > > > > > > > > > > > > > > > > http://stackoverflow.com/questions/29233487/launching-a-spark-program-using-oozie-workflow > > > > > ) and the action return error > > > > > > > > > > > Main class [org.apache.oozie.action.hadoop.JavaMain], exit code > [1] > > > > > > > > > > My questions are > > > > > 1. How can I see / access the complete stack trace to know what > > exactly > > > > > happened in my application execution? > > > > > 2. By default, Oozie shared lib directory provides libraries for > > Hive, > > > > Pig, > > > > > and Map-Reduce. If I want to run Spark job on Oozie, it might be > > better > > > > to > > > > > add Spark lib jars to Oozie's shared lib instead of copy them to > > app's > > > > lib > > > > > directory. How can I add Spark lib jars (including spark-core and > its > > > > > dependencies) to Oozie's shared lib? > > > > > > > > > > Thank you, > > > > > > > > > > -- > > > > > Muqtafi Akhmad > > > > > Software Engineer > > > > > Traveloka > > > > > > > > > > > > > -- > > > > _____________________________________________________________ > > > > The information contained in this communication is intended solely > for > > > the > > > > use of the individual or entity to whom it is addressed and others > > > > authorized to receive it. It may contain confidential or legally > > > privileged > > > > information. If you are not the intended recipient you are hereby > > > notified > > > > that any disclosure, copying, distribution or taking any action in > > > reliance > > > > on the contents of this information is strictly prohibited and may be > > > > unlawful. If you have received this communication in error, please > > notify > > > > us immediately by responding to this email and then delete it from > your > > > > system. The firm is neither liable for the proper and complete > > > transmission > > > > of the information contained in this communication nor for any delay > in > > > its > > > > receipt. > > > > > > > > > > > > > > > > -- > > > Muqtafi Akhmad > > > Software Engineer > > > Traveloka > > > > > > > -- > > _____________________________________________________________ > > The information contained in this communication is intended solely for > the > > use of the individual or entity to whom it is addressed and others > > authorized to receive it. It may contain confidential or legally > privileged > > information. If you are not the intended recipient you are hereby > notified > > that any disclosure, copying, distribution or taking any action in > reliance > > on the contents of this information is strictly prohibited and may be > > unlawful. If you have received this communication in error, please notify > > us immediately by responding to this email and then delete it from your > > system. The firm is neither liable for the proper and complete > transmission > > of the information contained in this communication nor for any delay in > its > > receipt. > > > > > > -- > Muqtafi Akhmad > Software Engineer > Traveloka > -- -- Laurent HATIER - Consultant Big Data & Business Intelligence chez CapGemini fr.linkedin.com/pub/laurent-hatier/25/36b/a86/ <http://fr.linkedin.com/pub/laurent-h/25/36b/a86/>
