thank you Laurent! But I think I'll go with the new Oozie version now that Oozie supports Spark apps
On Sat, Jun 6, 2015 at 4:59 AM, Laurent H <[email protected]> wrote: > Maybe add the spark-site.xml (if this file exists, I am not a spark > expert!) To the container! > > Le vendredi 5 juin 2015, Laurent H <[email protected]> a écrit : > > > Hi, > > > > I can give you some ideas. Hope to be helpful : > > > > Add spark.job.queuename or smthg like this in your spark program or in > > oozie configuration node (like mapred.job.queuename) you can add some > spark > > libs in your shell action with the # to put all of them in the same > > container. > > > > Good luck! > > > > Le jeudi 4 juin 2015, Muqtafi Akhmad <[email protected] > > <javascript:_e(%7B%7D,'cvml','[email protected]');>> a écrit : > > > >> dear all, > >> Currently I am trying to submit Spark application using spark-submit > (with > >> Spark application copy exists in all YARN nodes) by writing it down in > >> Oozie shell action, here is the workflow.xml : > >> http://pastebin.com/kHVtmNex > >> > >> When submitting this job, I kept getting > >> Error code : JA018 > >> Error message : Main class [org.apache.oozie.action.hadoop.ShellMain], > >> exit > >> code [1] > >> > >> What do those messages mean? > >> Thank you, > >> > >> > >> > >> On Mon, Jun 1, 2015 at 7:45 PM, Jaydeep Vishwakarma < > >> [email protected]> wrote: > >> > >> > I am not sure what to look. There is a detail log hyperlink option in > >> > Launcher log on ui. You can try to search the exception over there. > >> > > >> > You can add any library as part of sharelib. You just need to push the > >> lib > >> > directory inside the sharelib directory of hdfs and update the > >> sharelibs by > >> > "oozie admin -sharelibupdate" command. > >> > > >> > Spark libs have to present in hdfs, It is independent to modes. > >> > > >> > > >> > On Mon, Jun 1, 2015 at 3:24 PM, Muqtafi Akhmad <[email protected] > > > >> > wrote: > >> > > >> > > Hello Jaydeep, > >> > > Logs from > >> > > > >> > > > oozie job -info ... > >> > > > >> > > and > >> > > > >> > > > yarn logs -applicationId ... > >> > > > >> > > give the same explanation like the one I've got from Oozie web app. > Is > >> > > there any meaning for > >> > > > >> > > > Main class [org.apache.oozie.action.hadoop.JavaMain], exit code > [1] > >> > > > >> > > error? > >> > > > >> > > It is good that Spark action will be supported by Oozie, but given > our > >> > > current Oozie version, can developer add Spark (or other 'common') > >> > > libraries to Oozie's shared libs? When you mention > >> > > > >> > > > It is only working as local mode. > >> > > > >> > > does it mean that we don't need to copy Spark libs and apps to HDFS? > >> > > Thank you, > >> > > > >> > > On Mon, Jun 1, 2015 at 3:17 PM, Jaydeep Vishwakarma < > >> > > [email protected]> wrote: > >> > > > >> > > > You can get more details for this error on Launcher logs by > >> accessing > >> > the > >> > > > console url. > >> > > > Spark action has been recently introduced in oozie. It will come > in > >> > > 4.2.0 , > >> > > > But there are some issue with spark action, It is only working as > >> local > >> > > > mode. Here is the jira for same. > >> > > > https://issues.apache.org/jira/browse/OOZIE-2143. > >> > > > > >> > > > Regards, > >> > > > Jaydeep > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > On Mon, Jun 1, 2015 at 12:43 PM, Muqtafi Akhmad < > >> [email protected] > >> > > > >> > > > wrote: > >> > > > > >> > > > > Hello everyone, > >> > > > > I'm new to Oozie, currently I'm trying to use Oozie to manage > >> Spark > >> > > > > applications. I've tried to submit Spark application as Java > >> action ( > >> > > as > >> > > > > described in > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > http://stackoverflow.com/questions/29233487/launching-a-spark-program-using-oozie-workflow > >> > > > > ) and the action return error > >> > > > > > >> > > > > > Main class [org.apache.oozie.action.hadoop.JavaMain], exit > code > >> [1] > >> > > > > > >> > > > > My questions are > >> > > > > 1. How can I see / access the complete stack trace to know what > >> > exactly > >> > > > > happened in my application execution? > >> > > > > 2. By default, Oozie shared lib directory provides libraries for > >> > Hive, > >> > > > Pig, > >> > > > > and Map-Reduce. If I want to run Spark job on Oozie, it might be > >> > better > >> > > > to > >> > > > > add Spark lib jars to Oozie's shared lib instead of copy them to > >> > app's > >> > > > lib > >> > > > > directory. How can I add Spark lib jars (including spark-core > and > >> its > >> > > > > dependencies) to Oozie's shared lib? > >> > > > > > >> > > > > Thank you, > >> > > > > > >> > > > > -- > >> > > > > Muqtafi Akhmad > >> > > > > Software Engineer > >> > > > > Traveloka > >> > > > > > >> > > > > >> > > > -- > >> > > > _____________________________________________________________ > >> > > > The information contained in this communication is intended solely > >> for > >> > > the > >> > > > use of the individual or entity to whom it is addressed and others > >> > > > authorized to receive it. It may contain confidential or legally > >> > > privileged > >> > > > information. If you are not the intended recipient you are hereby > >> > > notified > >> > > > that any disclosure, copying, distribution or taking any action in > >> > > reliance > >> > > > on the contents of this information is strictly prohibited and may > >> be > >> > > > unlawful. If you have received this communication in error, please > >> > notify > >> > > > us immediately by responding to this email and then delete it from > >> your > >> > > > system. The firm is neither liable for the proper and complete > >> > > transmission > >> > > > of the information contained in this communication nor for any > >> delay in > >> > > its > >> > > > receipt. > >> > > > > >> > > > >> > > > >> > > > >> > > -- > >> > > Muqtafi Akhmad > >> > > Software Engineer > >> > > Traveloka > >> > > > >> > > >> > -- > >> > _____________________________________________________________ > >> > The information contained in this communication is intended solely for > >> the > >> > use of the individual or entity to whom it is addressed and others > >> > authorized to receive it. It may contain confidential or legally > >> privileged > >> > information. If you are not the intended recipient you are hereby > >> notified > >> > that any disclosure, copying, distribution or taking any action in > >> reliance > >> > on the contents of this information is strictly prohibited and may be > >> > unlawful. If you have received this communication in error, please > >> notify > >> > us immediately by responding to this email and then delete it from > your > >> > system. The firm is neither liable for the proper and complete > >> transmission > >> > of the information contained in this communication nor for any delay > in > >> its > >> > receipt. > >> > > >> > >> > >> > >> -- > >> Muqtafi Akhmad > >> Software Engineer > >> Traveloka > >> > > > > > > -- > > -- > > Laurent HATIER - Consultant Big Data & Business Intelligence chez > CapGemini > > fr.linkedin.com/pub/laurent-hatier/25/36b/a86/ > > <http://fr.linkedin.com/pub/laurent-h/25/36b/a86/> > > > > > > -- > -- > Laurent HATIER - Consultant Big Data & Business Intelligence chez CapGemini > fr.linkedin.com/pub/laurent-hatier/25/36b/a86/ > <http://fr.linkedin.com/pub/laurent-h/25/36b/a86/> > -- Muqtafi Akhmad Software Engineer Traveloka
