Maybe add the spark-site.xml (if this file exists, I am not a spark expert!) To the container!
Le vendredi 5 juin 2015, Laurent H <[email protected]> a écrit : > Hi, > > I can give you some ideas. Hope to be helpful : > > Add spark.job.queuename or smthg like this in your spark program or in > oozie configuration node (like mapred.job.queuename) you can add some spark > libs in your shell action with the # to put all of them in the same > container. > > Good luck! > > Le jeudi 4 juin 2015, Muqtafi Akhmad <[email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>> a écrit : > >> dear all, >> Currently I am trying to submit Spark application using spark-submit (with >> Spark application copy exists in all YARN nodes) by writing it down in >> Oozie shell action, here is the workflow.xml : >> http://pastebin.com/kHVtmNex >> >> When submitting this job, I kept getting >> Error code : JA018 >> Error message : Main class [org.apache.oozie.action.hadoop.ShellMain], >> exit >> code [1] >> >> What do those messages mean? >> Thank you, >> >> >> >> On Mon, Jun 1, 2015 at 7:45 PM, Jaydeep Vishwakarma < >> [email protected]> wrote: >> >> > I am not sure what to look. There is a detail log hyperlink option in >> > Launcher log on ui. You can try to search the exception over there. >> > >> > You can add any library as part of sharelib. You just need to push the >> lib >> > directory inside the sharelib directory of hdfs and update the >> sharelibs by >> > "oozie admin -sharelibupdate" command. >> > >> > Spark libs have to present in hdfs, It is independent to modes. >> > >> > >> > On Mon, Jun 1, 2015 at 3:24 PM, Muqtafi Akhmad <[email protected]> >> > wrote: >> > >> > > Hello Jaydeep, >> > > Logs from >> > > >> > > > oozie job -info ... >> > > >> > > and >> > > >> > > > yarn logs -applicationId ... >> > > >> > > give the same explanation like the one I've got from Oozie web app. Is >> > > there any meaning for >> > > >> > > > Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [1] >> > > >> > > error? >> > > >> > > It is good that Spark action will be supported by Oozie, but given our >> > > current Oozie version, can developer add Spark (or other 'common') >> > > libraries to Oozie's shared libs? When you mention >> > > >> > > > It is only working as local mode. >> > > >> > > does it mean that we don't need to copy Spark libs and apps to HDFS? >> > > Thank you, >> > > >> > > On Mon, Jun 1, 2015 at 3:17 PM, Jaydeep Vishwakarma < >> > > [email protected]> wrote: >> > > >> > > > You can get more details for this error on Launcher logs by >> accessing >> > the >> > > > console url. >> > > > Spark action has been recently introduced in oozie. It will come in >> > > 4.2.0 , >> > > > But there are some issue with spark action, It is only working as >> local >> > > > mode. Here is the jira for same. >> > > > https://issues.apache.org/jira/browse/OOZIE-2143. >> > > > >> > > > Regards, >> > > > Jaydeep >> > > > >> > > > >> > > > >> > > > >> > > > On Mon, Jun 1, 2015 at 12:43 PM, Muqtafi Akhmad < >> [email protected] >> > > >> > > > wrote: >> > > > >> > > > > Hello everyone, >> > > > > I'm new to Oozie, currently I'm trying to use Oozie to manage >> Spark >> > > > > applications. I've tried to submit Spark application as Java >> action ( >> > > as >> > > > > described in >> > > > > >> > > > > >> > > > >> > > >> > >> http://stackoverflow.com/questions/29233487/launching-a-spark-program-using-oozie-workflow >> > > > > ) and the action return error >> > > > > >> > > > > > Main class [org.apache.oozie.action.hadoop.JavaMain], exit code >> [1] >> > > > > >> > > > > My questions are >> > > > > 1. How can I see / access the complete stack trace to know what >> > exactly >> > > > > happened in my application execution? >> > > > > 2. By default, Oozie shared lib directory provides libraries for >> > Hive, >> > > > Pig, >> > > > > and Map-Reduce. If I want to run Spark job on Oozie, it might be >> > better >> > > > to >> > > > > add Spark lib jars to Oozie's shared lib instead of copy them to >> > app's >> > > > lib >> > > > > directory. How can I add Spark lib jars (including spark-core and >> its >> > > > > dependencies) to Oozie's shared lib? >> > > > > >> > > > > Thank you, >> > > > > >> > > > > -- >> > > > > Muqtafi Akhmad >> > > > > Software Engineer >> > > > > Traveloka >> > > > > >> > > > >> > > > -- >> > > > _____________________________________________________________ >> > > > The information contained in this communication is intended solely >> for >> > > the >> > > > use of the individual or entity to whom it is addressed and others >> > > > authorized to receive it. It may contain confidential or legally >> > > privileged >> > > > information. If you are not the intended recipient you are hereby >> > > notified >> > > > that any disclosure, copying, distribution or taking any action in >> > > reliance >> > > > on the contents of this information is strictly prohibited and may >> be >> > > > unlawful. If you have received this communication in error, please >> > notify >> > > > us immediately by responding to this email and then delete it from >> your >> > > > system. The firm is neither liable for the proper and complete >> > > transmission >> > > > of the information contained in this communication nor for any >> delay in >> > > its >> > > > receipt. >> > > > >> > > >> > > >> > > >> > > -- >> > > Muqtafi Akhmad >> > > Software Engineer >> > > Traveloka >> > > >> > >> > -- >> > _____________________________________________________________ >> > The information contained in this communication is intended solely for >> the >> > use of the individual or entity to whom it is addressed and others >> > authorized to receive it. It may contain confidential or legally >> privileged >> > information. If you are not the intended recipient you are hereby >> notified >> > that any disclosure, copying, distribution or taking any action in >> reliance >> > on the contents of this information is strictly prohibited and may be >> > unlawful. If you have received this communication in error, please >> notify >> > us immediately by responding to this email and then delete it from your >> > system. The firm is neither liable for the proper and complete >> transmission >> > of the information contained in this communication nor for any delay in >> its >> > receipt. >> > >> >> >> >> -- >> Muqtafi Akhmad >> Software Engineer >> Traveloka >> > > > -- > -- > Laurent HATIER - Consultant Big Data & Business Intelligence chez CapGemini > fr.linkedin.com/pub/laurent-hatier/25/36b/a86/ > <http://fr.linkedin.com/pub/laurent-h/25/36b/a86/> > > -- -- Laurent HATIER - Consultant Big Data & Business Intelligence chez CapGemini fr.linkedin.com/pub/laurent-hatier/25/36b/a86/ <http://fr.linkedin.com/pub/laurent-h/25/36b/a86/>
