dear all, Currently I am trying to submit Spark application using spark-submit (with Spark application copy exists in all YARN nodes) by writing it down in Oozie shell action, here is the workflow.xml : http://pastebin.com/kHVtmNex
When submitting this job, I kept getting Error code : JA018 Error message : Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1] What do those messages mean? Thank you, On Mon, Jun 1, 2015 at 7:45 PM, Jaydeep Vishwakarma < [email protected]> wrote: > I am not sure what to look. There is a detail log hyperlink option in > Launcher log on ui. You can try to search the exception over there. > > You can add any library as part of sharelib. You just need to push the lib > directory inside the sharelib directory of hdfs and update the sharelibs by > "oozie admin -sharelibupdate" command. > > Spark libs have to present in hdfs, It is independent to modes. > > > On Mon, Jun 1, 2015 at 3:24 PM, Muqtafi Akhmad <[email protected]> > wrote: > > > Hello Jaydeep, > > Logs from > > > > > oozie job -info ... > > > > and > > > > > yarn logs -applicationId ... > > > > give the same explanation like the one I've got from Oozie web app. Is > > there any meaning for > > > > > Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [1] > > > > error? > > > > It is good that Spark action will be supported by Oozie, but given our > > current Oozie version, can developer add Spark (or other 'common') > > libraries to Oozie's shared libs? When you mention > > > > > It is only working as local mode. > > > > does it mean that we don't need to copy Spark libs and apps to HDFS? > > Thank you, > > > > On Mon, Jun 1, 2015 at 3:17 PM, Jaydeep Vishwakarma < > > [email protected]> wrote: > > > > > You can get more details for this error on Launcher logs by accessing > the > > > console url. > > > Spark action has been recently introduced in oozie. It will come in > > 4.2.0 , > > > But there are some issue with spark action, It is only working as local > > > mode. Here is the jira for same. > > > https://issues.apache.org/jira/browse/OOZIE-2143. > > > > > > Regards, > > > Jaydeep > > > > > > > > > > > > > > > On Mon, Jun 1, 2015 at 12:43 PM, Muqtafi Akhmad <[email protected] > > > > > wrote: > > > > > > > Hello everyone, > > > > I'm new to Oozie, currently I'm trying to use Oozie to manage Spark > > > > applications. I've tried to submit Spark application as Java action ( > > as > > > > described in > > > > > > > > > > > > > > http://stackoverflow.com/questions/29233487/launching-a-spark-program-using-oozie-workflow > > > > ) and the action return error > > > > > > > > > Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [1] > > > > > > > > My questions are > > > > 1. How can I see / access the complete stack trace to know what > exactly > > > > happened in my application execution? > > > > 2. By default, Oozie shared lib directory provides libraries for > Hive, > > > Pig, > > > > and Map-Reduce. If I want to run Spark job on Oozie, it might be > better > > > to > > > > add Spark lib jars to Oozie's shared lib instead of copy them to > app's > > > lib > > > > directory. How can I add Spark lib jars (including spark-core and its > > > > dependencies) to Oozie's shared lib? > > > > > > > > Thank you, > > > > > > > > -- > > > > Muqtafi Akhmad > > > > Software Engineer > > > > Traveloka > > > > > > > > > > -- > > > _____________________________________________________________ > > > The information contained in this communication is intended solely for > > the > > > use of the individual or entity to whom it is addressed and others > > > authorized to receive it. It may contain confidential or legally > > privileged > > > information. If you are not the intended recipient you are hereby > > notified > > > that any disclosure, copying, distribution or taking any action in > > reliance > > > on the contents of this information is strictly prohibited and may be > > > unlawful. If you have received this communication in error, please > notify > > > us immediately by responding to this email and then delete it from your > > > system. The firm is neither liable for the proper and complete > > transmission > > > of the information contained in this communication nor for any delay in > > its > > > receipt. > > > > > > > > > > > -- > > Muqtafi Akhmad > > Software Engineer > > Traveloka > > > > -- > _____________________________________________________________ > The information contained in this communication is intended solely for the > use of the individual or entity to whom it is addressed and others > authorized to receive it. It may contain confidential or legally privileged > information. If you are not the intended recipient you are hereby notified > that any disclosure, copying, distribution or taking any action in reliance > on the contents of this information is strictly prohibited and may be > unlawful. If you have received this communication in error, please notify > us immediately by responding to this email and then delete it from your > system. The firm is neither liable for the proper and complete transmission > of the information contained in this communication nor for any delay in its > receipt. > -- Muqtafi Akhmad Software Engineer Traveloka
