What version of Spark is it? Why not take advantage of the Spark action directly? I highly recommend asking vendor specific questions in the associated forums.
On Feb 16, 2018 4:54 AM, "Malathi Thangarajan" <[email protected]> wrote: > Hi, > > I have a oozie shell action that executes a spark program. The action works > fine for small data. However when the dataset becomes large the spark > program runs longer than 2 hours. But in the oozie web console, I see that > the job failed with code JA017 after an hour. > > This is the error message from oozie web console: > > > *JA017: Could not lookup launched hadoop Job ID [job_1515381588900_7387] > which was associated with action > [0000751-180104034149429-oozie-oozi-W@taste-vector-affinity]. Failing > this > action!* > But actually the job runs in YARN and completes successfully after 2 hours. > So why does oozie fails the shell action? Could there be any configuration > that has gone wrong? > > *version deatils:* > Ambari - Version 2.4.0.4 > oozie - 4.2.0.2.5 > HDP-2.5.6.2 > > > [image: Crayon Data] > > [image: Twitter] <https://twitter.com/CrayonBigData> [image: Facebook] > <https://www.facebook.com/crayondata> [image: LinkedIn] > <http://www.linkedin.com/company/crayon-data> [image: Youtube] > <http://www.youtube.com/user/crayonbigdata> [image: Youtube] > <http://bigdata-madesimple.com/> > > > Malathi T / Software Developer > +91 9677211473 > > Crayon Data > www.crayondata.com > Top 5 in IBM's Watson Mobile Developer Challenge 2014 > <http://www.ibm.com/smarterplanet/us/en/ibmwatson/ > challenge.html/?utm_source=email> > Top 3 at Orange Fab 2015 <http://orangefab.asia/en/?utm_source=email>, > Tokyo > Finalist at Code_n 2014 @ CEBIT > <https://www.code-n.org/about/?utm_source=email>, Hanover >
