Attila, Using spark-submit this program finishes fine, so I don't there is a problem with the spark job itself. I am either not specifying the parameters in Oozie through Hue properly or there is an issues with Hue and/or Oozie.
Peter, Outside the yarn logs all I have are these oozie specific logs through hue: 2016-08-31 22:07:02,044 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[mint-cm.lax.adconion.com] USER[kpeng] GROUP[-] TOKEN[] APP[Spark_Action_Test] JOB[0003038-160817212356434-oozie-oozi-W] ACTION[0003038-160817212356434-oozie-oozi-W@:start:] Start action [0003038-160817212356434-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2016-08-31 22:07:02,045 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[mint-cm.lax.adconion.com] USER[kpeng] GROUP[-] TOKEN[] APP[Spark_Action_Test] JOB[0003038-160817212356434-oozie-oozi-W] ACTION[0003038-160817212356434-oozie-oozi-W@:start:] [***0003038-160817212356434-oozie-oozi-W@:start:***]Action status=DONE 2016-08-31 22:07:02,045 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[mint-cm.lax.adconion.com] USER[kpeng] GROUP[-] TOKEN[] APP[Spark_Action_Test] JOB[0003038-160817212356434-oozie-oozi-W] ACTION[0003038-160817212356434-oozie-oozi-W@:start:] [***0003038-160817212356434-oozie-oozi-W@:start:***]Action updated in DB! 2016-08-31 22:07:02,149 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[mint-cm.lax.adconion.com] USER[kpeng] GROUP[-] TOKEN[] APP[Spark_Action_Test] JOB[0003038-160817212356434-oozie-oozi-W] ACTION[0003038-160817212356434-oozie-oozi-W@spark-b023] Start action [0003038-160817212356434-oozie-oozi-W@spark-b023] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2016-08-31 22:07:09,919 INFO org.apache.oozie.action.hadoop.SparkActionExecutor: SERVER[mint-cm.lax.adconion.com] USER[kpeng] GROUP[-] TOKEN[] APP[Spark_Action_Test] JOB[0003038-160817212356434-oozie-oozi-W] ACTION[0003038-160817212356434-oozie-oozi-W@spark-b023] checking action, hadoop job ID [job_1470859874943_40237] status [RUNNING] 2016-08-31 22:07:09,921 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[mint-cm.lax.adconion.com] USER[kpeng] GROUP[-] TOKEN[] APP[Spark_Action_Test] JOB[0003038-160817212356434-oozie-oozi-W] ACTION[0003038-160817212356434-oozie-oozi-W@spark-b023] [***0003038-160817212356434-oozie-oozi-W@spark-b023***]Action status=RUNNING 2016-08-31 22:07:09,921 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[mint-cm.lax.adconion.com] USER[kpeng] GROUP[-] TOKEN[] APP[Spark_Action_Test] JOB[0003038-160817212356434-oozie-oozi-W] ACTION[0003038-160817212356434-oozie-oozi-W@spark-b023] [***0003038-160817212356434-oozie-oozi-W@spark-b023***]Action updated in DB! 2016-08-31 22:07:22,386 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[mint-cm.lax.adconion.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0003038-160817212356434-oozie-oozi-W] ACTION[0003038-160817212356434-oozie-oozi-W@spark-b023] callback for action [0003038-160817212356434-oozie-oozi-W@spark-b023] 2016-08-31 22:07:22,722 INFO org.apache.oozie.action.hadoop.SparkActionExecutor: SERVER[mint-cm.lax.adconion.com] USER[kpeng] GROUP[-] TOKEN[] APP[Spark_Action_Test] JOB[0003038-160817212356434-oozie-oozi-W] ACTION[0003038-160817212356434-oozie-oozi-W@spark-b023] action completed, external ID [job_1470859874943_40237] 2016-08-31 22:07:22,725 WARN org.apache.oozie.action.hadoop.SparkActionExecutor: SERVER[mint-cm.lax.adconion.com] USER[kpeng] GROUP[-] TOKEN[] APP[Spark_Action_Test] JOB[0003038-160817212356434-oozie-oozi-W] ACTION[0003038-160817212356434-oozie-oozi-W@spark-b023] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SparkMain], exit code [101] 2016-08-31 22:07:22,913 INFO org.apache.oozie.command.wf.ActionEndXCommand: SERVER[mint-cm.lax.adconion.com] USER[kpeng] GROUP[-] TOKEN[] APP[Spark_Action_Test] JOB[0003038-160817212356434-oozie-oozi-W] ACTION[0003038-160817212356434-oozie-oozi-W@spark-b023] ERROR is considered as FAILED for SLA 2016-08-31 22:07:22,977 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[mint-cm.lax.adconion.com] USER[kpeng] GROUP[-] TOKEN[] APP[Spark_Action_Test] JOB[0003038-160817212356434-oozie-oozi-W] ACTION[0003038-160817212356434-oozie-oozi-W@Kill] Start action [0003038-160817212356434-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2016-08-31 22:07:22,978 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[mint-cm.lax.adconion.com] USER[kpeng] GROUP[-] TOKEN[] APP[Spark_Action_Test] JOB[0003038-160817212356434-oozie-oozi-W] ACTION[0003038-160817212356434-oozie-oozi-W@Kill] [***0003038-160817212356434-oozie-oozi-W@Kill***]Action status=DONE 2016-08-31 22:07:22,978 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[mint-cm.lax.adconion.com] USER[kpeng] GROUP[-] TOKEN[] APP[Spark_Action_Test] JOB[0003038-160817212356434-oozie-oozi-W] ACTION[0003038-160817212356434-oozie-oozi-W@Kill] [***0003038-160817212356434-oozie-oozi-W@Kill***]Action updated in DB! Thanks, KP On Wed, Aug 31, 2016 at 2:49 PM, Attila Sasvari <[email protected]> wrote: > Hi Kevin, > > Have you tried to execute your spark application locally without Oozie and > it worked? I believe so. > > Looking at the error code and SparkSubmit, it turns out 101 means class not > found, see > https://github.com/apache/spark/blob/master/core/src/ > main/scala/org/apache/spark/deploy/SparkSubmit.scala#L87 > . > > I saw you mentioned external jars in your spark application (parquet > related). > - Prior to execution, try to upload dependency jar files to a directory on > HDFS an refer to that directory in *job.properties* using > oozie.libpath=${nameNode}/path/to/DEPENDENCY_JARS_DIR/ > > To hunt down this kind of issues, the following could also help: > - pass -verbose:class to your driver (e.g. --conf > spark.driver.extraJavaOptions="-verbose:class", --conf spark.executor. > extraJavaOptions="-verbose:class") > > Hope this helps, > - Attila > > This might be related also > https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/ > ClassNotFoundException-org-apache-htrace-Trace-exception- > in/m-p/29253/highlight/true#M915 > ? > > > On Wed, Aug 31, 2016 at 10:50 PM, Peter Cseh <[email protected]> > wrote: > > > Hi, > > Could you find the logs for the Spark job that had been running? > > This log has only this in info: > > > > >>> Invoking Spark class now >>> > > > > Intercepting System.exit(101) > > > > <<< Invocation of Main class completed <<< > > > > Failing Oozie Launcher, Main class > > [org.apache.oozie.action.hadoop.SparkMain], exit code [101] > > > > Oozie Launcher failed, finishing Hadoop job gracefully > > > > Which is not that useful. > > > > Thank you, > > > > Gp > > > > > > On Wed, Aug 31, 2016 at 10:46 PM, Kevin Peng <[email protected]> wrote: > > > > > Satish, > > > > > > I have attached the entire logs (which I believe has the stack trace in > > > this email). > > > > > > Thanks, > > > > > > KP > > > > > > On Wed, Aug 31, 2016 at 12:48 PM, satish saley < > [email protected]> > > > wrote: > > > > > >> Hi, > > >> Can you share the stacktrace you see in the actual spark job (the > child > > >> job)? > > >> > > >> On Wed, Aug 31, 2016 at 10:40 AM, Kevin Peng <[email protected]> > wrote: > > >> > > >> > Hi All, > > >> > > > >> > I am trying to get Spark Action to work in oozie. I am currently > > using > > >> CDH > > >> > 5.7.2 and Hue 3.9 that is bundled with the CDH version to create a > > test > > >> > workflow. I create a spark jar that had all it's needed > dependencies > > >> > compiled in the jar. All this jar does is read a parquet file from > > hdfs > > >> > and write back to a different directory in hdfs. When running this > > >> > workflow I am getting a 101 error: > > >> > > > >> > >>> Invoking Spark class now >>> > > >> > > > >> > > > >> > Intercepting System.exit(101) > > >> > > > >> > > > >> > <<< Invocation of Main class completed <<< > > >> > > > >> > > > >> > Failing Oozie Launcher, Main class > > >> > [org.apache.oozie.action.hadoop.SparkMain], exit code [101] > > >> > > > >> > > > >> > Oozie Launcher failed, finishing Hadoop job gracefully > > >> > > > >> > > > >> > Oozie Launcher, uploading action data to HDFS sequence file: > > >> > hdfs://nameservice1/user/kpeng/oozie-oozi/0002825- > > >> > 160817212356434-oozie-oozi-W/spark-b023--spark/action-data.seq > > >> > > > >> > > > >> > Oozie Launcher ends > > >> > > > >> > > > >> > Any help would be appreciated. > > >> > > > >> > > > >> > Thanks, > > >> > > > >> > > > >> > KP > > >> > > > >> > > > > > > > > > > > > -- > > Peter Cseh > > Software Engineer > > <http://www.cloudera.com> > > >
