Failed to run Spark action on YARN through Oozie workflow

[Tao Xiao]

I am using CDH 5.5.1 (Oozie 4.1 and Spark 1.5.1), and I'd like to run a
Spark action on YARN and submit it through Oozie workflow.


The Spark app is a Pi program contained in the example jar provided by CDH
("spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar") and the main
class is "org.apache.spark.examples.SparkPi".


My oozie job's directory is as follows:

- spark/

  - job.properties

  - workflow.xml

  - lib/

     - spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar


“job.properties” is as following:

NameNode=hdfs://nameservice1

RM=ecs2.njzd.com:8032

Master=yarn-client

oozie.use.system.libpath=true

oozie.wf.application.path=${NameNode}/user/tao/oozie/examples/apps/xt/spark


“workflow.xml” is as following:

<workflow-app name="WF-Spark" xmlns="uri:oozie:workflow:0.5">

  <start to='spark-node'/>

  <action name='spark-node'>

    <spark xmlns="uri:oozie:spark-action:0.1">

      <job-tracker>${RM}</job-tracker>

      <name-node>${NameNode}</name-node>

      <master>${Master}</master>

      <name>Spark-Job-Pi</name>

      <class>org.apache.spark.examples.SparkPi</class>

      <jar>
${NameNode}/user/tao/oozie/examples/apps/xt/spark/lib/spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
</jar>

    </spark>

    <ok to="end-node"/>

    <error to="fail-node"/>

  </action>

  <kill name="fail-node">

    <message>Spark Job Failed!</message>

  </kill>

  <end name="end-node"/>

</workflow-app>



When I submitted the workflow job, the launcher completed but the Spark
action failed with the following error:


Application application_1451571118974_0127 failed 2 times due to AM
Container for appattempt_1451571118974_0127_000002 exited with exitCode:
-1000

...

Diagnostics: Resource hdfs://
ecs3.njzd.com:8020/user/root/.sparkStaging/application_1451571118974_0127/spark-yarn_2.10-1.5.0-cdh5.5.1.jar
changed on src filesystem (expected 1452757035811, was 1452757065894

java.io.IOException: Resource hdfs://
ecs3.njzd.com:8020/user/root/.sparkStaging/application_1451571118974_0127/spark-yarn_2.10-1.5.0-cdh5.5.1.jar
changed on src filesystem (expected 1452757035811, was 1452757065894

at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)

at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)

at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)

at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)

at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)

at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Failing this attempt. Failing the application.


I don't know why "Resource hdfs://....spark-yarn_2.10-1.5.0-cdh5.5.1.jar
changed on src filesystem " is reported.

I'd like to know the reason and how to resolve it.


Thanks.