Hunt Tang created OOZIE-2389:
--------------------------------
Summary: Spark action failed with error starting MRAppMaster
Key: OOZIE-2389
URL: https://issues.apache.org/jira/browse/OOZIE-2389
Project: Oozie
Issue Type: Bug
Affects Versions: 4.2.0
Reporter: Hunt Tang
Priority: Blocker
I used spark-examples-1.5.1-hadoop2.6.0.jar to generate a test spark action in
Oozie, it succeeded sometimes, but in most of the times, it failed.
I checked the Hadoop job history, and it said
{quote}
ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting
MRAppMaster
java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.TaskLog.createLogSyncer()Ljava/util/concurrent/ScheduledExecutorService;
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.<init>(MRAppMaster.java:244)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.<init>(MRAppMaster.java:227)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1412)
2015-10-22 17:01:56,203 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
with status 1
{quote}
I'm using Hadoop 2.6.0, Spark 1.5.1, Oozie 4.2.0, and I put
spark-assembly-1.5.1-hadoop2.6.0.jar together with the example jar in the lib,
and my workflow.xml is as followings.
{code:xml}
<action name="sparkTest">
<spark xmlns="uri:oozie:spark-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<master>yarn-cluster</master>
<name>SparkPi</name>
<class>org.apache.spark.examples.JavaSparkPi</class>
<jar>${nameNode}/sparkOozieTest/workflow/lib/spark-examples-1.5.1-hadoop2.6.0.jar</jar>
<arg>10</arg>
</spark>
<ok to="end"/>
<error to="fail"/>
</action>
{code}
As it indeed succeeded sometimes, I don't think there is anything wrong with my
configs. So is it an Oozie bug or is there anything I missed?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
