[
https://issues.apache.org/jira/browse/OOZIE-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15347070#comment-15347070
]
Satish Subhashrao Saley edited comment on OOZIE-2389 at 6/23/16 8:37 PM:
-------------------------------------------------------------------------
Hi [~rkanter], [~pavan kumar]
We also faced same issue when a spark-examples.jar contained hadoop 2.6
classes. We solved it by setting
{code}
<property>
<name>oozie.launcher.mapreduce.task.classpath.user.precedence</name>
<value>false</value>
</property>
<property >
<name>oozie.launcher.mapreduce.user.classpath.first</name>
<value>false</value>
</property>
{code}
It has been a while since
[OOZIE-2071|https://issues.apache.org/jira/browse/OOZIE-2071], I see that Pavan
faced some issues and added these properties to solve those. I know it would be
hard to recollect, but really appreciate any inputs on this.
https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java#L106-L111
{code}
if (launcherJobConf.get("oozie.launcher." + TASK_USER_PRECEDENCE) ==
null) {
launcherJobConf.set(TASK_USER_PRECEDENCE, "true");
}
if (launcherJobConf.get("oozie.launcher." +
TASK_USER_CLASSPATH_PRECEDENCE) == null) {
launcherJobConf.set(TASK_USER_CLASSPATH_PRECEDENCE, "true");
}
{code}
I also tried the spark example we ship with oozie, it works without these
settings.
was (Author: satishsaley):
Hi [~rkanter],
We also faced same issue when a spark-examples.jar contained hadoop 2.6
classes. We solved it by setting
{code}
<property>
<name>oozie.launcher.mapreduce.task.classpath.user.precedence</name>
<value>false</value>
</property>
<property >
<name>oozie.launcher.mapreduce.user.classpath.first</name>
<value>false</value>
</property>
{code}
Is there any specific reason to set these to true by default?
https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java#L106-L111
{code}
if (launcherJobConf.get("oozie.launcher." + TASK_USER_PRECEDENCE) ==
null) {
launcherJobConf.set(TASK_USER_PRECEDENCE, "true");
}
if (launcherJobConf.get("oozie.launcher." +
TASK_USER_CLASSPATH_PRECEDENCE) == null) {
launcherJobConf.set(TASK_USER_CLASSPATH_PRECEDENCE, "true");
}
{code}
> Spark action failed with error starting MRAppMaster
> ---------------------------------------------------
>
> Key: OOZIE-2389
> URL: https://issues.apache.org/jira/browse/OOZIE-2389
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 4.2.0
> Reporter: Hunt Tang
> Priority: Blocker
> Attachments: application_1445333963855_0050.log
>
>
> I used spark-examples-1.5.1-hadoop2.6.0.jar to generate a test spark action
> in Oozie, it succeeded sometimes, but in most of the times, it failed.
> I checked the Hadoop job history, and it said
> {quote}
> ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting
> MRAppMaster
> java.lang.NoSuchMethodError:
> org.apache.hadoop.mapred.TaskLog.createLogSyncer()Ljava/util/concurrent/ScheduledExecutorService;
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.<init>(MRAppMaster.java:244)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.<init>(MRAppMaster.java:227)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1412)
> 2015-10-22 17:01:56,203 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1
> {quote}
> I'm using Hadoop 2.6.0, Spark 1.5.1, Oozie 4.2.0, and I put
> spark-assembly-1.5.1-hadoop2.6.0.jar together with the example jar in the
> lib, and my workflow.xml is as followings.
> {code:xml}
> <action name="sparkTest">
> <spark xmlns="uri:oozie:spark-action:0.1">
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <master>yarn-cluster</master>
> <name>SparkPi</name>
> <class>org.apache.spark.examples.JavaSparkPi</class>
>
> <jar>${nameNode}/sparkOozieTest/workflow/lib/spark-examples-1.5.1-hadoop2.6.0.jar</jar>
> <arg>10</arg>
> </spark>
> <ok to="end"/>
> <error to="fail"/>
> </action>
> {code}
> As it indeed succeeded sometimes, I don't think there is anything wrong with
> my configs. So is it an Oozie bug or is there anything I missed?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
