Satish Subhashrao Saley created OOZIE-2787:
----------------------------------------------
Summary: Oozie distributes application jar twice making the spark
job fail
Key: OOZIE-2787
URL: https://issues.apache.org/jira/browse/OOZIE-2787
Project: Oozie
Issue Type: Bug
Reporter: Satish Subhashrao Saley
Assignee: Satish Subhashrao Saley
Oozie adds the application jar to the list of files to be uploaded to
distributed cache. Since this gets added twice, the job fails. This is observed
from spark 2.1.0 which introduces a check for same file and fails the job.
{code}
--master
yarn
--deploy-mode
cluster
--name
oozieSparkStarter
--class
ScalaWordCount
--queue
default
--conf
spark.executor.extraClassPath=$PWD/*
--conf
spark.driver.extraClassPath=$PWD/*
--conf
spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
--conf
spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
--conf
spark.yarn.security.tokens.hive.enabled=false
--conf
spark.yarn.security.tokens.hbase.enabled=false
--files
hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
--properties-file
spark-defaults.conf
--verbose
spark-example.jar
samplefile.txt
output
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
