narendra maru created SPARK-19992:
-------------------------------------
Summary: spark-submit on deployment-mode cluster
Key: SPARK-19992
URL: https://issues.apache.org/jira/browse/SPARK-19992
Project: Spark
Issue Type: Bug
Components: Spark Submit
Affects Versions: 2.0.2
Environment: spark version 2.0.2
hadoop version 2.6.0
Reporter: narendra maru
spark version 2.0.2
hadoop version 2.6.0
spark -submit command
"spark-submit --class spark.mongohadoop.testing3 --master yarn --deploy-mode
cluster --jars /home/ec2-user/jars/hgmongonew.jar,
/home/ec2-user/jars/mongo-hadoop-spark-2.0.1.jar"
after adding following in
1 Spark-default.conf
spark.executor.extraJavaOptions -Dconfig.fuction.conf
spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/*
spark.eventLog.dir=hdfs://localhost:9000/user/spark/applicationHistory
spark.eventLog.enabled=true
2yarn-site.xml
<property>
<name>yarn.application.classpath</name>
<value>
/usr/local/hadoop-2.6.0/etc/hadoop,
/usr/local/hadoop-2.6.0/,
/usr/local/hadoop-2.6.0/lib/,
/usr/local/hadoop-2.6.0/share/hadoop/common/,
/usr/local/hadoop-2.6.0/share/hadoop/common/lib/
/usr/local/hadoop-2.6.0/share/hadoop/hdfs/,
/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/,
/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/,
/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/,
/usr/local/hadoop-2.6.0/share/hadoop/tools/lib/,
/usr/local/hadoop-2.6.0/share/hadoop/yarn/,
/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/*,
/usr/local/spark-2.0.2-bin-hadoop2.6/jars/spark-yarn_2.11-2.0.2.jar
</value>
</property>
Error on log:-
Error: Could not find or load main class
org.apache.spark.deploy.yarn.ApplicationMaster
Error on terminal:-
diagnostics: Application application_1489673977198_0002 failed 2 times due to
AM Container for appattempt_1489673977198_0002_000002 exited with exitCode: 1
For more detailed output, check application tracking
page:http://bdg-hdp-sparkmaster:8088/proxy/application_1489673977198_0002/Then,
click on links to logs of each attempt.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
