[jira] [Commented] (SPARK-19992) spark-submit on deployment-mode cluster
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939867#comment-15939867 ] Saisai Shao commented on SPARK-19992: - Sorry I cannot give you valid suggestions without knowing your actual environment. Basically running Spark on yarn you don't have to configure anything except HADOOP_CONF_DIR specified in spark-env.sh. Other than this default configurations should be enough. You could also send your problem to mail list, I think there will be more users who met the same problem before. Here JIRA is mainly used to track dev work of Spark, not for questions. > spark-submit on deployment-mode cluster > --- > > Key: SPARK-19992 > URL: https://issues.apache.org/jira/browse/SPARK-19992 > Project: Spark > Issue Type: Bug > Components: Spark Submit >Affects Versions: 2.0.2 > Environment: spark version 2.0.2 > hadoop version 2.6.0 >Reporter: narendra maru > > spark version 2.0.2 > hadoop version 2.6.0 > spark -submit command > "spark-submit --class spark.mongohadoop.testing3 --master yarn --deploy-mode > cluster --jars /home/ec2-user/jars/hgmongonew.jar, > /home/ec2-user/jars/mongo-hadoop-spark-2.0.1.jar" > after adding following in > 1 Spark-default.conf > spark.executor.extraJavaOptions -Dconfig.fuction.conf > spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/* > spark.eventLog.dir=hdfs://localhost:9000/user/spark/applicationHistory > spark.eventLog.enabled=true > 2yarn-site.xml > > yarn.application.classpath > > /usr/local/hadoop-2.6.0/etc/hadoop, > /usr/local/hadoop-2.6.0/, > /usr/local/hadoop-2.6.0/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/common/, > /usr/local/hadoop-2.6.0/share/hadoop/common/lib/ > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/, > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/tools/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/*, > /usr/local/spark-2.0.2-bin-hadoop2.6/jars/spark-yarn_2.11-2.0.2.jar > > > Error on log:- > Error: Could not find or load main class > org.apache.spark.deploy.yarn.ApplicationMaster > Error on terminal:- > diagnostics: Application application_1489673977198_0002 failed 2 times due to > AM Container for appattempt_1489673977198_0002_02 exited with exitCode: 1 > For more detailed output, check application tracking > page:http://bdg-hdp-sparkmaster:8088/proxy/application_1489673977198_0002/Then, > click on links to logs of each attempt. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19992) spark-submit on deployment-mode cluster
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939857#comment-15939857 ] narendra maru commented on SPARK-19992: --- Thanks sean and saisai for your rlyy I followed the same instruction of yours but not getting satisfactory result it will be great if you can help me out with some better suggestions > spark-submit on deployment-mode cluster > --- > > Key: SPARK-19992 > URL: https://issues.apache.org/jira/browse/SPARK-19992 > Project: Spark > Issue Type: Bug > Components: Spark Submit >Affects Versions: 2.0.2 > Environment: spark version 2.0.2 > hadoop version 2.6.0 >Reporter: narendra maru > > spark version 2.0.2 > hadoop version 2.6.0 > spark -submit command > "spark-submit --class spark.mongohadoop.testing3 --master yarn --deploy-mode > cluster --jars /home/ec2-user/jars/hgmongonew.jar, > /home/ec2-user/jars/mongo-hadoop-spark-2.0.1.jar" > after adding following in > 1 Spark-default.conf > spark.executor.extraJavaOptions -Dconfig.fuction.conf > spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/* > spark.eventLog.dir=hdfs://localhost:9000/user/spark/applicationHistory > spark.eventLog.enabled=true > 2yarn-site.xml > > yarn.application.classpath > > /usr/local/hadoop-2.6.0/etc/hadoop, > /usr/local/hadoop-2.6.0/, > /usr/local/hadoop-2.6.0/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/common/, > /usr/local/hadoop-2.6.0/share/hadoop/common/lib/ > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/, > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/tools/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/*, > /usr/local/spark-2.0.2-bin-hadoop2.6/jars/spark-yarn_2.11-2.0.2.jar > > > Error on log:- > Error: Could not find or load main class > org.apache.spark.deploy.yarn.ApplicationMaster > Error on terminal:- > diagnostics: Application application_1489673977198_0002 failed 2 times due to > AM Container for appattempt_1489673977198_0002_02 exited with exitCode: 1 > For more detailed output, check application tracking > page:http://bdg-hdp-sparkmaster:8088/proxy/application_1489673977198_0002/Then, > click on links to logs of each attempt. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19992) spark-submit on deployment-mode cluster
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936241#comment-15936241 ] Saisai Shao commented on SPARK-19992: - Oh, I see. Check the code again, looks like "/*" cannot be worked with "local" schema, only hadoop support schema like hdfs, file could support glob path. > spark-submit on deployment-mode cluster > --- > > Key: SPARK-19992 > URL: https://issues.apache.org/jira/browse/SPARK-19992 > Project: Spark > Issue Type: Bug > Components: Spark Submit >Affects Versions: 2.0.2 > Environment: spark version 2.0.2 > hadoop version 2.6.0 >Reporter: narendra maru > > spark version 2.0.2 > hadoop version 2.6.0 > spark -submit command > "spark-submit --class spark.mongohadoop.testing3 --master yarn --deploy-mode > cluster --jars /home/ec2-user/jars/hgmongonew.jar, > /home/ec2-user/jars/mongo-hadoop-spark-2.0.1.jar" > after adding following in > 1 Spark-default.conf > spark.executor.extraJavaOptions -Dconfig.fuction.conf > spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/* > spark.eventLog.dir=hdfs://localhost:9000/user/spark/applicationHistory > spark.eventLog.enabled=true > 2yarn-site.xml > > yarn.application.classpath > > /usr/local/hadoop-2.6.0/etc/hadoop, > /usr/local/hadoop-2.6.0/, > /usr/local/hadoop-2.6.0/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/common/, > /usr/local/hadoop-2.6.0/share/hadoop/common/lib/ > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/, > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/tools/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/*, > /usr/local/spark-2.0.2-bin-hadoop2.6/jars/spark-yarn_2.11-2.0.2.jar > > > Error on log:- > Error: Could not find or load main class > org.apache.spark.deploy.yarn.ApplicationMaster > Error on terminal:- > diagnostics: Application application_1489673977198_0002 failed 2 times due to > AM Container for appattempt_1489673977198_0002_02 exited with exitCode: 1 > For more detailed output, check application tracking > page:http://bdg-hdp-sparkmaster:8088/proxy/application_1489673977198_0002/Then, > click on links to logs of each attempt. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19992) spark-submit on deployment-mode cluster
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929762#comment-15929762 ] narendra maru commented on SPARK-19992: --- yes I am having same yarn jars and spark_home directory on all the node at the same location > spark-submit on deployment-mode cluster > --- > > Key: SPARK-19992 > URL: https://issues.apache.org/jira/browse/SPARK-19992 > Project: Spark > Issue Type: Bug > Components: Spark Submit >Affects Versions: 2.0.2 > Environment: spark version 2.0.2 > hadoop version 2.6.0 >Reporter: narendra maru > > spark version 2.0.2 > hadoop version 2.6.0 > spark -submit command > "spark-submit --class spark.mongohadoop.testing3 --master yarn --deploy-mode > cluster --jars /home/ec2-user/jars/hgmongonew.jar, > /home/ec2-user/jars/mongo-hadoop-spark-2.0.1.jar" > after adding following in > 1 Spark-default.conf > spark.executor.extraJavaOptions -Dconfig.fuction.conf > spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/* > spark.eventLog.dir=hdfs://localhost:9000/user/spark/applicationHistory > spark.eventLog.enabled=true > 2yarn-site.xml > > yarn.application.classpath > > /usr/local/hadoop-2.6.0/etc/hadoop, > /usr/local/hadoop-2.6.0/, > /usr/local/hadoop-2.6.0/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/common/, > /usr/local/hadoop-2.6.0/share/hadoop/common/lib/ > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/, > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/tools/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/*, > /usr/local/spark-2.0.2-bin-hadoop2.6/jars/spark-yarn_2.11-2.0.2.jar > > > Error on log:- > Error: Could not find or load main class > org.apache.spark.deploy.yarn.ApplicationMaster > Error on terminal:- > diagnostics: Application application_1489673977198_0002 failed 2 times due to > AM Container for appattempt_1489673977198_0002_02 exited with exitCode: 1 > For more detailed output, check application tracking > page:http://bdg-hdp-sparkmaster:8088/proxy/application_1489673977198_0002/Then, > click on links to logs of each attempt. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19992) spark-submit on deployment-mode cluster
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929547#comment-15929547 ] Saisai Shao commented on SPARK-19992: - Looks like I guess wrong from the url you provided. If you're trying to use "spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/*", make sure these jars existed in every node. > spark-submit on deployment-mode cluster > --- > > Key: SPARK-19992 > URL: https://issues.apache.org/jira/browse/SPARK-19992 > Project: Spark > Issue Type: Bug > Components: Spark Submit >Affects Versions: 2.0.2 > Environment: spark version 2.0.2 > hadoop version 2.6.0 >Reporter: narendra maru > > spark version 2.0.2 > hadoop version 2.6.0 > spark -submit command > "spark-submit --class spark.mongohadoop.testing3 --master yarn --deploy-mode > cluster --jars /home/ec2-user/jars/hgmongonew.jar, > /home/ec2-user/jars/mongo-hadoop-spark-2.0.1.jar" > after adding following in > 1 Spark-default.conf > spark.executor.extraJavaOptions -Dconfig.fuction.conf > spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/* > spark.eventLog.dir=hdfs://localhost:9000/user/spark/applicationHistory > spark.eventLog.enabled=true > 2yarn-site.xml > > yarn.application.classpath > > /usr/local/hadoop-2.6.0/etc/hadoop, > /usr/local/hadoop-2.6.0/, > /usr/local/hadoop-2.6.0/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/common/, > /usr/local/hadoop-2.6.0/share/hadoop/common/lib/ > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/, > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/tools/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/*, > /usr/local/spark-2.0.2-bin-hadoop2.6/jars/spark-yarn_2.11-2.0.2.jar > > > Error on log:- > Error: Could not find or load main class > org.apache.spark.deploy.yarn.ApplicationMaster > Error on terminal:- > diagnostics: Application application_1489673977198_0002 failed 2 times due to > AM Container for appattempt_1489673977198_0002_02 exited with exitCode: 1 > For more detailed output, check application tracking > page:http://bdg-hdp-sparkmaster:8088/proxy/application_1489673977198_0002/Then, > click on links to logs of each attempt. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19992) spark-submit on deployment-mode cluster
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929546#comment-15929546 ] narendra maru commented on SPARK-19992: --- hi saisai i am working on plain Hadoop 2.6.0 so is there need for any other configuration > spark-submit on deployment-mode cluster > --- > > Key: SPARK-19992 > URL: https://issues.apache.org/jira/browse/SPARK-19992 > Project: Spark > Issue Type: Bug > Components: Spark Submit >Affects Versions: 2.0.2 > Environment: spark version 2.0.2 > hadoop version 2.6.0 >Reporter: narendra maru > > spark version 2.0.2 > hadoop version 2.6.0 > spark -submit command > "spark-submit --class spark.mongohadoop.testing3 --master yarn --deploy-mode > cluster --jars /home/ec2-user/jars/hgmongonew.jar, > /home/ec2-user/jars/mongo-hadoop-spark-2.0.1.jar" > after adding following in > 1 Spark-default.conf > spark.executor.extraJavaOptions -Dconfig.fuction.conf > spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/* > spark.eventLog.dir=hdfs://localhost:9000/user/spark/applicationHistory > spark.eventLog.enabled=true > 2yarn-site.xml > > yarn.application.classpath > > /usr/local/hadoop-2.6.0/etc/hadoop, > /usr/local/hadoop-2.6.0/, > /usr/local/hadoop-2.6.0/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/common/, > /usr/local/hadoop-2.6.0/share/hadoop/common/lib/ > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/, > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/tools/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/*, > /usr/local/spark-2.0.2-bin-hadoop2.6/jars/spark-yarn_2.11-2.0.2.jar > > > Error on log:- > Error: Could not find or load main class > org.apache.spark.deploy.yarn.ApplicationMaster > Error on terminal:- > diagnostics: Application application_1489673977198_0002 failed 2 times due to > AM Container for appattempt_1489673977198_0002_02 exited with exitCode: 1 > For more detailed output, check application tracking > page:http://bdg-hdp-sparkmaster:8088/proxy/application_1489673977198_0002/Then, > click on links to logs of each attempt. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19992) spark-submit on deployment-mode cluster
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929545#comment-15929545 ] narendra maru commented on SPARK-19992: --- thanks sean for a quick reply I am suffering from last 4-5 days for same error and not getting any way to resolve the same can you please help me with any suggestions for spark deployment on yarn multinode cluster as the same command is working on deployment mode client > spark-submit on deployment-mode cluster > --- > > Key: SPARK-19992 > URL: https://issues.apache.org/jira/browse/SPARK-19992 > Project: Spark > Issue Type: Bug > Components: Spark Submit >Affects Versions: 2.0.2 > Environment: spark version 2.0.2 > hadoop version 2.6.0 >Reporter: narendra maru > > spark version 2.0.2 > hadoop version 2.6.0 > spark -submit command > "spark-submit --class spark.mongohadoop.testing3 --master yarn --deploy-mode > cluster --jars /home/ec2-user/jars/hgmongonew.jar, > /home/ec2-user/jars/mongo-hadoop-spark-2.0.1.jar" > after adding following in > 1 Spark-default.conf > spark.executor.extraJavaOptions -Dconfig.fuction.conf > spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/* > spark.eventLog.dir=hdfs://localhost:9000/user/spark/applicationHistory > spark.eventLog.enabled=true > 2yarn-site.xml > > yarn.application.classpath > > /usr/local/hadoop-2.6.0/etc/hadoop, > /usr/local/hadoop-2.6.0/, > /usr/local/hadoop-2.6.0/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/common/, > /usr/local/hadoop-2.6.0/share/hadoop/common/lib/ > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/, > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/tools/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/*, > /usr/local/spark-2.0.2-bin-hadoop2.6/jars/spark-yarn_2.11-2.0.2.jar > > > Error on log:- > Error: Could not find or load main class > org.apache.spark.deploy.yarn.ApplicationMaster > Error on terminal:- > diagnostics: Application application_1489673977198_0002 failed 2 times due to > AM Container for appattempt_1489673977198_0002_02 exited with exitCode: 1 > For more detailed output, check application tracking > page:http://bdg-hdp-sparkmaster:8088/proxy/application_1489673977198_0002/Then, > click on links to logs of each attempt. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19992) spark-submit on deployment-mode cluster
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929544#comment-15929544 ] Saisai Shao commented on SPARK-19992: - Are you using HDP environment, if so I guess need to configuration hdp.version in Spark, you could google it. > spark-submit on deployment-mode cluster > --- > > Key: SPARK-19992 > URL: https://issues.apache.org/jira/browse/SPARK-19992 > Project: Spark > Issue Type: Bug > Components: Spark Submit >Affects Versions: 2.0.2 > Environment: spark version 2.0.2 > hadoop version 2.6.0 >Reporter: narendra maru > > spark version 2.0.2 > hadoop version 2.6.0 > spark -submit command > "spark-submit --class spark.mongohadoop.testing3 --master yarn --deploy-mode > cluster --jars /home/ec2-user/jars/hgmongonew.jar, > /home/ec2-user/jars/mongo-hadoop-spark-2.0.1.jar" > after adding following in > 1 Spark-default.conf > spark.executor.extraJavaOptions -Dconfig.fuction.conf > spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/* > spark.eventLog.dir=hdfs://localhost:9000/user/spark/applicationHistory > spark.eventLog.enabled=true > 2yarn-site.xml > > yarn.application.classpath > > /usr/local/hadoop-2.6.0/etc/hadoop, > /usr/local/hadoop-2.6.0/, > /usr/local/hadoop-2.6.0/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/common/, > /usr/local/hadoop-2.6.0/share/hadoop/common/lib/ > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/, > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/tools/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/*, > /usr/local/spark-2.0.2-bin-hadoop2.6/jars/spark-yarn_2.11-2.0.2.jar > > > Error on log:- > Error: Could not find or load main class > org.apache.spark.deploy.yarn.ApplicationMaster > Error on terminal:- > diagnostics: Application application_1489673977198_0002 failed 2 times due to > AM Container for appattempt_1489673977198_0002_02 exited with exitCode: 1 > For more detailed output, check application tracking > page:http://bdg-hdp-sparkmaster:8088/proxy/application_1489673977198_0002/Then, > click on links to logs of each attempt. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19992) spark-submit on deployment-mode cluster
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929529#comment-15929529 ] Sean Owen commented on SPARK-19992: --- You have some build or environment problem. In particular I don't know that this is valid: spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/* > spark-submit on deployment-mode cluster > --- > > Key: SPARK-19992 > URL: https://issues.apache.org/jira/browse/SPARK-19992 > Project: Spark > Issue Type: Bug > Components: Spark Submit >Affects Versions: 2.0.2 > Environment: spark version 2.0.2 > hadoop version 2.6.0 >Reporter: narendra maru > > spark version 2.0.2 > hadoop version 2.6.0 > spark -submit command > "spark-submit --class spark.mongohadoop.testing3 --master yarn --deploy-mode > cluster --jars /home/ec2-user/jars/hgmongonew.jar, > /home/ec2-user/jars/mongo-hadoop-spark-2.0.1.jar" > after adding following in > 1 Spark-default.conf > spark.executor.extraJavaOptions -Dconfig.fuction.conf > spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/* > spark.eventLog.dir=hdfs://localhost:9000/user/spark/applicationHistory > spark.eventLog.enabled=true > 2yarn-site.xml > > yarn.application.classpath > > /usr/local/hadoop-2.6.0/etc/hadoop, > /usr/local/hadoop-2.6.0/, > /usr/local/hadoop-2.6.0/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/common/, > /usr/local/hadoop-2.6.0/share/hadoop/common/lib/ > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/, > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/tools/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/*, > /usr/local/spark-2.0.2-bin-hadoop2.6/jars/spark-yarn_2.11-2.0.2.jar > > > Error on log:- > Error: Could not find or load main class > org.apache.spark.deploy.yarn.ApplicationMaster > Error on terminal:- > diagnostics: Application application_1489673977198_0002 failed 2 times due to > AM Container for appattempt_1489673977198_0002_02 exited with exitCode: 1 > For more detailed output, check application tracking > page:http://bdg-hdp-sparkmaster:8088/proxy/application_1489673977198_0002/Then, > click on links to logs of each attempt. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
