[jira] [Commented] (SPARK-19992) spark-submit on deployment-mode cluster
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939857#comment-15939857 ] narendra maru commented on SPARK-19992: --- Thanks sean and saisai for your rlyy I followed the same instruction of yours but not getting satisfactory result it will be great if you can help me out with some better suggestions > spark-submit on deployment-mode cluster > --- > > Key: SPARK-19992 > URL: https://issues.apache.org/jira/browse/SPARK-19992 > Project: Spark > Issue Type: Bug > Components: Spark Submit >Affects Versions: 2.0.2 > Environment: spark version 2.0.2 > hadoop version 2.6.0 >Reporter: narendra maru > > spark version 2.0.2 > hadoop version 2.6.0 > spark -submit command > "spark-submit --class spark.mongohadoop.testing3 --master yarn --deploy-mode > cluster --jars /home/ec2-user/jars/hgmongonew.jar, > /home/ec2-user/jars/mongo-hadoop-spark-2.0.1.jar" > after adding following in > 1 Spark-default.conf > spark.executor.extraJavaOptions -Dconfig.fuction.conf > spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/* > spark.eventLog.dir=hdfs://localhost:9000/user/spark/applicationHistory > spark.eventLog.enabled=true > 2yarn-site.xml > > yarn.application.classpath > > /usr/local/hadoop-2.6.0/etc/hadoop, > /usr/local/hadoop-2.6.0/, > /usr/local/hadoop-2.6.0/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/common/, > /usr/local/hadoop-2.6.0/share/hadoop/common/lib/ > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/, > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/tools/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/*, > /usr/local/spark-2.0.2-bin-hadoop2.6/jars/spark-yarn_2.11-2.0.2.jar > > > Error on log:- > Error: Could not find or load main class > org.apache.spark.deploy.yarn.ApplicationMaster > Error on terminal:- > diagnostics: Application application_1489673977198_0002 failed 2 times due to > AM Container for appattempt_1489673977198_0002_02 exited with exitCode: 1 > For more detailed output, check application tracking > page:http://bdg-hdp-sparkmaster:8088/proxy/application_1489673977198_0002/Then, > click on links to logs of each attempt. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19992) spark-submit on deployment-mode cluster
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929762#comment-15929762 ] narendra maru commented on SPARK-19992: --- yes I am having same yarn jars and spark_home directory on all the node at the same location > spark-submit on deployment-mode cluster > --- > > Key: SPARK-19992 > URL: https://issues.apache.org/jira/browse/SPARK-19992 > Project: Spark > Issue Type: Bug > Components: Spark Submit >Affects Versions: 2.0.2 > Environment: spark version 2.0.2 > hadoop version 2.6.0 >Reporter: narendra maru > > spark version 2.0.2 > hadoop version 2.6.0 > spark -submit command > "spark-submit --class spark.mongohadoop.testing3 --master yarn --deploy-mode > cluster --jars /home/ec2-user/jars/hgmongonew.jar, > /home/ec2-user/jars/mongo-hadoop-spark-2.0.1.jar" > after adding following in > 1 Spark-default.conf > spark.executor.extraJavaOptions -Dconfig.fuction.conf > spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/* > spark.eventLog.dir=hdfs://localhost:9000/user/spark/applicationHistory > spark.eventLog.enabled=true > 2yarn-site.xml > > yarn.application.classpath > > /usr/local/hadoop-2.6.0/etc/hadoop, > /usr/local/hadoop-2.6.0/, > /usr/local/hadoop-2.6.0/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/common/, > /usr/local/hadoop-2.6.0/share/hadoop/common/lib/ > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/, > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/tools/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/*, > /usr/local/spark-2.0.2-bin-hadoop2.6/jars/spark-yarn_2.11-2.0.2.jar > > > Error on log:- > Error: Could not find or load main class > org.apache.spark.deploy.yarn.ApplicationMaster > Error on terminal:- > diagnostics: Application application_1489673977198_0002 failed 2 times due to > AM Container for appattempt_1489673977198_0002_02 exited with exitCode: 1 > For more detailed output, check application tracking > page:http://bdg-hdp-sparkmaster:8088/proxy/application_1489673977198_0002/Then, > click on links to logs of each attempt. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19992) spark-submit on deployment-mode cluster
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929546#comment-15929546 ] narendra maru commented on SPARK-19992: --- hi saisai i am working on plain Hadoop 2.6.0 so is there need for any other configuration > spark-submit on deployment-mode cluster > --- > > Key: SPARK-19992 > URL: https://issues.apache.org/jira/browse/SPARK-19992 > Project: Spark > Issue Type: Bug > Components: Spark Submit >Affects Versions: 2.0.2 > Environment: spark version 2.0.2 > hadoop version 2.6.0 >Reporter: narendra maru > > spark version 2.0.2 > hadoop version 2.6.0 > spark -submit command > "spark-submit --class spark.mongohadoop.testing3 --master yarn --deploy-mode > cluster --jars /home/ec2-user/jars/hgmongonew.jar, > /home/ec2-user/jars/mongo-hadoop-spark-2.0.1.jar" > after adding following in > 1 Spark-default.conf > spark.executor.extraJavaOptions -Dconfig.fuction.conf > spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/* > spark.eventLog.dir=hdfs://localhost:9000/user/spark/applicationHistory > spark.eventLog.enabled=true > 2yarn-site.xml > > yarn.application.classpath > > /usr/local/hadoop-2.6.0/etc/hadoop, > /usr/local/hadoop-2.6.0/, > /usr/local/hadoop-2.6.0/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/common/, > /usr/local/hadoop-2.6.0/share/hadoop/common/lib/ > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/, > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/tools/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/*, > /usr/local/spark-2.0.2-bin-hadoop2.6/jars/spark-yarn_2.11-2.0.2.jar > > > Error on log:- > Error: Could not find or load main class > org.apache.spark.deploy.yarn.ApplicationMaster > Error on terminal:- > diagnostics: Application application_1489673977198_0002 failed 2 times due to > AM Container for appattempt_1489673977198_0002_02 exited with exitCode: 1 > For more detailed output, check application tracking > page:http://bdg-hdp-sparkmaster:8088/proxy/application_1489673977198_0002/Then, > click on links to logs of each attempt. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19992) spark-submit on deployment-mode cluster
[ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929545#comment-15929545 ] narendra maru commented on SPARK-19992: --- thanks sean for a quick reply I am suffering from last 4-5 days for same error and not getting any way to resolve the same can you please help me with any suggestions for spark deployment on yarn multinode cluster as the same command is working on deployment mode client > spark-submit on deployment-mode cluster > --- > > Key: SPARK-19992 > URL: https://issues.apache.org/jira/browse/SPARK-19992 > Project: Spark > Issue Type: Bug > Components: Spark Submit >Affects Versions: 2.0.2 > Environment: spark version 2.0.2 > hadoop version 2.6.0 >Reporter: narendra maru > > spark version 2.0.2 > hadoop version 2.6.0 > spark -submit command > "spark-submit --class spark.mongohadoop.testing3 --master yarn --deploy-mode > cluster --jars /home/ec2-user/jars/hgmongonew.jar, > /home/ec2-user/jars/mongo-hadoop-spark-2.0.1.jar" > after adding following in > 1 Spark-default.conf > spark.executor.extraJavaOptions -Dconfig.fuction.conf > spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/* > spark.eventLog.dir=hdfs://localhost:9000/user/spark/applicationHistory > spark.eventLog.enabled=true > 2yarn-site.xml > > yarn.application.classpath > > /usr/local/hadoop-2.6.0/etc/hadoop, > /usr/local/hadoop-2.6.0/, > /usr/local/hadoop-2.6.0/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/common/, > /usr/local/hadoop-2.6.0/share/hadoop/common/lib/ > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/, > /usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/, > /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/tools/lib/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/, > /usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/*, > /usr/local/spark-2.0.2-bin-hadoop2.6/jars/spark-yarn_2.11-2.0.2.jar > > > Error on log:- > Error: Could not find or load main class > org.apache.spark.deploy.yarn.ApplicationMaster > Error on terminal:- > diagnostics: Application application_1489673977198_0002 failed 2 times due to > AM Container for appattempt_1489673977198_0002_02 exited with exitCode: 1 > For more detailed output, check application tracking > page:http://bdg-hdp-sparkmaster:8088/proxy/application_1489673977198_0002/Then, > click on links to logs of each attempt. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-19992) spark-submit on deployment-mode cluster
narendra maru created SPARK-19992: - Summary: spark-submit on deployment-mode cluster Key: SPARK-19992 URL: https://issues.apache.org/jira/browse/SPARK-19992 Project: Spark Issue Type: Bug Components: Spark Submit Affects Versions: 2.0.2 Environment: spark version 2.0.2 hadoop version 2.6.0 Reporter: narendra maru spark version 2.0.2 hadoop version 2.6.0 spark -submit command "spark-submit --class spark.mongohadoop.testing3 --master yarn --deploy-mode cluster --jars /home/ec2-user/jars/hgmongonew.jar, /home/ec2-user/jars/mongo-hadoop-spark-2.0.1.jar" after adding following in 1 Spark-default.conf spark.executor.extraJavaOptions -Dconfig.fuction.conf spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/* spark.eventLog.dir=hdfs://localhost:9000/user/spark/applicationHistory spark.eventLog.enabled=true 2yarn-site.xml yarn.application.classpath /usr/local/hadoop-2.6.0/etc/hadoop, /usr/local/hadoop-2.6.0/, /usr/local/hadoop-2.6.0/lib/, /usr/local/hadoop-2.6.0/share/hadoop/common/, /usr/local/hadoop-2.6.0/share/hadoop/common/lib/ /usr/local/hadoop-2.6.0/share/hadoop/hdfs/, /usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/, /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/, /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/, /usr/local/hadoop-2.6.0/share/hadoop/tools/lib/, /usr/local/hadoop-2.6.0/share/hadoop/yarn/, /usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/*, /usr/local/spark-2.0.2-bin-hadoop2.6/jars/spark-yarn_2.11-2.0.2.jar Error on log:- Error: Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster Error on terminal:- diagnostics: Application application_1489673977198_0002 failed 2 times due to AM Container for appattempt_1489673977198_0002_02 exited with exitCode: 1 For more detailed output, check application tracking page:http://bdg-hdp-sparkmaster:8088/proxy/application_1489673977198_0002/Then, click on links to logs of each attempt. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
