shihj@master:/usr/local/spark/spark-1.6.1-bin-hadoop2.6$ ./bin/spark-submit \ > --class org.apache.spark.examples.SparkPi \ > --master yarn-cluster \ > --driver-memory 512m \ > --num-executors 2 \ > --executor-memory 512m \ > --executor-cores 2 \ > hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar 16/06/22 14:36:10 INFO RMProxy: Connecting to ResourceManager at master/192.168.20.137:8032 16/06/22 14:36:10 INFO Client: Requesting a new application from cluster with 2 NodeManagers 16/06/22 14:36:10 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container) 16/06/22 14:36:10 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 16/06/22 14:36:10 INFO Client: Setting up container launch context for our AM 16/06/22 14:36:10 INFO Client: Setting up the launch environment for our AM container 16/06/22 14:36:10 INFO Client: Preparing resources for our AM container Java HotSpot(TM) Server VM warning: You have loaded library /tmp/libnetty-transport-native-epoll3453573359049032130.so which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'. 16/06/22 14:36:11 INFO Client: Source and destination file systems are the same. Not copying hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar 16/06/22 14:36:11 WARN Client: Resource hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar added multiple times to distributed cache. 16/06/22 14:36:11 INFO Client: Uploading resource file:/tmp/spark-cf23c5a3-d3fb-4f98-9cd2-bbf268766bbc/__spark_conf__7248368026523433025.zip -> hdfs://master:9000/user/shihj/.sparkStaging/application_1466568126079_0006/__spark_conf__7248368026523433025.zip 16/06/22 14:36:13 INFO SecurityManager: Changing view acls to: shihj 16/06/22 14:36:13 INFO SecurityManager: Changing modify acls to: shihj 16/06/22 14:36:13 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(shihj); users with modify permissions: Set(shihj) 16/06/22 14:36:13 INFO Client: Submitting application 6 to ResourceManager 16/06/22 14:36:13 INFO YarnClientImpl: Submitted application application_1466568126079_0006 16/06/22 14:36:14 INFO Client: Application report for application_1466568126079_0006 (state: ACCEPTED) 16/06/22 14:36:14 INFO Client: client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1466577373576 final status: UNDEFINED tracking URL: http://master:8088/proxy/application_1466568126079_0006/ user: shihj 16/06/22 14:36:15 INFO Client: Application report for application_1466568126079_0006 (state: ACCEPTED) 16/06/22 14:36:16 INFO Client: Application report for application_1466568126079_0006 (state: ACCEPTED) 16/06/22 14:36:17 INFO Client: Application report for application_1466568126079_0006 (state: ACCEPTED) 16/06/22 14:36:18 INFO Client: Application report for application_1466568126079_0006 (state: ACCEPTED) 16/06/22 14:36:19 INFO Client: Application report for application_1466568126079_0006 (state: ACCEPTED) 16/06/22 14:36:20 INFO Client: Application report for application_1466568126079_0006 (state: ACCEPTED) 16/06/22 14:36:21 INFO Client: Application report for application_1466568126079_0006 (state: ACCEPTED) 16/06/22 14:36:22 INFO Client: Application report for application_1466568126079_0006 (state: ACCEPTED) 16/06/22 14:36:23 INFO Client: Application report for application_1466568126079_0006 (state: ACCEPTED) 16/06/22 14:36:24 INFO Client: Application report for application_1466568126079_0006 (state: ACCEPTED) 16/06/22 14:36:25 INFO Client: Application report for application_1466568126079_0006 (state: ACCEPTED) 16/06/22 14:36:26 INFO Client: Application report for application_1466568126079_0006 (state: ACCEPTED) 16/06/22 14:36:27 INFO Client: Application report for application_1466568126079_0006 (state: FAILED) 16/06/22 14:36:27 INFO Client: client token: N/A diagnostics: Application application_1466568126079_0006 failed 2 times due to AM Container for appattempt_1466568126079_0006_000002 exited with exitCode: 1 For more detailed output, check application tracking page:http://master:8088/proxy/application_1466568126079_0006/Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_1466568126079_0006_02_000001 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1 Failing this attempt. Failing the application. ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1466577373576 final status: FAILED tracking URL: http://master:8088/cluster/app/application_1466568126079_0006 user: shihj 16/06/22 14:36:27 INFO Client: Deleting staging directory .sparkStaging/application_1466568126079_0006 Exception in thread "main" org.apache.spark.SparkException: Application application_1466568126079_0006 finished with failed status at org.apache.spark.deploy.yarn.Client.run(Client.scala:1034) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1081) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 16/06/22 14:36:27 INFO ShutdownHookManager: Shutdown hook called 16/06/22 14:36:27 INFO ShutdownHookManager: Deleting directory /tmp/spark-cf23c5a3-d3fb-4f98-9cd2-bbf268766bbc ------------------ ???????? ------------------ ??????: "Yash Sharma";<yash...@gmail.com>; ????????: 2016??6??22??(??????) ????2:34 ??????: "????????"<958943...@qq.com>; ????: "Saisai Shao"<sai.sai.s...@gmail.com>; "user"<user@spark.apache.org>; ????: Re: Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher Try with : --master yarn-cluster On Wed, Jun 22, 2016 at 4:30 PM, ???????? <958943...@qq.com> wrote: ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --driver-memory 512m --num-executors 2 --executor-memory 512m --executor-cores 2 hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar 10 Warning: Skip remote jar hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar. java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPi at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.util.Utils$.classForName(Utils.scala:174) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:689) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ------------------ ???????? ------------------ ??????: "Yash Sharma";<yash...@gmail.com>; ????????: 2016??6??22??(??????) ????2:28 ??????: "????????"<958943...@qq.com>; ????: "Saisai Shao"<sai.sai.s...@gmail.com>; "user"<user@spark.apache.org>; ????: Re: Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher Or better , try the master as yarn-cluster, ./bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --master yarn-cluster \ --driver-memory 512m \ --num-executors 2 \ --executor-memory 512m \ --executor-cores 2 \ hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar On Wed, Jun 22, 2016 at 4:27 PM, ???????? <958943...@qq.com> wrote: Is it able to run on local mode ? what mean?? standalone mode ? ------------------ ???????? ------------------ ??????: "Yash Sharma";<yash...@gmail.com>; ????????: 2016??6??22??(??????) ????2:18 ??????: "Saisai Shao"<sai.sai.s...@gmail.com>; ????: "????????"<958943...@qq.com>; "user"<user@spark.apache.org>; ????: Re: Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher Try providing the jar with the hdfs prefix. Its probably just because its not able to find the jar on all nodes. hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar Is it able to run on local mode ? On Wed, Jun 22, 2016 at 4:14 PM, Saisai Shao <sai.sai.s...@gmail.com> wrote: spark.yarn.jar(none)The location of the Spark jar file, in case overriding the default location is desired. By default, Spark on YARN will use a Spark jar installed locally, but the Spark jar can also be in a world-readable location on HDFS. This allows YARN to cache it on nodes so that it doesn't need to be distributed each time an application runs. To point to a jar on HDFS, for example, set this configuration to hdfs:///some/path. spark.yarn.jar is used for spark run-time system jar, which is spark assembly jar, not the application jar (example-assembly jar). So in your case you upload the example-assembly jar into hdfs, in which spark system jars are not packed, so ExecutorLaucher cannot be found. Thanks Saisai On Wed, Jun 22, 2016 at 2:10 PM, ???????? <958943...@qq.com> wrote: shihj@master:/usr/local/spark/spark-1.6.1-bin-hadoop2.6$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --driver-memory 512m --num-executors 2 --executor-memory 512m --executor-cores 2 /user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar 10 Warning: Local jar /user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar does not exist, skipping. java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPi at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.util.Utils$.classForName(Utils.scala:174) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:689) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) get error at once ------------------ ???????? ------------------ ??????: "Yash Sharma";<yash...@gmail.com>; ????????: 2016??6??22??(??????) ????2:04 ??????: "????????"<958943...@qq.com>; ????: "user"<user@spark.apache.org>; ????: Re: Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher How about supplying the jar directly in spark submit - ./bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --master yarn-client \ --driver-memory 512m \ --num-executors 2 \ --executor-memory 512m \ --executor-cores 2 \ /user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar On Wed, Jun 22, 2016 at 3:59 PM, ???????? <958943...@qq.com> wrote: i config this para at spark-defaults.conf spark.yarn.jar hdfs://master:9000/user/shihj/spark_lib/spark-examples-1.6.1-hadoop2.6.0.jar then ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --driver-memory 512m --num-executors 2 --executor-memory 512m --executor-cores 2 10: Error: Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher but i don't config that para ,there no error why???that para is only avoid Uploading resource file(jar package)??