Hi there I also got exception when running PI example on YARN
Spark version: spark-1.1.1-bin-hadoop2.4 My environment: Hortonworks HDP 2.2 My command: ./bin/spark-submit --master yarn-cluster --class org.apache.spark.examples.SparkPi lib/spark-examples*.jar 10 Output logs: 14/12/17 14:06:32 INFO yarn.Client: Application report from ResourceManager: application identifier: application_1418282263779_0024 appId: 24 clientToAMToken: null appDiagnostics: Application application_1418282263779_0024 failed 2 times due to AM Container for appattempt_1418282263779_0024_000002 exited with exitCode: 1 For more detailed output, check application tracking page: http://sparkvm.localdomain:8088/proxy/application_1418282263779_0024/Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_1418282263779_0024_02_000001 Exit code: 1 Exception message: /hadoop/yarn/local/usercache/root/appcache/application_1418282263779_0024/container_1418282263779_0024_02_000001/launch_container.sh: line 27: $PWD:$PWD/__spark__.jar:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure:$PWD/__app__.jar:$PWD/*: bad substitution Stack trace: ExitCodeException exitCode=1: /hadoop/yarn/local/usercache/root/appcache/application_1418282263779_0024/container_1418282263779_0024_02_000001/launch_container.sh: line 27: $PWD:$PWD/__spark__.jar:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure:$PWD/__app__.jar:$PWD/*: bad substitution at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 Failing this attempt. Failing the application. appMasterHost: N/A appQueue: default appMasterRpcPort: -1 appStartTime: 1418796384166 yarnAppState: FAILED distributedFinalState: FAILED appTrackingUrl: http://sparkvm.localdomain:8088/cluster/app/application_1418282263779_0024 appUser: root Exception in thread "main" org.apache.spark.SparkException: Application is not successful at org.apache.spark.deploy.yarn.Client.run(Client.scala:98) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:176) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:329) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Kyle 2014-12-17 2:45 GMT+08:00 Rahul Swaminathan <rahul.swaminat...@duke.edu>: > > Hi all, > > I am trying to run a simple Spark_Pi application through Yarn from Java > code. I have the Spark_Pi class and everything works fine if I run on > Spark. However, when I set master to “yarn-client” and set yarn mode to > true, I keep getting exceptions. I suspect this has something to do with > the wrong jars being in my class path, but I can’t seem to find any > documentation on exactly what I need. Can someone post an example of > running a Spark Job from Java code on Yarn? What config properties and jars > were necessary? > > Thanks! > Rahul >