Hi,
My test environment is Hortonworks HDP2.2 installed by Ambari 1.7.0, and Spark
1.2.0 pre-built, and failed with the following error:
[xiaobogu@lix1 ~]$ /opt/spark/bin/spark-submit --class
org.apache.spark.examples.SparkPi --master yarn-cluster --num-executors 1
--driver-memory 512m --executor-memory 512m --executor-cores 1
/opt/spark/lib/spark-examples*.jar 10
Spark assembly has been built with Hive, including Datanucleus jars on classpath
15/02/11 21:53:30 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/02/11 21:53:31 INFO client.RMProxy: Connecting to ResourceManager at
lix1.bh.com/192.168.100.3:8050
15/02/11 21:53:32 INFO yarn.Client: Requesting a new application from cluster
with 1 NodeManagers
15/02/11 21:53:32 INFO yarn.Client: Verifying our application has not requested
more than the maximum memory capability of the cluster (4096 MB per container)
15/02/11 21:53:32 INFO yarn.Client: Will allocate AM container, with 896 MB
memory including 384 MB overhead
15/02/11 21:53:32 INFO yarn.Client: Setting up container launch context for our
AM
15/02/11 21:53:32 INFO yarn.Client: Preparing resources for our AM container
15/02/11 21:53:33 WARN hdfs.BlockReaderLocal: The short-circuit local reads
feature cannot be used because libhadoop cannot be loaded.
15/02/11 21:53:33 INFO yarn.Client: Uploading resource
file:/opt/spark-1.2.0-bin-hadoop2.4/lib/spark-assembly-1.2.0-hadoop2.4.0.jar ->
hdfs://lix1.bh.com:8020/user/xiaobogu/.sparkStaging/application_1423657221172_0005/spark-assembly-1.2.0-hadoop2.4.0.jar
15/02/11 21:53:37 INFO yarn.Client: Uploading resource
file:/opt/spark/lib/spark-examples-1.2.0-hadoop2.4.0.jar ->
hdfs://lix1.bh.com:8020/user/xiaobogu/.sparkStaging/application_1423657221172_0005/spark-examples-1.2.0-hadoop2.4.0.jar
15/02/11 21:53:39 INFO yarn.Client: Setting up the launch environment for our
AM container
15/02/11 21:53:40 INFO spark.SecurityManager: Changing view acls to: xiaobogu
15/02/11 21:53:40 INFO spark.SecurityManager: Changing modify acls to: xiaobogu
15/02/11 21:53:40 INFO spark.SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(xiaobogu); users
with modify permissions: Set(xiaobogu)
15/02/11 21:53:40 INFO yarn.Client: Submitting application 5 to ResourceManager
15/02/11 21:53:40 INFO impl.YarnClientImpl: Submitted application
application_1423657221172_0005
15/02/11 21:53:41 INFO yarn.Client: Application report for
application_1423657221172_0005 (state: ACCEPTED)
15/02/11 21:53:41 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1423662820308
final status: UNDEFINED
tracking URL:
http://lix1.bh.com:8088/proxy/application_1423657221172_0005/
user: xiaobogu
15/02/11 21:53:42 INFO yarn.Client: Application report for
application_1423657221172_0005 (state: ACCEPTED)
15/02/11 21:53:43 INFO yarn.Client: Application report for
application_1423657221172_0005 (state: ACCEPTED)
15/02/11 21:53:44 INFO yarn.Client: Application report for
application_1423657221172_0005 (state: ACCEPTED)
15/02/11 21:53:45 INFO yarn.Client: Application report for
application_1423657221172_0005 (state: ACCEPTED)
15/02/11 21:53:46 INFO yarn.Client: Application report for
application_1423657221172_0005 (state: ACCEPTED)
15/02/11 21:53:47 INFO yarn.Client: Application report for
application_1423657221172_0005 (state: FAILED)
15/02/11 21:53:47 INFO yarn.Client:
client token: N/A
diagnostics: Application application_1423657221172_0005 failed 2 times
due to AM Container for appattempt_1423657221172_0005_000002 exited with
exitCode: 1
For more detailed output, check application tracking
page:http://lix1.bh.com:8088/proxy/application_1423657221172_0005/Then, click
on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1423657221172_0005_02_000001
Exit code: 1
Exception message:
/hadoop/yarn/local/usercache/xiaobogu/appcache/application_1423657221172_0005/container_1423657221172_0005_02_000001/launch_container.sh:
line 27:
$PWD:$PWD/__spark__.jar:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure:$PWD/__app__.jar:$PWD/*:
bad substitution
Stack trace: ExitCodeException exitCode=1:
/hadoop/yarn/local/usercache/xiaobogu/appcache/application_1423657221172_0005/container_1423657221172_0005_02_000001/launch_container.sh:
line 27:
$PWD:$PWD/__spark__.jar:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure:$PWD/__app__.jar:$PWD/*:
bad substitution
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1423662820308
final status: FAILED
tracking URL:
http://lix1.bh.com:8088/cluster/app/application_1423657221172_0005
user: xiaobogu
Exception in thread "main" org.apache.spark.SparkException: Application
finished with failed status
at
org.apache.spark.deploy.yarn.ClientBase$class.run(ClientBase.scala:504)
at org.apache.spark.deploy.yarn.Client.run(Client.scala:35)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:139)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
[xiaobogu@lix1 ~]$
