I have build a single node hadoop/spark sandbox based on the latest
Apachue Bigtop 1.1.0 build. Spark in standalone mode + HDFS functions
perfectly, however, fails if yarn-client/yarn-master mode is used as follow:
*>>spark-submit --class org.apache.spark.examples.SparkPi --master
yarn-client /usr/lib/spark/lib/spark-examples-1.5.1-hadoop2.7.1.jar 10*
16/02/17 05:19:52 ERROR YarnClientSchedulerBackend: Yarn application has
already exited with state FINISHED!
Exception in thread "main" java.lang.IllegalStateException: Cannot call
methods on a stopped SparkContext
at org.apache.spark.SparkContext.org
$apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:104)
......
Looking at yarn Application log file, there is a *RECEIVED SIGNAL 15:
SIGTERM *termination signal from the the yarn container.
>>*yarn logs -applicationId application_1455683261278_0001*
YARN executor launch context:
env:
CLASSPATH ->
{{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
SPARK_LOG_URL_STDERR -> http://2657cd5dc2f6:8042/node/containerlogs
===============================================================================
16/02/17 04:29:18 INFO impl.ContainerManagementProtocolProxy: Opening proxy
: 2657cd5dc2f6:33785
16/02/17 04:29:18 ERROR yarn.ApplicationMaster: *RECEIVED SIGNAL 15:
SIGTERM*
16/02/17 04:29:18 INFO yarn.ApplicationMaster: Final app status: UNDEFINED,
exitCode: 0, (reason: Shutdown hook called before final status was
reported.)
16/02/17 04:29:18 INFO yarn.ApplicationMaster: Unregistering
ApplicationMaster with UNDEFINED (diag message: Shutdown hook called before
final status was reported.)
16/02/17 04:29:18 INFO impl.AMRMClientImpl: Waiting for application to be
successfully unregistered.
16/02/17 04:29:18 INFO yarn.ApplicationMaster: Deleting staging directory
.sparkStaging/application_1455683261278_0001
16/02/17 04:29:18 INFO util.ShutdownHookManager: Shutdown hook called
End of LogType:stderr
BTW, I have successfully tested hadoop yarn by running Teragen/Terasort
mapreduce job.
Before i start debugging, has anyone tested spark in yarn-client mode?
