That'd be the perfect question to the EMR team, who did the work on this new version of the Spark. I fixed a few standalone issues with Spark 1.5.1, but I have no clue about YARN though...
Cos
On Wed, Feb 17, 2016 at 09:46PM, MrAsanjar . wrote:
> I have build a single node hadoop/spark sandbox based on the latest
> Apachue Bigtop 1.1.0 build. Spark in standalone mode + HDFS functions
> perfectly, however, fails if yarn-client/yarn-master mode is used as follow:
>
> *>>spark-submit --class org.apache.spark.examples.SparkPi --master
> yarn-client /usr/lib/spark/lib/spark-examples-1.5.1-hadoop2.7.1.jar 10*
> 16/02/17 05:19:52 ERROR YarnClientSchedulerBackend: Yarn application has
> already exited with state FINISHED!
> Exception in thread "main" java.lang.IllegalStateException: Cannot call
> methods on a stopped SparkContext
> at org.apache.spark.SparkContext.org
> $apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:104)
> ......
>
>
> Looking at yarn Application log file, there is a *RECEIVED SIGNAL 15:
> SIGTERM *termination signal from the the yarn container.
> >>*yarn logs -applicationId application_1455683261278_0001*
>
> YARN executor launch context:
> env:
> CLASSPATH ->
> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
> SPARK_LOG_URL_STDERR -> http://2657cd5dc2f6:8042/node/containerlogs
> ===============================================================================
>
> 16/02/17 04:29:18 INFO impl.ContainerManagementProtocolProxy: Opening proxy
> : 2657cd5dc2f6:33785
> 16/02/17 04:29:18 ERROR yarn.ApplicationMaster: *RECEIVED SIGNAL 15:
> SIGTERM*
> 16/02/17 04:29:18 INFO yarn.ApplicationMaster: Final app status: UNDEFINED,
> exitCode: 0, (reason: Shutdown hook called before final status was
> reported.)
> 16/02/17 04:29:18 INFO yarn.ApplicationMaster: Unregistering
> ApplicationMaster with UNDEFINED (diag message: Shutdown hook called before
> final status was reported.)
> 16/02/17 04:29:18 INFO impl.AMRMClientImpl: Waiting for application to be
> successfully unregistered.
> 16/02/17 04:29:18 INFO yarn.ApplicationMaster: Deleting staging directory
> .sparkStaging/application_1455683261278_0001
> 16/02/17 04:29:18 INFO util.ShutdownHookManager: Shutdown hook called
> End of LogType:stderr
>
>
> BTW, I have successfully tested hadoop yarn by running Teragen/Terasort
> mapreduce job.
> Before i start debugging, has anyone tested spark in yarn-client mode?
signature.asc
Description: Digital signature
