if anyone interested I have a script that builds the sandbox (tested in Docker).
On Thu, Feb 18, 2016 at 2:01 AM, Konstantin Boudnik <[email protected]> wrote: > That'd be the perfect question to the EMR team, who did the work on this > new > version of the Spark. I fixed a few standalone issues with Spark 1.5.1, > but I > have no clue about YARN though... > > Cos > > On Wed, Feb 17, 2016 at 09:46PM, MrAsanjar . wrote: > > I have build a single node hadoop/spark sandbox based on the latest > > Apachue Bigtop 1.1.0 build. Spark in standalone mode + HDFS functions > > perfectly, however, fails if yarn-client/yarn-master mode is used as > follow: > > > > *>>spark-submit --class org.apache.spark.examples.SparkPi --master > > yarn-client /usr/lib/spark/lib/spark-examples-1.5.1-hadoop2.7.1.jar 10* > > 16/02/17 05:19:52 ERROR YarnClientSchedulerBackend: Yarn application has > > already exited with state FINISHED! > > Exception in thread "main" java.lang.IllegalStateException: Cannot call > > methods on a stopped SparkContext > > at org.apache.spark.SparkContext.org > > $apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:104) > > ...... > > > > > > Looking at yarn Application log file, there is a *RECEIVED SIGNAL 15: > > SIGTERM *termination signal from the the yarn container. > > >>*yarn logs -applicationId application_1455683261278_0001* > > > > YARN executor launch context: > > env: > > CLASSPATH -> > > > {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/* > > SPARK_LOG_URL_STDERR -> http://2657cd5dc2f6:8042/node/containerlogs > > > =============================================================================== > > > > 16/02/17 04:29:18 INFO impl.ContainerManagementProtocolProxy: Opening > proxy > > : 2657cd5dc2f6:33785 > > 16/02/17 04:29:18 ERROR yarn.ApplicationMaster: *RECEIVED SIGNAL 15: > > SIGTERM* > > 16/02/17 04:29:18 INFO yarn.ApplicationMaster: Final app status: > UNDEFINED, > > exitCode: 0, (reason: Shutdown hook called before final status was > > reported.) > > 16/02/17 04:29:18 INFO yarn.ApplicationMaster: Unregistering > > ApplicationMaster with UNDEFINED (diag message: Shutdown hook called > before > > final status was reported.) > > 16/02/17 04:29:18 INFO impl.AMRMClientImpl: Waiting for application to be > > successfully unregistered. > > 16/02/17 04:29:18 INFO yarn.ApplicationMaster: Deleting staging directory > > .sparkStaging/application_1455683261278_0001 > > 16/02/17 04:29:18 INFO util.ShutdownHookManager: Shutdown hook called > > End of LogType:stderr > > > > > > BTW, I have successfully tested hadoop yarn by running Teragen/Terasort > > mapreduce job. > > Before i start debugging, has anyone tested spark in yarn-client mode? >
