Have you tried Bigtop Provisioner? https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Provisioner+User+Guide
----- $ cd bigtop/bigtop-deploy/vm/vagrant-puppet-vm $ cat vagrantconfig.yaml # You just need to update repo and add spark into component list memory_size: 4096 number_cpus: 1 box: "puppetlabs/centos-7.0-64-nocm" repo: "http://bigtop-repos.s3.amazonaws.com/releases/1.1.0/centos/7/x86_64"; num_instances: 1 distro: centos components: [hadoop, yarn, spark] enable_local_repo: false run_smoke_tests: false smoke_test_components: [mapreduce, pig] jdk: "java-1.7.0-openjdk-devel.x86_64" $ ./docker-hadoop.sh -c 1 # wait for 5mins $ vagrant ssh bigtop1 $ cd /usr/lib/spark $ spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client /usr/lib/spark/lib/spark-examples-1.5.1-hadoop2.7.1.jar 10 ... Pi is roughly 3.144728 ----- Is this what you want? I can run this either on Docker or on a CentOS 7 VM. Sorry I don't have PPC machine to test. 2016-02-18 11:46 GMT+08:00 MrAsanjar . <[email protected]>: > I have build a single node hadoop/spark sandbox based on the latest > Apachue Bigtop 1.1.0 build. Spark in standalone mode + HDFS functions > perfectly, however, fails if yarn-client/yarn-master mode is used as > follow: > > *>>spark-submit --class org.apache.spark.examples.SparkPi --master > yarn-client /usr/lib/spark/lib/spark-examples-1.5.1-hadoop2.7.1.jar 10* > 16/02/17 05:19:52 ERROR YarnClientSchedulerBackend: Yarn application has > already exited with state FINISHED! > Exception in thread "main" java.lang.IllegalStateException: Cannot call > methods on a stopped SparkContext > at org.apache.spark.SparkContext.org > $apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:104) > ...... > > > Looking at yarn Application log file, there is a *RECEIVED SIGNAL 15: > SIGTERM *termination signal from the the yarn container. > >>*yarn logs -applicationId application_1455683261278_0001* > > YARN executor launch context: > env: > CLASSPATH -> > > {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/* > SPARK_LOG_URL_STDERR -> http://2657cd5dc2f6:8042/node/containerlogs > > =============================================================================== > > 16/02/17 04:29:18 INFO impl.ContainerManagementProtocolProxy: Opening proxy > : 2657cd5dc2f6:33785 > 16/02/17 04:29:18 ERROR yarn.ApplicationMaster: *RECEIVED SIGNAL 15: > SIGTERM* > 16/02/17 04:29:18 INFO yarn.ApplicationMaster: Final app status: UNDEFINED, > exitCode: 0, (reason: Shutdown hook called before final status was > reported.) > 16/02/17 04:29:18 INFO yarn.ApplicationMaster: Unregistering > ApplicationMaster with UNDEFINED (diag message: Shutdown hook called before > final status was reported.) > 16/02/17 04:29:18 INFO impl.AMRMClientImpl: Waiting for application to be > successfully unregistered. > 16/02/17 04:29:18 INFO yarn.ApplicationMaster: Deleting staging directory > .sparkStaging/application_1455683261278_0001 > 16/02/17 04:29:18 INFO util.ShutdownHookManager: Shutdown hook called > End of LogType:stderr > > > BTW, I have successfully tested hadoop yarn by running Teragen/Terasort > mapreduce job. > Before i start debugging, has anyone tested spark in yarn-client mode? >
