And you can easily make a local repo and then share it via httpd. In this case, it can be served to anywhere.
Cos On Fri, Feb 19, 2016 at 10:15AM, Evans Ye wrote: > You can make a local apt repo anytime. That should do the trick for 1 node > cluster if you just want to test the feature. > > 2016-02-19 2:59 GMT+08:00 MrAsanjar . <[email protected]>: > > > Evans, we couldn't use BigTop Provisioner since ppc artifacts are not in > > the repository yet. That is why we had to made the script. Thanks > > > > > > On Thu, Feb 18, 2016 at 1:40 PM, Konstantin Boudnik <[email protected]> > > wrote: > > > > > On Fri, Feb 19, 2016 at 02:07AM, Evans Ye wrote: > > > > Have you tried Bigtop Provisioner? > > > > > > > > > https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Provisioner+User+Guide > > > > > > > > ----- > > > > $ cd bigtop/bigtop-deploy/vm/vagrant-puppet-vm > > > > $ cat vagrantconfig.yaml # You just need to update repo and add spark > > > into > > > > component list > > > > memory_size: 4096 > > > > number_cpus: 1 > > > > box: "puppetlabs/centos-7.0-64-nocm" > > > > repo: " > > > http://bigtop-repos.s3.amazonaws.com/releases/1.1.0/centos/7/x86_64"; > > > > num_instances: 1 > > > > distro: centos > > > > components: [hadoop, yarn, spark] > > > > enable_local_repo: false > > > > run_smoke_tests: false > > > > smoke_test_components: [mapreduce, pig] > > > > jdk: "java-1.7.0-openjdk-devel.x86_64" > > > > > > > > $ ./docker-hadoop.sh -c 1 # wait for 5mins > > > > $ vagrant ssh bigtop1 > > > > $ cd /usr/lib/spark > > > > $ spark-submit --class org.apache.spark.examples.SparkPi --master > > > > yarn-client /usr/lib/spark/lib/spark-examples-1.5.1-hadoop2.7.1.jar 10 > > > > ... > > > > Pi is roughly 3.144728 > > > > > > Good enough for 1.1.0 release ;) > > > > > > But seriously: we have a real good tool to provision clusters quickly and > > > painlessly. And with Puppet to guarantee the consistency. > > > > > > Cos > > > > > > > ----- > > > > > > > > Is this what you want? > > > > I can run this either on Docker or on a CentOS 7 VM. > > > > Sorry I don't have PPC machine to test. > > > > > > > > > > > > 2016-02-18 11:46 GMT+08:00 MrAsanjar . <[email protected]>: > > > > > > > > > I have build a single node hadoop/spark sandbox based on the latest > > > > > Apachue Bigtop 1.1.0 build. Spark in standalone mode + HDFS functions > > > > > perfectly, however, fails if yarn-client/yarn-master mode is used as > > > > > follow: > > > > > > > > > > *>>spark-submit --class org.apache.spark.examples.SparkPi --master > > > > > yarn-client /usr/lib/spark/lib/spark-examples-1.5.1-hadoop2.7.1.jar > > > 10* > > > > > 16/02/17 05:19:52 ERROR YarnClientSchedulerBackend: Yarn application > > > has > > > > > already exited with state FINISHED! > > > > > Exception in thread "main" java.lang.IllegalStateException: Cannot > > call > > > > > methods on a stopped SparkContext > > > > > at org.apache.spark.SparkContext.org > > > > > $apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:104) > > > > > ...... > > > > > > > > > > > > > > > Looking at yarn Application log file, there is a *RECEIVED SIGNAL 15: > > > > > SIGTERM *termination signal from the the yarn container. > > > > > >>*yarn logs -applicationId application_1455683261278_0001* > > > > > > > > > > YARN executor launch context: > > > > > env: > > > > > CLASSPATH -> > > > > > > > > > > > > > > > {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/* > > > > > SPARK_LOG_URL_STDERR -> > > > http://2657cd5dc2f6:8042/node/containerlogs > > > > > > > > > > > > > > > =============================================================================== > > > > > > > > > > 16/02/17 04:29:18 INFO impl.ContainerManagementProtocolProxy: Opening > > > proxy > > > > > : 2657cd5dc2f6:33785 > > > > > 16/02/17 04:29:18 ERROR yarn.ApplicationMaster: *RECEIVED SIGNAL 15: > > > > > SIGTERM* > > > > > 16/02/17 04:29:18 INFO yarn.ApplicationMaster: Final app status: > > > UNDEFINED, > > > > > exitCode: 0, (reason: Shutdown hook called before final status was > > > > > reported.) > > > > > 16/02/17 04:29:18 INFO yarn.ApplicationMaster: Unregistering > > > > > ApplicationMaster with UNDEFINED (diag message: Shutdown hook called > > > before > > > > > final status was reported.) > > > > > 16/02/17 04:29:18 INFO impl.AMRMClientImpl: Waiting for application > > to > > > be > > > > > successfully unregistered. > > > > > 16/02/17 04:29:18 INFO yarn.ApplicationMaster: Deleting staging > > > directory > > > > > .sparkStaging/application_1455683261278_0001 > > > > > 16/02/17 04:29:18 INFO util.ShutdownHookManager: Shutdown hook called > > > > > End of LogType:stderr > > > > > > > > > > > > > > > BTW, I have successfully tested hadoop yarn by running > > Teragen/Terasort > > > > > mapreduce job. > > > > > Before i start debugging, has anyone tested spark in yarn-client > > mode? > > > > > > > > > >
signature.asc
Description: Digital signature
