Indeed... I actually use that in other cases but for some reason I didn't get it right this time :/. Thanks for catching this!
Best, Sam Best, Samuel -- Samuel Cozannet Cloud, Big Data and IoT Strategy Team Business Development - Cloud and ISV Ecosystem Changing the Future of Cloud Ubuntu <http://ubuntu.com> / Canonical UK LTD <http://canonical.com> / Juju <https://jujucharms.com> [email protected] mob: +33 616 702 389 skype: samnco Twitter: @SaMnCo_23 On Tue, Feb 3, 2015 at 1:43 AM, Andrew Wilkins <[email protected] > wrote: > On Mon, Feb 2, 2015 at 9:57 PM, Samuel Cozannet < > [email protected]> wrote: > >> Excellent! Happy to help you through your discovery of awesomeness with >> Juju :) >> >> Note that, if you have jq installed (which I advise, sudo apt-get install >> jq) >> juju stat | python -c 'import sys, yaml, json; >> json.dump(yaml.load(sys.stdin), sys.stdout, indent=4)' | jq >> '.services."<SERVICE NAME>".units."<SERVICE NAME>/0".machine' | tr -d >> "\"" >> > > FYI, you can do "juju status --format=json" and skip the Python bit. > > will return the ID of the machine for <SERVICE NAME> (replace that by >> yarn-master or what ever the name you gave), which saves you the browsing >> of several pages of juju status... >> >> Let us know how your testing goes! >> >> Best, >> Sam >> >> >> Best, >> Samuel >> >> -- >> Samuel Cozannet >> Cloud, Big Data and IoT Strategy Team >> Business Development - Cloud and ISV Ecosystem >> Changing the Future of Cloud >> Ubuntu <http://ubuntu.com> / Canonical UK LTD <http://canonical.com> / >> Juju <https://jujucharms.com> >> [email protected] >> mob: +33 616 702 389 >> skype: samnco >> Twitter: @SaMnCo_23 >> >> On Mon, Feb 2, 2015 at 2:52 PM, Ken Williams <[email protected]> wrote: >> >>> Hi Sam, >>> >>> Just to confirm that deploying the spark-master and the >>> yarn-hdfs-master to >>> the same machine seems to have worked ! :-) >>> >>> // use 'juju status' to find which machine yarn-hdfs-master is on >>> juju status >>> [ etc...] >>> // say...machine: "4" >>> >>> // deploy spark-master to same machine >>> juju deploy --to 4 spark-master >>> >>> // add relations >>> juju add-relation yarn-hdfs-master:resourcemanager spark-master >>> juju add-relation yarn-hdfs-master:namenode spark-master >>> >>> >>> // run test >>> root@ip-172-31-21-92:~# spark-submit --class >>> org.apache.spark.examples.SparkPi /tmp/spark-examples-1.2.0-hadoop2.4.0.jar >>> 25 --master yarn --num-executors 3 --driver-memory 1g >>> --executor-memory 1g --executor-cores 1 --deploy-mode cluster >>> --queue thequeue /tmp/spark-examples-1.2.0-hadoop2.4.0.jar >>> Spark assembly has been built with Hive, including Datanucleus jars on >>> classpath >>> 15/02/02 13:40:45 WARN NativeCodeLoader: Unable to load native-hadoop >>> library for your platform... using builtin-java classes where applicable >>> Pi is roughly 3.1405888 >>> >>> >>> Many thanks again for all your help, >>> >>> Best Regards, >>> >>> Ken >>> >>> >>> >>> On 30 January 2015 at 18:11, Ken Williams <[email protected]> wrote: >>> >>>> >>>> Ok - Sam, I'll try this and let you know. >>>> >>>> Thanks again for all your help, >>>> >>>> Best Regards, >>>> >>>> Ken >>>> >>>> >>>> >>>> On 30 January 2015 at 18:09, Samuel Cozannet < >>>> [email protected]> wrote: >>>> >>>>> I'll have a look asap, but probably not before Tuesday. >>>>> >>>>> This may be "my guts tell me that" but, if you have the time, try to >>>>> collocate YARN and Spark, that will guarantee you have the YARN_CONF_DIR >>>>> set. I am 90% sure it will fix your problem. >>>>> >>>>> YARN itself will not eat much resources, you should be alright and it >>>>> may allow you to move forward instead of being stuck. >>>>> >>>>> Best, >>>>> Sam >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Best, >>>>> Samuel >>>>> >>>>> -- >>>>> Samuel Cozannet >>>>> Cloud, Big Data and IoT Strategy Team >>>>> Business Development - Cloud and ISV Ecosystem >>>>> Changing the Future of Cloud >>>>> Ubuntu <http://ubuntu.com> / Canonical UK LTD <http://canonical.com> / >>>>> Juju <https://jujucharms.com> >>>>> [email protected] >>>>> mob: +33 616 702 389 >>>>> skype: samnco >>>>> Twitter: @SaMnCo_23 >>>>> >>>>> On Fri, Jan 30, 2015 at 7:01 PM, Ken Williams <[email protected]> wrote: >>>>> >>>>>> Hi Sam, >>>>>> >>>>>> Attached is my bundles.yaml file. >>>>>> >>>>>> Also, there is no file 'directories.sh' on my spark-master/0 >>>>>> machine (see below), >>>>>> >>>>>> ubuntu@ip-172-31-54-245:~$ ls -l /etc/profile.d/ >>>>>> total 12 >>>>>> -rw-r--r-- 1 root root 1559 Jul 29 2014 Z97-byobu.sh >>>>>> -rwxr-xr-x 1 root root 2691 Oct 6 13:19 Z99-cloud-locale-test.sh >>>>>> -rw-r--r-- 1 root root 663 Apr 7 2014 bash_completion.sh >>>>>> ubuntu@ip-172-31-54-245:~$ >>>>>> >>>>>> >>>>>> Many thanks again your help, >>>>>> >>>>>> Ken >>>>>> >>>>>> >>>>>> On 30 January 2015 at 15:45, Samuel Cozannet < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hey, >>>>>>> >>>>>>> can you send the bundle you're using (in the GUI, bottom right, >>>>>>> "export" button should give you a bundles.yaml file, please send that to >>>>>>> me, so I can bootstrap the same environment as you are playing with. >>>>>>> >>>>>>> also >>>>>>> * can you let me know if you have a file >>>>>>> /etc/profile.d/directories.sh? >>>>>>> * if yes, can you execute it from your command line, then do the >>>>>>> spark command again, and let me know? >>>>>>> >>>>>>> Thx, >>>>>>> Sam >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Best, >>>>>>> Samuel >>>>>>> >>>>>>> -- >>>>>>> Samuel Cozannet >>>>>>> Cloud, Big Data and IoT Strategy Team >>>>>>> Business Development - Cloud and ISV Ecosystem >>>>>>> Changing the Future of Cloud >>>>>>> Ubuntu <http://ubuntu.com> / Canonical UK LTD >>>>>>> <http://canonical.com> / Juju <https://jujucharms.com> >>>>>>> [email protected] >>>>>>> mob: +33 616 702 389 >>>>>>> skype: samnco >>>>>>> Twitter: @SaMnCo_23 >>>>>>> >>>>>>> On Fri, Jan 30, 2015 at 3:46 PM, Ken Williams <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Ok - I have been able to add the relation using this, >>>>>>>> >>>>>>>> juju add-relation yarn-hdfs-master:resourcemanager >>>>>>>> spark-master >>>>>>>> >>>>>>>> But I still cannot see a /etc/hadoop/conf directory on the >>>>>>>> spark-master machine >>>>>>>> so I still get the same error about HADOOP_CONF_DIR and >>>>>>>> YARN_CONF_DIR (below), >>>>>>>> >>>>>>>> >>>>>>>> root@ip-172-31-60-53:~# spark-submit --class >>>>>>>> org.apache.spark.examples.SparkPi --master yarn-client >>>>>>>> --num-executors 3 --driver-memory 1g --executor-memory 1g >>>>>>>> --executor-cores 1 --queue thequeue lib/spark-examples*.jar >>>>>>>> 10 >>>>>>>> Spark assembly has been built with Hive, including Datanucleus jars >>>>>>>> on classpath >>>>>>>> Exception in thread "main" java.lang.Exception: When running with >>>>>>>> master 'yarn-client' either HADOOP_CONF_DIR or YARN_CONF_DIR must be >>>>>>>> set in >>>>>>>> the environment. >>>>>>>> at >>>>>>>> org.apache.spark.deploy.SparkSubmitArguments.checkRequiredArguments(SparkSubmitArguments.scala:177) >>>>>>>> at >>>>>>>> org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:81) >>>>>>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:70) >>>>>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>>>>>>> root@ip-172-31-60-53:~# >>>>>>>> >>>>>>>> Should there be a /etc/hadoop/conf directory ? >>>>>>>> >>>>>>>> Thanks for any help, >>>>>>>> >>>>>>>> Ken >>>>>>>> >>>>>>>> >>>>>>>> On 30 January 2015 at 12:59, Samuel Cozannet < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Have you tried without ':master": >>>>>>>>> >>>>>>>>> juju add-relation yarn-hdfs-master:resourcemanager spark-master >>>>>>>>> >>>>>>>>> I think Spark master consumes the relationship but doesn't have to >>>>>>>>> expose its master relationship. >>>>>>>>> >>>>>>>>> Rule of thumb, when a relation is non ambiguous on one of its >>>>>>>>> ends, there is no requirement to specify it when adding it. >>>>>>>>> >>>>>>>>> Another option if this doesn't work is to use the GUI to create >>>>>>>>> the relation. It will give you a dropdown of available relationships >>>>>>>>> between entities. >>>>>>>>> >>>>>>>>> Let me know how it goes, >>>>>>>>> Thx, >>>>>>>>> Sam >>>>>>>>> >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Samuel >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Samuel Cozannet >>>>>>>>> Cloud, Big Data and IoT Strategy Team >>>>>>>>> Business Development - Cloud and ISV Ecosystem >>>>>>>>> Changing the Future of Cloud >>>>>>>>> Ubuntu <http://ubuntu.com> / Canonical UK LTD >>>>>>>>> <http://canonical.com> / Juju <https://jujucharms.com> >>>>>>>>> [email protected] >>>>>>>>> mob: +33 616 702 389 >>>>>>>>> skype: samnco >>>>>>>>> Twitter: @SaMnCo_23 >>>>>>>>> >>>>>>>>> On Fri, Jan 30, 2015 at 1:09 PM, Ken Williams <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Sam, >>>>>>>>>> >>>>>>>>>> I understand what you are saying but when I try to add the 2 >>>>>>>>>> relations I get this error, >>>>>>>>>> >>>>>>>>>> root@adminuser-VirtualBox:~# juju add-relation >>>>>>>>>> yarn-hdfs-master:resourcemanager spark-master:master >>>>>>>>>> ERROR no relations found >>>>>>>>>> root@adminuser-VirtualBox:~# juju add-relation >>>>>>>>>> yarn-hdfs-master:namenode spark-master:master >>>>>>>>>> ERROR no relations found >>>>>>>>>> >>>>>>>>>> Am I adding the relations right ? >>>>>>>>>> >>>>>>>>>> Attached is my 'juju status' file. >>>>>>>>>> >>>>>>>>>> Thanks for all your help, >>>>>>>>>> >>>>>>>>>> Ken >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 30 January 2015 at 11:16, Samuel Cozannet < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hey Ken, >>>>>>>>>>> >>>>>>>>>>> Yes, you need to create the relationship between the 2 entities >>>>>>>>>>> to they know about each other. >>>>>>>>>>> >>>>>>>>>>> Looking at the list of hooks for the charm >>>>>>>>>>> <https://github.com/Archethought/spark-charm/tree/master/hooks> you >>>>>>>>>>> can see there are 2 hooks named namenode-relation-changed >>>>>>>>>>> <https://github.com/Archethought/spark-charm/blob/master/hooks/namenode-relation-changed> >>>>>>>>>>> and resourcemanager-relation-changed >>>>>>>>>>> <https://github.com/Archethought/spark-charm/blob/master/hooks/resourcemanager-relation-changed> >>>>>>>>>>> which >>>>>>>>>>> are related to YARN/Hadoop. >>>>>>>>>>> Looking deeper in the code, you'll notice they reference a >>>>>>>>>>> function found in bdutils.py called "setHadoopEnvVar()", which >>>>>>>>>>> based on its >>>>>>>>>>> name should set the HADOOP_CONF_DIR. >>>>>>>>>>> >>>>>>>>>>> There are 2 relations, so add both of them. >>>>>>>>>>> >>>>>>>>>>> Note that I didn't test this myself, but I expect this should >>>>>>>>>>> fix the problem. If it doesn't please come back to us... >>>>>>>>>>> >>>>>>>>>>> Thanks! >>>>>>>>>>> Sam >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Samuel >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Samuel Cozannet >>>>>>>>>>> Cloud, Big Data and IoT Strategy Team >>>>>>>>>>> Business Development - Cloud and ISV Ecosystem >>>>>>>>>>> Changing the Future of Cloud >>>>>>>>>>> Ubuntu <http://ubuntu.com> / Canonical UK LTD >>>>>>>>>>> <http://canonical.com> / Juju <https://jujucharms.com> >>>>>>>>>>> [email protected] >>>>>>>>>>> mob: +33 616 702 389 >>>>>>>>>>> skype: samnco >>>>>>>>>>> Twitter: @SaMnCo_23 >>>>>>>>>>> >>>>>>>>>>> On Fri, Jan 30, 2015 at 11:51 AM, Ken Williams <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thanks, Kapil - this works :-) >>>>>>>>>>>> >>>>>>>>>>>> I can now run the SparkPi example successfully. >>>>>>>>>>>> root@ip-172-31-60-53:~# spark-submit --class >>>>>>>>>>>> org.apache.spark.examples.SparkPi >>>>>>>>>>>> /tmp/spark-examples-1.2.0-hadoop2.4.0.jar >>>>>>>>>>>> Spark assembly has been built with Hive, including Datanucleus >>>>>>>>>>>> jars on classpath >>>>>>>>>>>> 15/01/30 10:29:33 WARN NativeCodeLoader: Unable to load >>>>>>>>>>>> native-hadoop library for your platform... using builtin-java >>>>>>>>>>>> classes where >>>>>>>>>>>> applicable >>>>>>>>>>>> Pi is roughly 3.14318 >>>>>>>>>>>> >>>>>>>>>>>> root@ip-172-31-60-53:~# >>>>>>>>>>>> >>>>>>>>>>>> I'm now trying to run the same example with the spark-submit >>>>>>>>>>>> '--master' option set to either 'yarn-cluster' or 'yarn-client' >>>>>>>>>>>> but I keep getting the same error : >>>>>>>>>>>> >>>>>>>>>>>> root@ip-172-31-60-53:~# spark-submit --class >>>>>>>>>>>> org.apache.spark.examples.SparkPi --master yarn-client >>>>>>>>>>>> --num-executors 3 --driver-memory 1g --executor-memory 1g >>>>>>>>>>>> --executor-cores 1 --queue thequeue >>>>>>>>>>>> lib/spark-examples*.jar 10 >>>>>>>>>>>> Spark assembly has been built with Hive, including Datanucleus >>>>>>>>>>>> jars on classpath >>>>>>>>>>>> Exception in thread "main" java.lang.Exception: When running >>>>>>>>>>>> with master 'yarn-client' either HADOOP_CONF_DIR or YARN_CONF_DIR >>>>>>>>>>>> must be >>>>>>>>>>>> set in the environment. >>>>>>>>>>>> >>>>>>>>>>>> But on my spark-master/0 machine there is no /etc/hadoop/conf >>>>>>>>>>>> directory. >>>>>>>>>>>> So what should the HADOOP_CONF_DIR or YARN_CONF_DIR value be ? >>>>>>>>>>>> Do I need to add a juju relation between spark-master and ... >>>>>>>>>>>> yarn-hdfs-master to make them aware of each other ? >>>>>>>>>>>> >>>>>>>>>>>> Thanks for any help, >>>>>>>>>>>> >>>>>>>>>>>> Ken >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 28 January 2015 at 19:32, Kapil Thangavelu < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Jan 28, 2015 at 1:54 PM, Ken Williams <[email protected] >>>>>>>>>>>>> > wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Sam/Amir, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I've been able to 'juju ssh spark-master/0' and I >>>>>>>>>>>>>> successfully ran the two >>>>>>>>>>>>>> simple examples for pyspark and spark-shell, >>>>>>>>>>>>>> >>>>>>>>>>>>>> ./bin/pyspark >>>>>>>>>>>>>> >>> sc.parallelize(range(1000)).count() >>>>>>>>>>>>>> 1000 >>>>>>>>>>>>>> >>>>>>>>>>>>>> ./bin/spark-shell >>>>>>>>>>>>>> scala> sc.parallelize(1 to 1000).count() >>>>>>>>>>>>>> 1000 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Now I want to run some of the spark examples in the >>>>>>>>>>>>>> spark-exampes*.jar >>>>>>>>>>>>>> file, which I have on my local machine. How do I copy the jar >>>>>>>>>>>>>> file from >>>>>>>>>>>>>> my local machine to the AWS machine ? >>>>>>>>>>>>>> >>>>>>>>>>>>>> I have tried 'scp' and 'juju scp' from the local command-line >>>>>>>>>>>>>> but both fail (below), >>>>>>>>>>>>>> >>>>>>>>>>>>>> root@adminuser:~# scp >>>>>>>>>>>>>> /tmp/spark-examples-1.2.0-hadoop2.4.0.jar ubuntu@ip-172-31-59 >>>>>>>>>>>>>> :/tmp >>>>>>>>>>>>>> ssh: Could not resolve hostname ip-172-31-59: Name or service >>>>>>>>>>>>>> not known >>>>>>>>>>>>>> lost connection >>>>>>>>>>>>>> root@adminuser:~# juju scp >>>>>>>>>>>>>> /tmp/spark-examples-1.2.0-hadoop2.4.0.jar ubuntu@ip-172-31-59 >>>>>>>>>>>>>> :/tmp >>>>>>>>>>>>>> ERROR exit status 1 (nc: getaddrinfo: Name or service not >>>>>>>>>>>>>> known) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Any ideas ? >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> juju scp /tmp/spark-examples-1.2.0-hadoop2.4.0.jar >>>>>>>>>>>>> spark-master/0:/tmp >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ken >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 28 January 2015 at 17:29, Samuel Cozannet < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Glad it worked! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'll make a merge request to the upstream so that it works >>>>>>>>>>>>>>> natively from the store asap. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks for catching that! >>>>>>>>>>>>>>> Samuel >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>> Samuel >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Samuel Cozannet >>>>>>>>>>>>>>> Cloud, Big Data and IoT Strategy Team >>>>>>>>>>>>>>> Business Development - Cloud and ISV Ecosystem >>>>>>>>>>>>>>> Changing the Future of Cloud >>>>>>>>>>>>>>> Ubuntu <http://ubuntu.com> / Canonical UK LTD >>>>>>>>>>>>>>> <http://canonical.com> / Juju <https://jujucharms.com> >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> mob: +33 616 702 389 >>>>>>>>>>>>>>> skype: samnco >>>>>>>>>>>>>>> Twitter: @SaMnCo_23 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Jan 28, 2015 at 6:15 PM, Ken Williams < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Sam (and Maarten), >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Cloning Spark 1.2.0 from github seems to have worked! >>>>>>>>>>>>>>>> I can install the Spark examples afterwards. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks for all your help! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Yes - Andrew and Angie both say 'hi' :-) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Ken >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 28 January 2015 at 16:43, Samuel Cozannet < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hey Ken, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> So I had a closer look to your Spark problem and found out >>>>>>>>>>>>>>>>> what went wrong. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The charm available on the charmstore is trying to >>>>>>>>>>>>>>>>> download Spark 1.0.2, and the versions available on the >>>>>>>>>>>>>>>>> Apache website are >>>>>>>>>>>>>>>>> 1.1.0, 1.1.1 and 1.2.0. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> There is another version of the charm available on GitHub >>>>>>>>>>>>>>>>> that actually will deploy 1.2.0 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1. On your computer, the below folders & get there: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> cd ~ >>>>>>>>>>>>>>>>> mkdir charms >>>>>>>>>>>>>>>>> mkdir charms/trusty >>>>>>>>>>>>>>>>> cd charms/trusty >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2. Branch the Spark charm. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> git clone https://github.com/Archethought/spark-charm >>>>>>>>>>>>>>>>> spark >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 3. Deploy Spark from local repository >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> juju deploy --repository=~/charms local:trusty/spark >>>>>>>>>>>>>>>>> spark-master >>>>>>>>>>>>>>>>> juju deploy --repository=~/charms local:trusty/spark >>>>>>>>>>>>>>>>> spark-slave >>>>>>>>>>>>>>>>> juju add-relation spark-master:master spark-slave:slave >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Worked on AWS for me just minutes ago. Let me know how it >>>>>>>>>>>>>>>>> goes for you. Note that this version of the charm does NOT >>>>>>>>>>>>>>>>> install the >>>>>>>>>>>>>>>>> Spark examples. The files are present though, so you'll find >>>>>>>>>>>>>>>>> them in >>>>>>>>>>>>>>>>> /var/lib/juju/agents/unit-spark-master-0/charm/files/archive >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hope that helps... >>>>>>>>>>>>>>>>> Let me know if it works for you! >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>> Sam >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>> Samuel >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Samuel Cozannet >>>>>>>>>>>>>>>>> Cloud, Big Data and IoT Strategy Team >>>>>>>>>>>>>>>>> Business Development - Cloud and ISV Ecosystem >>>>>>>>>>>>>>>>> Changing the Future of Cloud >>>>>>>>>>>>>>>>> Ubuntu <http://ubuntu.com> / Canonical UK LTD >>>>>>>>>>>>>>>>> <http://canonical.com> / Juju <https://jujucharms.com> >>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>> mob: +33 616 702 389 >>>>>>>>>>>>>>>>> skype: samnco >>>>>>>>>>>>>>>>> Twitter: @SaMnCo_23 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Wed, Jan 28, 2015 at 4:44 PM, Ken Williams < >>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi folks, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I'm completely new to juju so any help is appreciated. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I'm trying to create a hadoop/analytics-type platform. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I've managed to install the >>>>>>>>>>>>>>>>>> 'data-analytics-with-sql-like' bundle >>>>>>>>>>>>>>>>>> (using this command) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> juju quickstart >>>>>>>>>>>>>>>>>> bundle:data-analytics-with-sql-like/data-analytics-with-sql-like >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> This is very impressive, and gives me virtually >>>>>>>>>>>>>>>>>> everything that I want >>>>>>>>>>>>>>>>>> (hadoop, hive, etc) - but I also need Spark. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The Spark charm ( >>>>>>>>>>>>>>>>>> http://manage.jujucharms.com/~asanjar/trusty/spark) >>>>>>>>>>>>>>>>>> and bundle ( >>>>>>>>>>>>>>>>>> http://manage.jujucharms.com/bundle/~asanjar/spark/spark-cluster >>>>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>>>> however do not seem stable or available and I can't >>>>>>>>>>>>>>>>>> figure out how to install them. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Should I just download and install the Spark tar-ball on >>>>>>>>>>>>>>>>>> the nodes >>>>>>>>>>>>>>>>>> in my AWS cluster, or is there a better way to do this ? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks in advance, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Ken >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> Juju mailing list >>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>> Modify settings or unsubscribe at: >>>>>>>>>>>>>>>>>> https://lists.ubuntu.com/mailman/listinfo/juju >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Juju mailing list >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>> Modify settings or unsubscribe at: >>>>>>>>>>>>>> https://lists.ubuntu.com/mailman/listinfo/juju >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >> -- >> Juju mailing list >> [email protected] >> Modify settings or unsubscribe at: >> https://lists.ubuntu.com/mailman/listinfo/juju >> >> >
-- Juju mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
