Re: is Hdp bundle still maintained/supported?

[Samuel Cozannet]

Hi Patrizio,

My recommendation is that you go to the freenode channel #juju, and then
talk to Kevin (kwmonroe, also cced). He has a lot of wits on these
solutions and will be able to help you, tell you where report bugs and also
see if there is a potential implementation path for your issues

Best,
Sam


--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Tue, Jun 20, 2017 at 2:01 PM, Patrizio Bassi <patrizio.ba...@gmail.com>
wrote:

> Dear Samuel
>
> i deployed the hadoop-processing bundle successfully.
> I had some troubles because openstack instances didn't get the dns entry
> so hadoop-hdfs-namenode/datanode could not resolve their own addresses
> first, then the peers'.
>
> I did a fast hack just manually adding the entries in /etc/hosts while
> waiting for Designate service to be deployed and active.
>
> It would be great if you could add in the documentation the dns need.
>
> This said, i run the smoke tests and they all failed. HDFS storage is
> reported working instead.
> I'm totally newbie in hadoop clusters so i didn't start investigation, but
> i wanted to report in case you may check on your local working copy and
> verify it's fine upstream.
>
> Thank you
>
> Patrizio
>
>
>
>
> 2017-06-16 12:08 GMT+02:00 Patrizio Bassi <patrizio.ba...@gmail.com>:
>
>> Hi Samuel,
>>
>> thank you for fast and honest reply. I will start to take a look at
>> hadoop-processing bundle.
>> Regards,
>>
>> Patrizio
>>
>> 2017-06-16 9:54 GMT+02:00 Samuel Cozannet <samuel.cozan...@canonical.com>
>> :
>>
>>> Hi Patrizio,
>>>
>>> Thanks for reaching out. There is a list of more recent charms based on
>>> Big Top here: https://jujucharms.com/q/bigtop?type=charm
>>> The current strategy is to focus on big top, so products from
>>> Hortonworks are no longer supported as you could notice.
>>>
>>> The team to track is here : https://jujucharms.com/u/bigdata-charmers/
>>>
>>> In addition, there are 2 community teams actively involved in Big Data
>>> solutions via Juju, who can help for big data solutions:
>>> * http://spicule.co.uk/ (on IRC reach out to MagicalTrout)
>>> * http://tengu.intec.ugent.be/v1/ (reach out to
>>> merlijn.sebrec...@gmail.com)
>>>
>>> Best,
>>> Sam
>>>
>>>
>>>
>>>
>>> --
>>> Samuel Cozannet
>>> Cloud, Big Data and IoT Strategy Team
>>> Business Development - Cloud and ISV Ecosystem
>>> Changing the Future of Cloud
>>> Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> /
>>> Juju <https://jujucharms.com>
>>> samuel.cozan...@canonical.com
>>> mob: +33 616 702 389
>>> skype: samnco
>>> Twitter: @SaMnCo_23
>>> [image: View Samuel Cozannet's profile on LinkedIn]
>>> <https://es.linkedin.com/in/scozannet>
>>>
>>> On Fri, Jun 16, 2017 at 9:36 AM, Patrizio Bassi <
>>> patrizio.ba...@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> looking at a hadoop distro in the jujucharms is found
>>>> https://jujucharms.com/hdp-hadoop/trusty/1 charm,
>>>> following the announcement in https://insights.ubuntu.com
>>>> /2015/02/19/ubuntu-hortonworks-and-microsoft-big-data-hosted-solution/
>>>>  .
>>>>
>>>>
>>>> I would like to know if it's still maintained, i don't see commits
>>>> since 2015, version is stuck to trusty for ubuntu and hdp at 2.1.3 while
>>>> upstream is on 2.2 series.
>>>> the Contact Information show a no more valid email address too.
>>>>
>>>> Is Canonical looking at Apache bundle only?
>>>>
>>>> Thank you
>>>>
>>>> Patrizio
>>>>
>>>> --
>>>> Juju mailing list
>>>> Juju@lists.ubuntu.com
>>>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>>>> an/listinfo/juju
>>>>
>>>>
>>>
>>
>>
>> --
>>
>> Patrizio Bassi
>> www.patriziobassi.it
>> http://piazzadelpopolo.patriziobassi.it
>>
>
>
>
> --
>
> Patrizio Bassi
> www.patriziobassi.it
> http://piazzadelpopolo.patriziobassi.it
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: is Hdp bundle still maintained/supported?

[Samuel Cozannet]

Hi Patrizio,

Thanks for reaching out. There is a list of more recent charms based on Big
Top here: https://jujucharms.com/q/bigtop?type=charm
The current strategy is to focus on big top, so products from Hortonworks
are no longer supported as you could notice.

The team to track is here : https://jujucharms.com/u/bigdata-charmers/

In addition, there are 2 community teams actively involved in Big Data
solutions via Juju, who can help for big data solutions:
* http://spicule.co.uk/ (on IRC reach out to MagicalTrout)
* http://tengu.intec.ugent.be/v1/ (reach out to merlijn.sebrec...@gmail.com)

Best,
Sam




--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Fri, Jun 16, 2017 at 9:36 AM, Patrizio Bassi <patrizio.ba...@gmail.com>
wrote:

> Hi all,
>
> looking at a hadoop distro in the jujucharms is found
> https://jujucharms.com/hdp-hadoop/trusty/1 charm,
> following the announcement in https://insights.ubuntu.
> com/2015/02/19/ubuntu-hortonworks-and-microsoft-big-data-hosted-solution/
>  .
>
>
> I would like to know if it's still maintained, i don't see commits since
> 2015, version is stuck to trusty for ubuntu and hdp at 2.1.3 while upstream
> is on 2.2 series.
> the Contact Information show a no more valid email address too.
>
> Is Canonical looking at Apache bundle only?
>
> Thank you
>
> Patrizio
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: un-promulgating big data charms next week

[Samuel Cozannet]

+1 from me.

Quality and community is what makes charms and bundles shine, this is a
good move to keep the best breed.

++
Sam

On Apr 29, 2017 01:34, "Kevin Monroe"  wrote:

> Hey Juju,
>
> TL;DR:  i'm proposing to un-promulgate 45 outdated big data charms/bundles
> [0] next Friday, May 5th.
>
> Earlier this week, Merlijn mentioned improvements that would help users
> searching for things like 'hadoop'.  The gist is that we still list
> outdated charms as 'recommended', even though they are not.  To address
> this, I'm proposing to un-promulgate 15 bundles and 30 charms as shown here:
>
> https://docs.google.com/spreadsheets/d/1TjWl86BTOWKSe8LBNHTQFAwMoeNG5
> sKz_9YdCl0BPkQ/edit?usp=sharing
>
> Each of the 'n' items have either been superseded or are no longer being
> maintained.  These will still be available in the charm store for now, but
> will require a namespace to reach them.  For example:
>
> $ juju deploy apache-pig
>
> becomes:
>
> $ juju deploy ~bigdata-charmers/apache-pig
>
> If removing any of these from the recommended section of the charm store
> will impact your work, please let me know here or as a comment in the doc
> linked above.
>
> Following un-promulgation, I'll put at least some of these through the
> more formal 'unmaintained charm' process, which provides a 30-day window
> for anyone that may want to take up ownership:
>
> https://jujucharms.com/docs/stable/charm-unmaintained-process
>
> I'll send a separate note with details on that effort.
>
> [0] - Items up for un-promulgation:
>
> BUNDLES
> apache-analytics-pig
> apache-analytics-sql
> apache-core-batch-processing
> apache-hadoop-spark
> apache-hadoop-spark-notebook
> apache-hadoop-spark-zeppelin
> apache-ingestion-kafka
> apache-processing-mapreduce
> apache-processing-spark
> data-analytics-with-pig-latin
> data-analytics-with-sql-like
> hdp-core-batch-processing
> high-performance-batch-processing
> kafka-ingestion
> realtime-analytics-with-storm
>
> CHARMS
> apache-flume-hdfs
> apache-flume-kafka
> apache-flume-syslog
> apache-flume-twitter
> apache-hadoop-client
> apache-hadoop-compute-slave
> apache-hadoop-hdfs-master
> apache-hadoop-hdfs-secondary
> apache-hadoop-namenode
> apache-hadoop-plugin
> apache-hadoop-resourcemanager
> apache-hadoop-slave
> apache-hadoop-yarn-master
> apache-hive
> apache-kafka
> apache-pig
> apache-spark
> apache-spark-notebook
> apache-zeppelin
> apache-zookeeper/4
> gobblin
> hdp-hadoop
> hdp-hive
> hdp-pig
> hdp-storm
> hdp-tez
> hdp-zookeeper
>
> Please let me know if you have questions/concerns. Thanks!
> -Kevin
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: CDK + Deis

[Samuel Cozannet]

Hi James,

Deis is definitely on my map, I am happy to participate in the effort.
Afaik, the 2 main issues are
* getting Postgresql and Minio to have persistent storage.
* Managing the expectation of ELB vs Ingress.

This is pretty easy on AWS, I guess a little less on the metal bc LB. Not
sure on GCE and azure.

Any hints from your previous efforts? Regarding AWS integration (I believe
it is your primary target, I did publish a post on medium about it, would
love feedback)

++
Sam

On Apr 21, 2017 11:16 PM, "James Beedy"  wrote:

> Wondering if there is any interest out in running Deis on top of CDK?
>
> I was able to get Deis working atop CDK with some elbow grease and some
> inside info from @lazyPower a while back, but it was a bit rough.
>
> One of the departments in my company insists on running/deploying their
> apps using Deis, so I will be putting a bit more effort into finding a path
> forward for here in hopes we can get CDK running underneath.
>
> If anyone is interested in, or has done work with Deis + CDK give me a
> shout and lets match notes.
>
> I'm going to be giving it another go with the latest CDK and the latest
> Deis here this weekend/next week, hopefully I can come up with some solid
> docs at least.
>
> I'll report back with my findings~
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Kubernetes charms now support 1.6!

[Samuel Cozannet]

Yooohoo!! Congrats!

I tested the GPU deployment and it worked flawlessly. I have another team
using it now, also worked OOTB.

I'll be updating all my blog posts to outline the changes.

Good job on snaps as well, it makes reconfiguration of the cluster so much
easier. I intend to do a short write up about this in a near future.

++
Sam




--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Thu, Apr 13, 2017 at 12:34 AM, Matt Bruzek <matthew.bru...@canonical.com>
wrote:

> We are proud to release the latest Charms supporting Kubernetes version
> 1.6.1!
>
>
> Kubernetes 1.6 is a major milestone for the community, we’ve got a full
> write up of features and support on our blog <https://insights.ubuntu.com/
> 2017/04/12/general-availability-of-kubernetes-1-6-on-ubuntu/>
> Getting Started
>
> Here’s the simplest way to get a Kubernetes cluster up and running on an
> Ubuntu 16.04 system:
>
> sudo snap install conjure-up --classic
> conjure-up kubernetes
>
>
> During the installation conjure-up will ask you what cloud you want to
> deploy on and prompt you for the proper credentials. If you’re deploying to
> local containers (LXD) see these instructions <https://kubernetes.io/docs/
> getting-started-guides/ubuntu/local/> for localhost-specific
> considerations.
>
> For production grade deployments and cluster lifecycle management it is
> recommended to read the full Canonical Distribution of Kubernetes
> documentation <https://kubernetes.io/docs/getting-started-guides/ubuntu/>.
> Upgrading an existing cluster
>
> If you’ve got a cluster already deployed, we’ve got instructions to help
> get you upgraded. If possible, deploying a new cluster will be the easiest
> route. Otherwise, the instructions for upgrading are outlined here:
> https://insights.ubuntu.com/2017/04/12/general-
> availability-of-kubernetes-1-6-on-ubuntu/#upgrades
> Changes in this release
>
>-
>
>Support for Kubernetes v1.6, with the current release being 1.6.1
>-
>
>Installation of components via snaps: kubectl, kube-apiserver,
>kube-controller-manager, kube-scheduler, kubelet, and kube-proxy. To learn
>more about snaps: https://snapcraft.io
>-
>
>Added ‘allow-privileged’ config option on kubernetes-master and
>kubernetes-worker charms. Valid values are true|false|auto (default: auto).
>If the value is ‘auto’, containers will run in unprivileged mode unless GPU
>hardware is detected on a worker node. If there are GPUs, or the value is
>true, Kubernetes will set `--allow-privileged=true`. Otherwise the flag is
>set to false.
>-
>
>Added GPU support (beta). If Nvidia GPU hardware is detected on a
>worker node, Nvidia drivers and CUDA packages will be installed, and
>kubelet will be restarted with the flags required to use the GPU hardware.
>The ‘allow-privileged’ config option must be ‘true’ or ‘auto’.
>-
>
>   Nvidia driver version = 375.26; CUDA version = 8.0.61; these will
>   be configurable future charm releases.
>   -
>
>   GPU support does not currently work on lxd.
>   -
>
>   This feature is beta - feedback on the implementation is welcomed.
>   -
>
>Added support for running your own private registry, see the docs here
>
> <https://github.com/juju-solutions/kubernetes/tree/1.6-staging/cluster/juju/layers/kubernetes-worker#private-registry>
>for instructions.
>
> General Fixes:
>
>-
>
>Fixed a bug in the kubeapi-load-balancer not properly forwarding
>SPDY/HTTP2 traffic for `kubectl exec` commands.
>
> Etcd specific changes:
>
>-
>
>Installation of etcd and etcdctl is now done using the `snap install`
>command.
>-
>
>We support upgrading the previous etcd charm, to the latest charm with
>snap delivery mechanism.  See manual upgrade process for updating existing
>etcd clusters.
>
> Changes to the bundles and layers:
>
>-
>
>Add registry action to the kubernetes-worker layer, which deploys a
>Docker registry in Kubernetes.
>-
>
>Add support for kube-proxy cluster-cidr option.
>
> Test results
>
> The Canonical Distribution of Kubernetes is running daily tests to verify
> it works with the upstream code. As part of the Kubernetes test
>

Re: Goodbye Opsworks

[Samuel Cozannet]

Whaow congrats! Do you have a blog post in the work for this explaining the
process?

Best,
Sam


--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Sat, Apr 1, 2017 at 3:06 AM, James Beedy <jamesbe...@gmail.com> wrote:

> The day has finally come for me to turn down the last of our Opsworks
> instances for our PRM application. This marks the completion of one of many
> Opsworks -> Juju conversion projects I've taken on. Thanks everyone for
> your help along the way!
>
> Goodbye Opsworks - http://imgur.com/a/4pkgP
>
> Hello Juju PRM!
>
> Staging - http://paste.ubuntu.com/24291143/
> Demo - http://paste.ubuntu.com/24291156/
> Production - http://paste.ubuntu.com/24291133/
> Walmart - http://paste.ubuntu.com/24291173/
>
> W00T!
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Can't juju deploy p2.xlarge in aws/us-east-1

[Samuel Cozannet]

Isn't this spending time on a problem that will tend to disappear (and is
probably already only like 0.1% of the user base)?

Not sure it's worth the effort. Just specify Juju requires a VPC in the
docs, and explain exactly how to fix it (call AWS...) and there you go. No
code, problem solved.

++
Sam


--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Tue, Mar 21, 2017 at 12:04 AM, Andrew Wilkins <
andrew.wilk...@canonical.com> wrote:

> On Mon, Mar 20, 2017 at 10:17 PM Junien Fridrick <
> junien.fridr...@canonical.com> wrote:
>
>> And once/if you have no Classic instance running in a region, you should
>> file a ticket to AWS Support asking for a default VPC creation. Once
>> it's done, you don't need to supply the VPC ID anymore.
>>
>> Could juju maybe be instructed to use a VPC if and only if a single VPC
>> exist in a region, without specifying its ID ? For example, --config
>> "force-vpc=true" or something.
>>
>
> Yes we could do that. I'd like to go a bit further; what I would like to
> have happen is:
>
> 1. juju will use the VPC specified in config, if any
> 2. else, Juju will look for a VPC called (say) "juju-default-vpc"
> 3. else, Juju will create a VPC with the same name, if it can*, and use
> that
> 4. else, Juju will use the default VPC, if available
> 5. else, bootstrap/add-model will fail
>
> The outcome being that Juju will then *always* use VPC, and the user
> shouldn't need to do a thing. This would also allow the provider to be
> cleaned up (assuming there's a migration path for existing deployments),
> and new features to be implemented more easily (e.g. support for EFS).
>
> Everyone on the team is currently fully booked, so I can't give you an
> estimate of when that'll come to fruition yet. We could do everything
> except have Juju fail pretty easily I would think.
>
> * "if it can", because there are pretty severe limits on the number of
> VPCs you can create: 5 per region.
>
> Cheers,
> Andrew
>
>
>> On Thu, Mar 16, 2017 at 08:04:22AM -0400, Tim Van Steenburgh wrote:
>> > A. That was it. After passing --config "vpc-id=vpc-924fc7f6" to
>> `juju
>> > bootstrap`, I can now deploy p2 instance types. Thanks Andrew!
>> >
>> > On Thu, Mar 16, 2017 at 4:21 AM, Samuel Cozannet <
>> > samuel.cozan...@canonical.com> wrote:
>> >
>> > > Aaah whaow. I have a default VPC myself, so that may explain the
>> problem
>> > > Tim is having. Early adopters problem!!
>> > >
>> > >
>> > > --
>> > > Samuel Cozannet
>> > > Cloud, Big Data and IoT Strategy Team
>> > > Business Development - Cloud and ISV Ecosystem
>> > > Changing the Future of Cloud
>> > > Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com>
>> /
>> > > Juju <https://jujucharms.com>
>> > > samuel.cozan...@canonical.com
>> > > mob: +33 616 702 389
>> > > skype: samnco
>> > > Twitter: @SaMnCo_23
>> > > [image: View Samuel Cozannet's profile on LinkedIn]
>> > > <https://es.linkedin.com/in/scozannet>
>> > >
>> > > On Thu, Mar 16, 2017 at 9:17 AM, Andrew Wilkins <
>> > > andrew.wilk...@canonical.com> wrote:
>> > >
>> > >> On Thu, Mar 16, 2017 at 3:57 PM Samuel Cozannet <
>> > >> samuel.cozan...@canonical.com> wrote:
>> > >>
>> > >>> I am using the default settings, no change as far as I know to what
>> Juju
>> > >>> would do by default.
>> > >>>
>> > >>
>> > >> What Juju will do depends on what is available in your EC2 account.
>> Not
>> > >> all EC2 accounts were born alike.
>> > >>
>> > >> If your account has a default VPC, that will be used by Juju. In that
>> > >> case, you'll have p2 instance types available. I expect this to be
>> the case
>> > >> for most people - all accounts created since 2013-12-04 will have a
>> default
>> > >> VPC.
>> > >>
>> > >> If you've got an older account, then you may or may not have a
&

Re: Can't juju deploy p2.xlarge in aws/us-east-1

[Samuel Cozannet]

Aaah whaow. I have a default VPC myself, so that may explain the problem
Tim is having. Early adopters problem!!


--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Thu, Mar 16, 2017 at 9:17 AM, Andrew Wilkins <
andrew.wilk...@canonical.com> wrote:

> On Thu, Mar 16, 2017 at 3:57 PM Samuel Cozannet <
> samuel.cozan...@canonical.com> wrote:
>
>> I am using the default settings, no change as far as I know to what Juju
>> would do by default.
>>
>
> What Juju will do depends on what is available in your EC2 account. Not
> all EC2 accounts were born alike.
>
> If your account has a default VPC, that will be used by Juju. In that
> case, you'll have p2 instance types available. I expect this to be the case
> for most people - all accounts created since 2013-12-04 will have a default
> VPC.
>
> If you've got an older account, then you may or may not have a default
> VPC. If you do not, then Juju will fall back to EC2 Classic. In that case,
> no p2 instance types.
>
> Cheers,
> Andrew
>
>
>> --
>> Samuel Cozannet
>> Cloud, Big Data and IoT Strategy Team
>> Business Development - Cloud and ISV Ecosystem
>> Changing the Future of Cloud
>> Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> /
>> Juju <https://jujucharms.com>
>> samuel.cozan...@canonical.com
>> mob: +33 616 702 389
>> skype: samnco
>> Twitter: @SaMnCo_23
>> [image: View Samuel Cozannet's profile on LinkedIn]
>> <https://es.linkedin.com/in/scozannet>
>>
>> On Thu, Mar 16, 2017 at 8:52 AM, Andrew Wilkins <
>> andrew.wilk...@canonical.com> wrote:
>>
>> On Tue, Mar 14, 2017 at 8:48 PM Tim Van Steenburgh <
>> tim.van.steenbu...@canonical.com> wrote:
>>
>> 2.1.1 juju client and controller, controller bootstrapped in
>> aws/us-east-1:
>>
>> juju deploy ./kubernetes-worker --constraints "instance-type=p2.xlarge" 
>> kubernetes-worker-gpu
>> Deploying charm "local:xenial/kubernetes-worker-1".
>> ERROR cannot add application "kubernetes-worker-gpu": invalid constraint 
>> value: instance-type=p2.xlarge
>> valid values are: [m1.small cc2.8xlarge cr1.8xlarge g2.2xlarge r3.8xlarge 
>> i2.xlarge t1.micro c1.xlarge g2.8xlarge m3.xlarge m3.medium c3.4xlarge 
>> hs1.8xlarge r3.2xlarge m1.xlarge c3.xlarge c3.large c3.8xlarge r3.xlarge 
>> m2.xlarge m1.large i2.2xlarge i2.8xlarge cg1.4xlarge d2.2xlarge m2.2xlarge 
>> m3.2xlarge hi1.4xlarge m2.4xlarge r3.4xlarge r3.large d2.xlarge c1.medium 
>> d2.8xlarge m3.large m1.medium c3.2xlarge i2.4xlarge d2.4xlarge]
>>
>> Are you using VPC? p2 instance types only support VPC.
>>
>> I /am/ able to deploy a p2.xlarge in aws/us-east-1 using the AWS console. 
>> Looking at the code it seems this instance-type should be available: 
>> https://github.com/juju/juju/blob/juju-2.1.1/provider/ec2/internal/ec2instancetypes/generated.go#L6165
>>
>> Not sure if this is a bug or PEBKAC. Grateful for any ideas while I continue 
>> to poke at it.
>>
>>
>> Tim
>>
>> --
>> Juju mailing list
>> Juju@lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/
>> mailman/listinfo/juju
>>
>>
>> --
>> Juju mailing list
>> Juju@lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/
>> mailman/listinfo/juju
>>
>>
>>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Can't juju deploy p2.xlarge in aws/us-east-1

[Samuel Cozannet]

I am using the default settings, no change as far as I know to what Juju
would do by default.


--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Thu, Mar 16, 2017 at 8:52 AM, Andrew Wilkins <
andrew.wilk...@canonical.com> wrote:

> On Tue, Mar 14, 2017 at 8:48 PM Tim Van Steenburgh <
> tim.van.steenbu...@canonical.com> wrote:
>
>> 2.1.1 juju client and controller, controller bootstrapped in
>> aws/us-east-1:
>>
>> juju deploy ./kubernetes-worker --constraints "instance-type=p2.xlarge" 
>> kubernetes-worker-gpu
>> Deploying charm "local:xenial/kubernetes-worker-1".
>> ERROR cannot add application "kubernetes-worker-gpu": invalid constraint 
>> value: instance-type=p2.xlarge
>> valid values are: [m1.small cc2.8xlarge cr1.8xlarge g2.2xlarge r3.8xlarge 
>> i2.xlarge t1.micro c1.xlarge g2.8xlarge m3.xlarge m3.medium c3.4xlarge 
>> hs1.8xlarge r3.2xlarge m1.xlarge c3.xlarge c3.large c3.8xlarge r3.xlarge 
>> m2.xlarge m1.large i2.2xlarge i2.8xlarge cg1.4xlarge d2.2xlarge m2.2xlarge 
>> m3.2xlarge hi1.4xlarge m2.4xlarge r3.4xlarge r3.large d2.xlarge c1.medium 
>> d2.8xlarge m3.large m1.medium c3.2xlarge i2.4xlarge d2.4xlarge]
>>
>> Are you using VPC? p2 instance types only support VPC.
>
>> I /am/ able to deploy a p2.xlarge in aws/us-east-1 using the AWS console. 
>> Looking at the code it seems this instance-type should be available: 
>> https://github.com/juju/juju/blob/juju-2.1.1/provider/ec2/internal/ec2instancetypes/generated.go#L6165
>>
>> Not sure if this is a bug or PEBKAC. Grateful for any ideas while I continue 
>> to poke at it.
>>
>>
>> Tim
>>
>> --
>> Juju mailing list
>> Juju@lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/
>> mailman/listinfo/juju
>>
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Can't juju deploy p2.xlarge in aws/us-east-1

[Samuel Cozannet]

TBH I can't remember which juju version I have been using, but I tend to be
on the edge of 16.10, so the latest there is what I have.

I deployed like 5 times yesterday and 2 this morning, so for me no
problemas at all.

On Mar 14, 2017 14:31, "Tim Van Steenburgh" <
tim.van.steenbu...@canonical.com> wrote:

Yeah, and it also doesn't list any p2 types in the list of valid values.
I'm not sure why. IIRC, 2.1 introduced a querying-clouds-for-instance-types
feature - I wonder if that could be related.

I might just drop back to 2.0 and see if it works there. Sam, curious if
you've tried a p2 with 2.1?

On Tue, Mar 14, 2017 at 9:09 AM, Samuel Cozannet <
samuel.cozan...@canonical.com> wrote:

> It is weird, it fails because it doesn't recognize the instance type in
> the constraint.
>
> I haven't seen that in the past, though to spin more than one you need to
> ask to upgrade your quota.
>
> ++
> Sam
>
> On Mar 14, 2017 13:48, "Tim Van Steenburgh" <tim.van.steenburgh@canonical.
> com> wrote:
>
>> 2.1.1 juju client and controller, controller bootstrapped in
>> aws/us-east-1:
>>
>> juju deploy ./kubernetes-worker --constraints "instance-type=p2.xlarge" 
>> kubernetes-worker-gpu
>> Deploying charm "local:xenial/kubernetes-worker-1".
>> ERROR cannot add application "kubernetes-worker-gpu": invalid constraint 
>> value: instance-type=p2.xlarge
>> valid values are: [m1.small cc2.8xlarge cr1.8xlarge g2.2xlarge r3.8xlarge 
>> i2.xlarge t1.micro c1.xlarge g2.8xlarge m3.xlarge m3.medium c3.4xlarge 
>> hs1.8xlarge r3.2xlarge m1.xlarge c3.xlarge c3.large c3.8xlarge r3.xlarge 
>> m2.xlarge m1.large i2.2xlarge i2.8xlarge cg1.4xlarge d2.2xlarge m2.2xlarge 
>> m3.2xlarge hi1.4xlarge m2.4xlarge r3.4xlarge r3.large d2.xlarge c1.medium 
>> d2.8xlarge m3.large m1.medium c3.2xlarge i2.4xlarge d2.4xlarge]
>>
>>
>> I /am/ able to deploy a p2.xlarge in aws/us-east-1 using the AWS console. 
>> Looking at the code it seems this instance-type should be available: 
>> https://github.com/juju/juju/blob/juju-2.1.1/provider/ec2/internal/ec2instancetypes/generated.go#L6165
>>
>> Not sure if this is a bug or PEBKAC. Grateful for any ideas while I continue 
>> to poke at it.
>>
>>
>> Tim
>>
>>
>> --
>> Juju mailing list
>> Juju@lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>> an/listinfo/juju
>>
>>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Can't juju deploy p2.xlarge in aws/us-east-1

[Samuel Cozannet]

It is weird, it fails because it doesn't recognize the instance type in the
constraint.

I haven't seen that in the past, though to spin more than one you need to
ask to upgrade your quota.

++
Sam

On Mar 14, 2017 13:48, "Tim Van Steenburgh" <
tim.van.steenbu...@canonical.com> wrote:

> 2.1.1 juju client and controller, controller bootstrapped in aws/us-east-1:
>
> juju deploy ./kubernetes-worker --constraints "instance-type=p2.xlarge" 
> kubernetes-worker-gpu
> Deploying charm "local:xenial/kubernetes-worker-1".
> ERROR cannot add application "kubernetes-worker-gpu": invalid constraint 
> value: instance-type=p2.xlarge
> valid values are: [m1.small cc2.8xlarge cr1.8xlarge g2.2xlarge r3.8xlarge 
> i2.xlarge t1.micro c1.xlarge g2.8xlarge m3.xlarge m3.medium c3.4xlarge 
> hs1.8xlarge r3.2xlarge m1.xlarge c3.xlarge c3.large c3.8xlarge r3.xlarge 
> m2.xlarge m1.large i2.2xlarge i2.8xlarge cg1.4xlarge d2.2xlarge m2.2xlarge 
> m3.2xlarge hi1.4xlarge m2.4xlarge r3.4xlarge r3.large d2.xlarge c1.medium 
> d2.8xlarge m3.large m1.medium c3.2xlarge i2.4xlarge d2.4xlarge]
>
>
> I /am/ able to deploy a p2.xlarge in aws/us-east-1 using the AWS console. 
> Looking at the code it seems this instance-type should be available: 
> https://github.com/juju/juju/blob/juju-2.1.1/provider/ec2/internal/ec2instancetypes/generated.go#L6165
>
> Not sure if this is a bug or PEBKAC. Grateful for any ideas while I continue 
> to poke at it.
>
>
> Tim
>
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Can't juju ssh to one of my units

[Samuel Cozannet]

Hi Derek,

You need to upgrade your Juju to 2.0.3 at least. This is a known issue (I
am all thumbs so can't find the LP ref right now but there are a bunch of
tickets about it)

To do so, first upgrade your controller:

sudo apt update && sudo apt upgrade
juju switch controller
juju upgrade-juju

Then upgrade units of the model

juju switch 
juju upgrade-juju

If this doesn't work out for you, I had success with upgrading a controller
to 2.1-beta5 by adding the devel ppa, upgrading locally, upgrading the
controller, then the units.


Let us know how it gets for you,
Best
Sam

On Feb 13, 2017 21:52, "Derek DeMoss"  wrote:

> Hey All,
> I'm getting a host key verification failure when I try to juju ssh to one
> of my units.
>
> username@os-nuc-01:~/.ssh$ juju ssh  postgresql/14 -v
> OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g  1 Mar 2016
> debug1: Reading configuration data /etc/ssh/ssh_config
> debug1: /etc/ssh/ssh_config line 19: Applying options for *
> debug1: Connecting to  [] port 22.
> debug1: Connection established.
> debug1: identity file /home/username/.local/share/juju/ssh/juju_id_rsa
> type 1
> debug1: key_load_public: No such file or directory
> debug1: identity file /home/username/.local/share/juju/ssh/juju_id_rsa-cert
> type -1
> debug1: identity file /home/username/.ssh/id_rsa type 1
> debug1: key_load_public: No such file or directory
> debug1: identity file /home/username/.ssh/id_rsa-cert type -1
> debug1: Enabling compatibility mode for protocol 2.0
> debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
> debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2
> Ubuntu-4ubuntu2.1
> debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.1 pat OpenSSH* compat
> 0x0400
> debug1: Authenticating to :22 as 'ubuntu'
> debug1: SSH2_MSG_KEXINIT sent
> debug1: SSH2_MSG_KEXINIT received
> debug1: kex: algorithm: curve25519-sha...@libssh.org
> debug1: kex: host key algorithm: ecdsa-sha2-nistp256
> debug1: kex: server->client cipher: chacha20-poly1...@openssh.com MAC:
>  compression: none
> debug1: kex: client->server cipher: chacha20-poly1...@openssh.com MAC:
>  compression: none
> debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
> debug1: Server host key: ecdsa-sha2-nistp256 SHA256:
> th86nMHQbr0RyuXXnsrBbeOs2JkchFl6gXVBY7p6S2k
> @@@
> @WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
> @@@
> IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
> Someone could be eavesdropping on you right now (man-in-the-middle attack)!
> It is also possible that a host key has just been changed.
> The fingerprint for the ECDSA key sent by the remote host is
> SHA256:th86nMHQbr0RyuXXnsrBbeOs2JkchFl6gXVBY7p6S2k.
> Please contact your system administrator.
> Add correct host key in /tmp/ssh_known_hosts736182584 to get rid of this
> message.
> Offending RSA key in /tmp/ssh_known_hosts736182584:7
>   remove with:
>   ssh-keygen -f "/tmp/ssh_known_hosts736182584" -R 
> ECDSA host key for  has changed and you have requested strict
> checking.
> Host key verification failed.
>
>
> If I just do: ssh ubuntu@ it connects just fine.
>
> So my question is, where is Juju getting the original for
> /tmp/ssh_known_hosts736182584:7
>
> I've tried searching a bunch on my juju-controller machine, but I haven't
> been able to find anything that looks like the problematic known_hosts
> file...
>
> All the other machines/units are juju ssh-able, and I have no idea why
> this one would be different [other than the drive filled up from
> postgresql, and I had to clear some WAL files before things got running
> again].
>
> Any insight or help would be appreciated!
> Derek DeMoss
> Dark Horse Comics, Inc.
> System Administrator I
> "Truth...  Is where you seek it."-Lomar
>
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: [ANNOUNCE] Canonical Distribution of Kubernetes - Release 1.5.2

[Samuel Cozannet]

Awesome, thanks Chuck!!

Can't wait to try this out :)

++
Sam

On Fri, Jan 20, 2017 at 1:01 AM, Charles Butler <
charles.but...@canonical.com> wrote:

> other members of the Kubernetes community





--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: [ANN] Canonical Distribution of Kubernetes Release Notes [week of 10/13]

[Samuel Cozannet]

This is awesome, thanks for the update.

Also, you forgot to mention the addition of the NFS packages, which means
CDK can now consume NFS storage OOTB in addition to having Ceph as a
backend.

Great work folks :)

++
Sam


--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Thu, Oct 13, 2016 at 9:53 PM, Charles Butler <
charles.but...@canonical.com> wrote:

> 10/13 Release Notes
>
> Greetings everyone! It's been a short but busy week for us.  We've landed
> a lot more bugfixes and some new features for you all to kick the tires.
> We're excited to push this week’s rollup as it contains the early work
> (alpha?) in consuming Ceph RBD volumes for persistent volume storage in
> your kubernetes workloads.
>
> It’s missing from the readme, so here is a quick rundown below the release
> notes.  As always, bugs/comments/questions are all welcome.
> https://github.com/juju-solutions/bundle-canonical-kubernetes/issues
>
> Or you can find us on irc in #juju on irc.freenode.net
>
>
> Layer-docker
>
>
>-
>
>Added DOCKER_OPTS passthrough config option. This enables end users to
>configure the runtime of their docker-engine (Such as insecure registries)
>without having to add python code to the layers and/or re-build a fork.
>
>
>-
>
>Corrected an immutable config when attempting to switch between
>archive docker package and the docker-engine package from upstream.
>
>
> Thanks @brianlbaird and @simonklb for driving this feature during
> dev/testing.
>
> Flannel
>
>
>-
>
>Corrected the directory glob pattern on flannel which was failing and
>causing some false positives in the cloud weather report testing tool.
>
>
> Kubernetes Master
>
>-
>
>Added a create-rbd-pv action to enlist persistent storage from Ceph.
>-
>
>   This requires the use of the ceph-mon charm from the
>   openstack-charmers-next branch.
>   -
>
>Closed a bug where running microbots would yield an EOF error due to
>proxy settings. Consult the README for limited egress environments. (Thanks
>@ryebot and @cynerva)
>
>
> Kubernetes Worker
>
>-
>
>Added a kubectl wrapper for context with manifests, and a kubectl
>wrapper for arbitrary keyword args.
>-
>
>Various lint fixes.
>-
>
>Worker nodes now cleanly remove themselves from the cluster during the
>stop hook. (Thanks to @ryebot and @cynerva)
>-
>
>The ingress controller now scales to the number of deployed worker
>units. 1 ingress controller per 1 worker unit. (Thanks to @ryebot and
>@cynerva)
>
>
> Canonical Distribution of Kubernetes Bundle
>
>
>-
>
>Added documentation for proxy settings in network limited environments
>to the bundle README. (Thanks to @ryebot and @cynerva)
>-
>
>Updated the README with additional limitation notes about which charms
>are not compatible enough to run in LXD at this time.
>-
>
>Bumped each charm to their latest revision.
>
>
> Kubernetes Core Bundle
>
> A minimalist bundle, only deploying the bare minimum required to evaluate
> kubernetes. Useful when doing laptop development or resource constrained
> environments. (Thanks @cynerva and @ryebot)
>
>
>
>-
>
>The kubernetes core bundle has been updated with our latest release of
>the Canonical Distribution of Kubernetes (CDK) charms.
>-
>
>Brand new README imported from the CDK bundle.
>
>
> https://github.com/juju-solutions/bundle-kubernetes-core  - we’re still
> testing this minimal bundle, and it will be published in the charm store as
> early as next week. Thanks for your early interest!
>
> Etcd
>
>-
>
>Refactored the test to gate on the status messages before reading the
>deployment as ready and proceeding with executing tests.
>
>
>
> Quick Rundown on how to enlist RBD PV’s
>
> You’ll need to be running at bare-minimum the ceph-mon charm from the
> ~openstack-charmers-next namespace.
>
> juju deploy cs:~openstack-charmers-next/xenial/ceph-mon -n 3
>
> juju deploy cs:ceph-osd -n 3
>
> From here you will need to enlist the OSD storage devices:
>
> For example on GCE you would fulfill this request with GCE Persistent
> Disks:
>
>

Re: Reliably share a persistent filesystem between units

[Samuel Cozannet]

Hi Robin,

Interesting question, I'm expecting a lot of answers and comments :). Here
are mines:

* If the volume of data is small and its life expectancy is long, with few
writes, but potentially lots of reads (like config files for example), you
may want to use a simple system like etcd or consul. They are primarily
used as service discovery, but this can also be a valid use case for the
technology.
* If you need faster or bigger data that you can represent as KV, Redis
would provide you with the speed, and some architectures offer HA though it
doesn't scale much by default at least in write mode. Here you would
essentially use redis with key = $(cat document)
* If you absolutely need file system storage,
  * There are a few p2p sync technologies like https://getsync.com/ that
would give you this functionality
  * I wouldn't focus on NFS being the PoF. Actually, this is not your
problem, but the storage provider's problem to provide a valid, reachable
endpoint for storage. Your app shall not be aware of the underlying
storage, and NFS is a service you just consume. The shining new EFS on
Amazon is a distributed storage system with an NFS endpoint. Ceph can offer
NFS endpoints as well, and so do a lot of storage backends. So NFS is just
a way to present a mounted file system to the network.
  * S3 offer a lot of tooling to sync files, so if speed, latency and cost
are not a problem, you may want to use that. It however has the drawback of
requiring secrets.
  * Samba is also an option eventually.
* If your workloads is containerized, ClusterHQ Flocker is offering a semi
viable option for container storage, where they allocate cloud block
storage to containers and manage resiliency. As most clouds offer block
storage backup / snapshot, you would get the feature you want. However,
Flocker currently has a SPOF itself as the master is not HA.

My 2 cents,
Best,
Sam




--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Thu, Jul 7, 2016 at 12:02 PM, Robin Winslow <ro...@canonical.com> wrote:

> Does anyone know of the best way to share a folder between Juju units in a
> persistent and reliable way?
>
> Up until now we've been dealing with shared data that needs persistence by
> storing it in Swift, and so I have written my applications to interact
> with a Swift server directly (and therefore the need to be provided with
> the Swift credentials).
>
> However, I now have a situation where it would be much neater if the
> application could simply interact with a local folder without any knowledge
> of the underlying storage system. So I need a way to have the data in that
> folder reliably shared between all the application units with Juju, and
> also persisted somewhere outside the deployment in case the environment is
> destroyed.
>
> I assume the sharing could be simply achieved using NFS
> <https://jujucharms.com/nfs/> or similar, but AFAIK that doesn't in and
> of itself provide any redundancy or help with persisting the data.
>
> Has anyone done anything like this?
>
> From looking into it, it looks like CephFS
> <http://docs.ceph.com/docs/master/cephfs/> might do what I want (and it
> looks like
> <http://docs.ceph.com/docs/master/cephfs/best-practices/#which-ceph-version>
> it became officially "stable" in the Jewel release
> <http://ceph.com/releases/v10-2-2-jewel-released/> on June 15th),
> allowing me to mount a remote Ceph setup at a specific folder within my
> unit (correct me if I'm wrong here). However I have no prior experience
> with setting up or interacting with Ceph either within or outside Juju.
>
> Has anyone implemented CephFS before? Does anyone know if it can be done
> with existing Juju charms? If not, I'd be happy to try writing something,
> although Ceph will be a significant learning curve for me. It looks like
> it needs a Ceph Metadata Server (MDS) which I couldn't see a charm for
> <https://jujucharms.com/q/ceph>.
>
> Or is there a simpler solution that makes sense? Perhaps I'm
> overestimating the problems with using the NFS charm for this?
>
> Any help would be much appreciated. Thanks.
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Docker compose in Juju

[Samuel Cozannet]

Sort of yeah.
You'll have to manage collocation of several charms on the units to run
several containers on the same instance. The Docker layer will take care of
avoiding to replicate install several times though, so the second charm you
deploy will essentially be docker pull / docker run. You can actually force
this behavior in the layer options (do not install docker)
I haven't played that much with the latest versions of the Docker layer, so
I'll let Chuck pick up this one, but some stuff I think you'll need to take
into account:

* Ports / networking: containers usually expose a network port. If you run
several colocated Docker charms, you will have to manage yourself the fact
the ports don't overlap. That means your install section will also have to
be clever about the ports it opens in the cloud fw.
* Relations: app containers are supposed to be orchestrated. By themselves,
they know nothing about the state of the rest of the world. Also, in a Juju
bundle, you don't have a warranty on the order in which they will spin. So
you will have to create a relation logic that reacts to the availability of
other components to, for example, reload a docker image with the new set of
env variables, or just make sure such container waits for others before it
runs. Essentially, you'll have to recreate the notion of dependency you may
have experienced with Fleet for example.
* Scale out:
  * Juju so far doesn't have a notion of ELB or other similar cloud level
load balancer. So if you run your containers in a cluster, and spawn
several replicas, that won't be managed by Juju either, you'll have to do
the load balancing yourself
  * The way Juju scales out today is by adding units/instances. In the case
of Docker containers, this can be tricky, as you are actually spinning more
containers, but in a limited cluster. You don't necessarily want to scale
that cluster. So my recommendation is not to use the default juju add-unit,
but rather add a config option in the charm to scale out the container,
which you'll trigger via "juju set config-option"
* Storage / resiliency: Not sure if we have a story to integrate storage
resiliency yet. If you need to use stateful apps, I'd recommend using a
classic charm and use Juju's abstractions for storage. Again, I haven't
used the latest Docker stuff in Juju, maybe Chuck has better insight.

I'd personally be interested in your scale out usage, and if you usually
consume helpers like Flocker, or Weave for the storage and network layers.
Any info would be appreciated.

++
Sam


--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Tue, Jun 7, 2016 at 7:30 AM, Gayan Gunarathne <csega...@gmail.com> wrote:

> Hi Samuel,
>
> Thank you very much for the details.
>
> As you mention if we wrap the Docker image in the Charm, it is just like a
> single instance.Isn't? It just like that I run this docker containers in a
> single instance in IaaS like EC2, GCE etc. Within this instance it will
> spawn the containers. So for my deployment I need to manager it within the
> docker image in the Charm.Isn't?
>
> Thanks,
> Gayan
>
>
>
>
>
> On Mon, Jun 6, 2016 at 7:39 PM, Samuel Cozannet <
> samuel.cozan...@canonical.com> wrote:
>
>> Hi Gayan,
>>
>> Juju cannot operate with app containers as it operates with "os-based"
>> systems.
>> The difference is that Juju expects the node it spins to run a Juju
>> agent. In the case of a Docker container, which runs a single process, that
>> would mean the sole purpose of the container is running... Juju itself. So
>> long for your app.
>> So that means Juju cannot provision Docker containers "directly" (which
>> is what you seem to expect with the provider idea).
>>
>> If you want to run Docker containers via Juju, you have the following
>> methods:
>> * Wrap the Docker image in the charm. Essentially, the charm would then
>> be a set of commands to run, scale, integrate... the container with the
>> rest of the world (for example, managing the fact that the host port is not
>> necessarily the same as the container port, or managing storage for the
>> image and so on and so on)
>> In this context, you would run docker images, under the form of a docker
>> compose, on a single host.
>> * Wrap the docker-compose in the Swarm layer: here you would do the same
>> as above, but Juju would run at the host level of your Swarm cluster.
>> * Create a sp

Re: Docker compose in Juju

[Samuel Cozannet]

Hi Gayan,

Juju cannot operate with app containers as it operates with "os-based"
systems.
The difference is that Juju expects the node it spins to run a Juju agent.
In the case of a Docker container, which runs a single process, that would
mean the sole purpose of the container is running... Juju itself. So long
for your app.
So that means Juju cannot provision Docker containers "directly" (which is
what you seem to expect with the provider idea).

If you want to run Docker containers via Juju, you have the following
methods:
* Wrap the Docker image in the charm. Essentially, the charm would then be
a set of commands to run, scale, integrate... the container with the rest
of the world (for example, managing the fact that the host port is not
necessarily the same as the container port, or managing storage for the
image and so on and so on)
In this context, you would run docker images, under the form of a docker
compose, on a single host.
* Wrap the docker-compose in the Swarm layer: here you would do the same as
above, but Juju would run at the host level of your Swarm cluster.
* Create a specific charm to interact with the cluster API (k8s, Mesos,
Swarm): this charm would be your deployer. It could receive hooks from the
outside world, triggering deployment or other stuff. In this case, the
charm can be a subordinate (running on all cluster instances), which you
could potentially load balance, or a standalone service running on the
side.

Now, if you are looking into building a container based IaaS equivalent, I
suggest you look into LXD
<https://linuxcontainers.org/lxd/getting-started-cli/>, which would spin
containers that look and behave like VMs.
>From a Juju perspective, there is a LXD provider.
The diff with Docker is that LXD doesn't natively provide image based
deployment, but follows the Juju philosophy to always start with a blank
image.
So your charms will have to deploy the code in LXD from scratch.

Additional nodes: in the docs, pick the "/devel" branch, it's better if you
use Juju 2.x

I hope this helps.
++
Sam




--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Mon, Jun 6, 2016 at 2:36 PM, Gayan Gunarathne <csega...@gmail.com> wrote:

>
> Hi Charles/Samuel,
>
> Thanks for the information.
>
> What I am try to do is to run a docker compose to start docker containers
> in a single machine. Still I am not clear how we can use the docker with
> Juju.I am following this[1]
>
> Simple I need to use the Juju UI[2] and create my deployment there.Then
> can I do the deployment with the docker.How can I select  the docker as a
> IaaS. I couldn't find the docker configuration section in
> environment.yaml.I am not clear how can I provide the docker as a cloud
> provider.
>
> [1]https://jujucharms.com/docs/stable/getting-started
> [2]https://demo.jujucharms.com/
>
> Thanks,
> Gayan
>
> On Wed, Jun 1, 2016 at 3:08 AM, Samuel Cozannet <
> samuel.cozan...@canonical.com> wrote:
>
>> And finally to add to this, there is a community effort around Mesos and
>> DCOS to provide the same idea as the k8s stuff.
>>
>> I am also interested in your use case and feedback :)
>>
>> Best,
>> Sam
>> On May 31, 2016 7:17 PM, "Charles Butler" <charles.but...@canonical.com>
>> wrote:
>>
>>> Greetings Gayan!
>>>
>>> You most certainly can compose and use a full docker-provided stack, mix
>>> and match with charms - thats the power of wrapping your compose-based
>>> service with layer-docker.
>>>
>>> https://github.com/juju-solutions/layer-docker
>>>
>>> Which is a great starting place for charming up your dockerized app
>>> using charms.docker to lend a hand with the ops knowledge to get it running
>>> :)
>>>
>>> https://github.com/juju-solutions/charms.docker
>>> http://pythonhosted.org/charms.docker/modules.html
>>>
>>> There are several examples in the charm store from myself (lazypower)
>>> and Matt Bruzek (mbruzek)  that are docker based. I believe the most
>>> straight forward example that I can illustrate today is the swarm layer,
>>> which builds the swarm charm.
>>>
>>> https://github.com/juju-solutions/layer-swarm
>>> https://jujucharms.com/u/containers/swarm-core
>>>
>>> This is a multi-series charm that delivers swarm

Re: User Mgmt - 2.0 and Beyond

[Samuel Cozannet]

Good start. I would add

* external AAA connectivity (LDAP/AD) to find where the actual data is
stored.
* add user to units only (no access to Juju)
* share

++
Sam


--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Wed, Jun 1, 2016 at 10:58 AM, James Beedy <jamesbe...@gmail.com> wrote:

> I've "conjured-up" what I think would be a great enhancement to the
> current user mgmt capability in 2.0.
>
> As things stand, one can add a user to a model, and set a permission
> category of either read (read), or write (read/write). This functionality
> is awesome, and a huge step for juju (applause)!
>
> Admins of juju can now create, manage, and maintain the users and users
> access policy associated with a model (applause, again, seriously).
>
> As a logical next step, why don't we take the user all the way to the
> instance?
>
> What I'm thinking of is an '--os' flag that could be specified on user
> creation!
>
> This flag would signify that the user need be created on the instances in
> the current model. Ssh keys key(s) for a user could be added, and
> *associated*, and provisioned alongside the respective user, and user
> account on the machine.
>
> This functionality would give juju deployed infrastructure a huge edge in
> the ease of user management/maintainability for any organization, and
> massive bragging rights in enterprise land due to the increased PCI
> compliance revolving around finer granularity in user access accounts.
>
> I feel like the majority of the big pieces are already In place, the
> primary road blocks I foresee (probably a lot more):
> 1. User sensitive ssh-keys
> 2. Machine-level user provisioning template /UserManagerModel
> 3. Os-level user access/permission policy (what is generic/default yet
> tuned and hardened?)
>
>
> That about wraps it up, hopefully I got my point across to some degree.
>
> Thoughts?
>
>
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Subordinate charms

[Samuel Cozannet]

Hello,

What is then the recommendation for monitoring / logging / (...) agents
that run independently on the units and are loosely connected to the local
charms, but potentially still need to know about the apps running?

Wouldn´t colocation make them cumbersome? Still encoded as subordinate
charms they lack flexibility as they can´t change/be replaced.

For the Django example (and any framework sort of charms) I would have
thought that a layer is needed and the plugin is an option of the layer.

++
Sam





--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Tue, May 31, 2016 at 2:50 PM, Rick Harding <rick.hard...@canonical.com>
wrote:

> Yes, this is the way that subordinates work. They can often do very
> intrusive things. Imagine a django subordinate that installs and sets up a
> plugin into the django application it's deployed to. We don't currently
> support the removal of them. They kind of become one with the principle
> application.
>
> On Thu, May 26, 2016 at 5:41 PM André Moreira <andre@gmail.com> wrote:
>
>> Hi Tom,
>>
>> I found this:
>>
>> The current model of subordinates doesn't include support for removing
>> subordinate units from their principal service apart from removing the
>> principal service itself. This limitation stems from the current policy
>> around service shutdown and the invocation of stop hooks.
>> https://jujucharms.com/docs/1.25/authors-subordinate-services
>>
>> Is this policy still valid?
>>
>> André
>>
>> 2016-05-26 18:38 GMT-03:00 Tom Barber <t...@analytical-labs.com>:
>>
>>> Hi Andre
>>>
>>> Can you give us a clue what you are installing/uninstalling because I
>>> believe Bilal is correct, I've not see it wipe out the parent charms either.
>>>
>>> Tom
>>>
>>> --
>>>
>>> Director Meteorite.bi - Saiku Analytics Founder
>>> Tel: +44(0)5603641316
>>>
>>> (Thanks to the Saiku community we reached our Kickstart
>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>> goal, but you can always help by sponsoring the project
>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>
>>> On 26 May 2016 at 22:36, André Moreira <andre@gmail.com> wrote:
>>>
>>>> Using this, it also removes the unit of the charm it is subordinated to.
>>>>
>>>> 2016-05-26 18:34 GMT-03:00 Bilal Baqar <bba...@plumgrid.com>:
>>>>
>>>>> Try the normal charm remove command:
>>>>> *juju remove-service  *
>>>>>
>>>>> Regards
>>>>>
>>>>> 2016-05-27 2:24 GMT+05:00 André Moreira <andre@gmail.com>:
>>>>>
>>>>>> How can I remove a subordinate charm without removing the principal?
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Le doux charme de maint songe
>>>>>> Par leur bel art inventé
>>>>>> Sous les habits du mensonge
>>>>>> Nous offre la vérité.
>>>>>> -La Fontaine
>>>>>>
>>>>>> --
>>>>>> Juju mailing list
>>>>>> Juju@lists.ubuntu.com
>>>>>> Modify settings or unsubscribe at:
>>>>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Bilal Baqar
>>>>> MTS - PLUMgrid Inc.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Le doux charme de maint songe
>>>> Par leur bel art inventé
>>>> Sous les habits du mensonge
>>>> Nous offre la vérité.
>>>> -La Fontaine
>>>>
>>>> --
>>>> Juju mailing list
>>>> Juju@lists.ubuntu.com
>>>> Modify settings or unsubscribe at:
>>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>
>>>>
>>>
>>
>>
>> --
>>
>> Le doux charme de maint songe
>> Par leur bel art inventé
>> Sous les habits du mensonge
>> Nous offre la vérité.
>> -La Fontaine
>> --
>> Juju mailing list
>> Juju@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju
>>
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Docker compose in Juju

[Samuel Cozannet]

And finally to add to this, there is a community effort around Mesos and
DCOS to provide the same idea as the k8s stuff.

I am also interested in your use case and feedback :)

Best,
Sam
On May 31, 2016 7:17 PM, "Charles Butler" 
wrote:

> Greetings Gayan!
>
> You most certainly can compose and use a full docker-provided stack, mix
> and match with charms - thats the power of wrapping your compose-based
> service with layer-docker.
>
> https://github.com/juju-solutions/layer-docker
>
> Which is a great starting place for charming up your dockerized app using
> charms.docker to lend a hand with the ops knowledge to get it running :)
>
> https://github.com/juju-solutions/charms.docker
> http://pythonhosted.org/charms.docker/modules.html
>
> There are several examples in the charm store from myself (lazypower) and
> Matt Bruzek (mbruzek)  that are docker based. I believe the most straight
> forward example that I can illustrate today is the swarm layer, which
> builds the swarm charm.
>
> https://github.com/juju-solutions/layer-swarm
> https://jujucharms.com/u/containers/swarm-core
>
> This is a multi-series charm that delivers swarm via containers, backended
> by consul or etcd as the discovery mechanism. This has a good mix of
> relationships, base layers, top-layers (what you would be writing), and
> uses the docker-native tooling to bring everything up with some help from
> juju.
>
> There are even more complex examples, such as Kubernetes - all being
> brought up and controlled in a similar manner
>
> https://github.com/kubernetes/kubernetes/tree/master/cluster/juju
> https://jujucharms.com/u/containers/kubernetes-core
>
> If you need any help charming with Docker you can get in touch with myself
> on the mailing list here, or join us on irc in #juju on irc.freenode.net
>
> I'd love to hear any feedback/questions/comments about the developer
> tooling we have here, as its really ramped up our capacity to churn out
> high quality charms quickly that are docker based, and if there's any rough
> edges we can sand out for other developers would be great starting points.
>
> All the best,
>
> Charles
>
> On Tue, May 31, 2016 at 9:41 AM Gayan Gunarathne 
> wrote:
>
>> Hi Marco,
>>
>> Thanks for the details.
>>
>> Actually I just want to do something like this. Lets say I have tomcat
>> and mysql composite application. So I need to deploy these two application
>> in docker with depends on(I am really glad if I can use demo UI[1]). How
>> can I do that?
>>
>> Do you already have charms that run in docker with the same? Can you
>> point me to some sample?
>>
>> [1]https://demo.jujucharms.com/
>>
>> Thanks,
>> Gayan
>>
>> On Tue, May 31, 2016 at 6:52 PM, Marco Ceppi 
>> wrote:
>>
>>> Hi Gayan,
>>>
>>> I've added the general Juju list which covers more of these general
>>> topics.
>>>
>>> So, because of the nature of LXC machines and Docker style application
>>> containers it's hard to model that style application container in Juju in
>>> the same way LXC machines work. However, it's quite easy to wrap something
>>> like a Docker container, which works really well as a payload/software
>>> delivery tool, but then you can use Juju to wrap that immutable object and
>>> make it mutable inside of a Juju deployment.
>>>
>>> I know there are quite a few people on the juju mailing list doing this
>>> today, so I'll let them weigh in. In short, yes you can use Docker and
>>> docker style application containers with Juju, but not in the same direct
>>> way you would a LXC machine just because of the differences in function and
>>> form.
>>>
>>> Marco
>>>
>>> On Tue, May 31, 2016 at 7:09 AM Gayan Gunarathne 
>>> wrote:
>>>
 Hello,

 Can we run docker directly with Juju? I saw Juju is supporting the LXC
 containers. I need to know whether we can spawn docker containers as the
 same.

 If we support this can you point me to any document?

 Thanks,
 Gayan
 --
 Juju-dev mailing list
 juju-...@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

>>>
>>
>>
>> --
>> Best Regards,
>> Gayan
>> --
>> Juju-dev mailing list
>> juju-...@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
> --
> Juju Charmer
> Canonical Group Ltd.
> Ubuntu - Linux for human beings | www.ubuntu.com
> Juju - The fastest way to model your service | www.jujucharms.com
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Promulgated charms (production readiness)

[Samuel Cozannet]

Just bringing in a bit of work I've been doing with a few monitoring
(/logging) solutions such as Zabbix, Telegraf, Fluentd...

I have taken the opposite approach as what is mostly proposed here. I'm
from a more ops background, which means my devs usually had no clue
whatsoever of how I would manage their stuff. Also, even if they did most
of the job, I would probably need my own ops features.
So I had a bunch of questions for them, would deploy their stuff for them,
add the magic ingredient, and everyone would be happy.

My point is if I was a CTO somewhere using Juju right now, I wouldn't
expect my developers to actively write monitoring or logging entry points.
Also, I would expect an ops team to redo it anyway.
So I would rather create a "mynamespace-basic-layer", essentially extending
the base layer with the tooling I need (eventually even incorporating
Config management).
At this point I would expect my monitoring and logging **installed**

Then I use a "self assessment" listing the charms installed locally
function like :

function charm::lib::self_assessment() {
[ -z ${JUJU_CONTEXT_ID+x} ] && \
echo 0 || \
{
METADATA="$(find "${JUJU_CHARM_DIR}/../.." -name "metadata.yaml")"
for FILE in ${METADATA}
do
CHARM+=" $(cat "${FILE}" | grep 'name' | head -n1 | cut -f2 -d' ')"
done
}
echo "${CHARM}"
}

Essentially, I'm giving my supporting charms the ability to understand the
local environment at the unit level and adapt, even without an explicit
relation.

I store all monitoring/logging templates centrally which gives me the
ability to update them out of band, as you would for an antivirus DB. If I
was to store them in-charm, I'd need a charm upgrade to update them, which
can be cumbersome, especially if that is in the application charm.

In the end, I am building very intelligent supporting charms for monitoring
and logging, that understand the target logic based on their own knowledge
of the charm world, and adapt and evolve over time.
Anyone and everyone can then improve the templates, so it's even more
goodness even from people not using Juju.

My first experiment with Zabbix just used self assessment, added that to
the agent metadata, and the server would have a specific list of templates,
react to auto-discovery of the agents by looking up the metadata and
associate the proper templates automagically. It would also create groups
of machines, and eventually autoscaling rules.

Now I'm doing the same with fluentd (logging), storing the list of
templates in github and the agent downloads the templates it needs at
install time, and also for Telegraf (InfluxDB)

>From a user perspective, it's really "let's add monitoting", creating a
"juju add-relation all" feature like :

juju status --format json | jq '.services | keys[]' | tr -d '"' | xargs -I
'{}' juju add-relation ntp {}

and off you go...

Thoughts?

++
Sam














--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Mon, May 16, 2016 at 10:10 PM, Marco Ceppi <marco.ce...@canonical.com>
wrote:

> I think a layer:nrpe isn't the right choice, but a layer:monitoring might
> be. Esp with the layer/interface/subordinate model now, it seems that we
> could actualize an abstract means to declare what to monitor and have
> nrpe/zabbix-agent/promethous translate that.
>
> Marco
>
>
> On Mon, May 16, 2016 at 11:19 AM Cory Johns <cory.jo...@canonical.com>
> wrote:
>
>> I think this is a strong argument for creating an interface:nrpe layer to
>> make supporting this as easy as possible.  There was also discussion a long
>> time ago about creating a translation layer of sorts for supporting
>> multiple different monitoring solutions (like in the Puppet example).  I
>> think with layers and layer APIs that's now more possible than ever.
>>
>> Once we have a simplified interface, I do think that the review process
>> should strongly recommend that monitoring support be included, though I
>> don't think we'll be able to make it a hard requirement.
>>
>> On Mon, May 16, 2016 at 10:06 AM, Tom Barber <t...@analytical-labs.com>
>> wrote:
>>
>>> NRPE can be related as well, this is true. Maybe I misstated it a bit,
>>> I'll blame the jetlag ;)
>>>
>>> Put it this way, if a user is implementing a to-be promulgated charm, as
>>> a minimum (for those who expose such a thing) why not ensure the port i

Re: juju 1.x add machine command: how to specify flavors in an openstack environment

[Samuel Cozannet]

Hi Eddy,

Thanks for using Juju :)

You can do

juju deploy  --constraints "mem=8G cpu-cores=4 root-disk=32G"
then
juju set-constraints  "mem=8G..."

You can find the list of available constraints here :
https://jujucharms.com/docs/1.25/reference-constraints

(for Juju 1.X)

and there: https://jujucharms.com/docs/devel/reference-constraints
(for Juju 2.X)

Best,
Samuel


--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Wed, May 4, 2016 at 8:41 PM, Eddy Truyen <eddy.tru...@cs.kuleuven.be>
wrote:

> Hey,
>
> How can I  specify in juju1 in the `juju machine add` command the type of
> flavor (number of virtual cpus, GB of RAM).?
>
> I work in an openstack environment. As far as I can see I can only specify
> desired memory: juju machine add --constraints mem=8G
>
> but how do I specify the number of virtual cpus?
>
> Best
>
> Eddy
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: LXC/LXD profiles through Juju

[Samuel Cozannet]

Hi Tom,

I've been doing what James says recently, it's fully operational.

Alternatively, when you use lxd, the output of juju status will tell you
the names of the containers, which you can then map to "lxc list"
==> you can apply different profiles individually to different units, even
post deployment.
The drawback is that you may need to restart some services inside the
containers when you do so, so some logic must be applied in this context.

++
Sam


--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Sat, Apr 23, 2016 at 10:14 PM, Tom Barber <t...@analytical-labs.com>
wrote:

> Thanks chaps!
>
> I figured the worst comes to the worst I must be able to compile juju with
> the flag in there, but if there is a way like James suggests that means I
> don't have to, all the better :)
>
> --
>
> Director Meteorite.bi - Saiku Analytics Founder
> Tel: +44(0)5603641316
>
> (Thanks to the Saiku community we reached our Kickstart
> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
> goal, but you can always help by sponsoring the project
> <http://www.meteorite.bi/products/saiku/sponsorship>)
>
> On 23 April 2016 at 21:12, Marco Ceppi <marco.ce...@canonical.com> wrote:
>
>> While not a fix, you may be interested in this
>> https://bugs.launchpad.net/juju-core/+bug/1565872
>>
>> On Sat, Apr 23, 2016, 4:08 PM James Page <james.p...@ubuntu.com> wrote:
>>
>>> Hi Tom
>>>
>>> On Sat, 23 Apr 2016 at 15:38 Tom Barber <t...@analytical-labs.com> wrote:
>>>
>>>> Hello folks
>>>>
>>>> Working on my presentation for ApacheCon and trying to leverage some of
>>>> the great new stuff available.
>>>>
>>>> I've not tested it yet, so this is a slightly lazy question but as I'm
>>>> a way off of being at this stage I might as well ask in anticipation
>>>>
>>>> I want to run Docker inside LXD Local, which I can do manually by
>>>> passing the docker profile. Does/Can you do the same inside Juju?
>>>>
>>>
>>> I do something like that for some OpenStack on LXD stuff I've been
>>> working on:
>>>
>>>   https://github.com/openstack-charmers/openstack-on-lxd
>>>
>>> You can update the juju- profiles prior to  deploying any
>>> services to them which is a little coarse (model, not service level), but
>>> would let you use the docker profile for Juju managed LXD machines.
>>>
>>> Cheers
>>>
>>> James
>>> --
>>> Juju mailing list
>>> Juju@lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>
>>
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: DCOS on Juju

[Samuel Cozannet]

Hi Tom,

Adding also Artyom from DataArt as initial writers of the Mesos charms.
Artyom, it looks like we have a PoC for DC/OS charms. Let's coordinate also
with your team to see how we can get that further.

@Tom: No worries. Whenever you're ready send us links to the layers, and
we'll have a look.

So I have created this repo : https://github.com/SaMnCo/layers-dcos.git
so we can have all the code for all layers involved in one place. Empty for
now.

I've added a gitter here : https://gitter.im/SaMnCo/layers-dcos

Everyone interested please join this channel and let's move the
conversation there.

++
Sam






--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Mon, Apr 25, 2016 at 2:31 PM, Tom Barber <t...@analytical-labs.com> wrote:

> Hi smart people!
>
> I'll tidy up my initial DCOS dump and publish is over the next day or so.
> As a disclaimer, I'm no MESOS/DCOS expert so I'll defer to you guys on
> pretty much everything! ;)
>
> Tom
>
> --
>
> Director Meteorite.bi - Saiku Analytics Founder
> Tel: +44(0)5603641316
>
> (Thanks to the Saiku community we reached our Kickstart
> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
> goal, but you can always help by sponsoring the project
> <http://www.meteorite.bi/products/saiku/sponsorship>)
>
> On 25 April 2016 at 12:31, Andy Petrella <nooot...@data-fellas.guru>
> wrote:
>
>> awesome, my turn to add Rad, who had created a terraform dcos integration
>> to be open sourced soon.
>> in combination with François you'll have all the information necessary
>>
>> On Mon, 25 Apr 2016 13:20 Samuel Cozannet, <samuel.cozan...@canonical.com>
>> wrote:
>>
>>> Adding some smart people working with us on the Mesos stack.
>>>
>>> Andy, Adam, François, have a look at this DCOS charms.
>>> Tom, look at the project in my github (samnco/ods-dl)
>>>
>>> Let's open a gutter room later this week and see how we can collaborate
>>> on this
>>> On Apr 25, 2016 03:03, "Tom Barber" <t...@analytical-labs.com> wrote:
>>>
>>>> GPU Mesos on Power, that sounds cool!
>>>>
>>>> I did originally just stand up Mesos again, but DCOS is in the news so
>>>> I figured, why not.
>>>>
>>>> Okay sounds like a plan. Tomorrow is a bit rubbish, I'm around today
>>>> and Wednesday though, just ping me off list with a time.
>>>>
>>>> Tom
>>>>
>>>> --
>>>>
>>>> Director Meteorite.bi - Saiku Analytics Founder
>>>> Tel: +44(0)5603641316
>>>>
>>>> (Thanks to the Saiku community we reached our Kickstart
>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>>> goal, but you can always help by sponsoring the project
>>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>>
>>>> On 25 April 2016 at 01:06, Samuel Cozannet <
>>>> samuel.cozan...@canonical.com> wrote:
>>>>
>>>>> Tom,
>>>>>
>>>>> I've been working with a bunch of people on this as well for GPU
>>>>> enabled Mesos on Power. We're also doing more regarding DCOS Open Source.
>>>>>
>>>>> Let's connect and optimize who works on what. I am at OpenStack summit
>>>>> in Austin this week, but can do a call early in my mornings. Pick your 
>>>>> time
>>>>> and day.
>>>>>
>>>>> ++
>>>>> Sam
>>>>> On Apr 24, 2016 5:00 PM, "Tom Barber" <t...@analytical-labs.com> wrote:
>>>>>
>>>>>> Figured this was worth distributing to try and prevent duplicated
>>>>>> effort etc:
>>>>>>
>>>>>>
>>>>>> ​
>>>>>> Not ready for distribution yet, but I've stood up DCOS orchestration
>>>>>> as part of my ApacheCon demo (as its based on MESOS). Deploys master and
>>>>>> slave nodes to EC2 and works a treat(from a novice perspective).
>>>>>>
>>>>>> If anyone is interested in collaboration as my bandwidth is pretty
>>>>>> much maxed at the mo, let me know and we'll sort this stuff out.
>>>>>>
>>>>>> Tom
>>>>>> --
>>>>>>
>>>>>> Director Meteorite.bi - Saiku Analytics Founder
>>>>>> Tel: +44(0)5603641316
>>>>>>
>>>>>> (Thanks to the Saiku community we reached our Kickstart
>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>>>>> goal, but you can always help by sponsoring the project
>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>>>>
>>>>>> --
>>>>>> Juju mailing list
>>>>>> Juju@lists.ubuntu.com
>>>>>> Modify settings or unsubscribe at:
>>>>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>>>
>>>>>>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: DCOS on Juju

[Samuel Cozannet]

Adding some smart people working with us on the Mesos stack.

Andy, Adam, François, have a look at this DCOS charms.
Tom, look at the project in my github (samnco/ods-dl)

Let's open a gutter room later this week and see how we can collaborate on
this
On Apr 25, 2016 03:03, "Tom Barber" <t...@analytical-labs.com> wrote:

> GPU Mesos on Power, that sounds cool!
>
> I did originally just stand up Mesos again, but DCOS is in the news so I
> figured, why not.
>
> Okay sounds like a plan. Tomorrow is a bit rubbish, I'm around today and
> Wednesday though, just ping me off list with a time.
>
> Tom
>
> --
>
> Director Meteorite.bi - Saiku Analytics Founder
> Tel: +44(0)5603641316
>
> (Thanks to the Saiku community we reached our Kickstart
> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
> goal, but you can always help by sponsoring the project
> <http://www.meteorite.bi/products/saiku/sponsorship>)
>
> On 25 April 2016 at 01:06, Samuel Cozannet <samuel.cozan...@canonical.com>
> wrote:
>
>> Tom,
>>
>> I've been working with a bunch of people on this as well for GPU enabled
>> Mesos on Power. We're also doing more regarding DCOS Open Source.
>>
>> Let's connect and optimize who works on what. I am at OpenStack summit in
>> Austin this week, but can do a call early in my mornings. Pick your time
>> and day.
>>
>> ++
>> Sam
>> On Apr 24, 2016 5:00 PM, "Tom Barber" <t...@analytical-labs.com> wrote:
>>
>>> Figured this was worth distributing to try and prevent duplicated effort
>>> etc:
>>>
>>>
>>> ​
>>> Not ready for distribution yet, but I've stood up DCOS orchestration as
>>> part of my ApacheCon demo (as its based on MESOS). Deploys master and slave
>>> nodes to EC2 and works a treat(from a novice perspective).
>>>
>>> If anyone is interested in collaboration as my bandwidth is pretty much
>>> maxed at the mo, let me know and we'll sort this stuff out.
>>>
>>> Tom
>>> --
>>>
>>> Director Meteorite.bi - Saiku Analytics Founder
>>> Tel: +44(0)5603641316
>>>
>>> (Thanks to the Saiku community we reached our Kickstart
>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>> goal, but you can always help by sponsoring the project
>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>
>>> --
>>> Juju mailing list
>>> Juju@lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>
>>>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: DCOS on Juju

[Samuel Cozannet]

Tom,

I've been working with a bunch of people on this as well for GPU enabled
Mesos on Power. We're also doing more regarding DCOS Open Source.

Let's connect and optimize who works on what. I am at OpenStack summit in
Austin this week, but can do a call early in my mornings. Pick your time
and day.

++
Sam
On Apr 24, 2016 5:00 PM, "Tom Barber"  wrote:

> Figured this was worth distributing to try and prevent duplicated effort
> etc:
>
>
> ​
> Not ready for distribution yet, but I've stood up DCOS orchestration as
> part of my ApacheCon demo (as its based on MESOS). Deploys master and slave
> nodes to EC2 and works a treat(from a novice perspective).
>
> If anyone is interested in collaboration as my bandwidth is pretty much
> maxed at the mo, let me know and we'll sort this stuff out.
>
> Tom
> --
>
> Director Meteorite.bi - Saiku Analytics Founder
> Tel: +44(0)5603641316
>
> (Thanks to the Saiku community we reached our Kickstart
> 
> goal, but you can always help by sponsoring the project
> )
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: LXD v2.0.0-rc8 does not work with Juju v2.0-beta3

[Samuel Cozannet]

FYI, had the same issue on IBM hardware, and Antonio helped me get through
it:

* dpkg-reconfigure lxd was necessary to rename the bridge
* Then bootstrap still failed with
2016-04-07 16:35:17 ERROR cmd supercommand.go:448 no registered provider
for "lxd"
ERROR failed to bootstrap model: subprocess encountered error code 1

The fix for that was to add the --upload-tools tag in bootstrap.

So "juju bootstrap deeplearning lxd --upload-tools got me going.

Thanks!
Sam
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: [Juju] Minimum policies for Juju to work on public clouds

[Samuel Cozannet]

Yeah, I tried to add the VPC as well, but didn't work either. There is
something about the "bucket" created at the beginning, I thought S3 perms
would do, but no luck.




--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Sun, Mar 6, 2016 at 2:41 PM, Tom Barber <t...@analytical-labs.com> wrote:

> Do you need to offer up some VPC permissions as well on VPC default EC2
> accounts?
> On 6 Mar 2016 13:24, "Samuel Cozannet" <samuel.cozan...@canonical.com>
> wrote:
>
>> Hi All,
>>
>> I have been setting up many different environments on AWS, GCE, Azure
>> (...), but my most used cloud by far until now has been AWS.
>>
>> The way I have operated until now is to create an admin group in IAM,
>> then adding users in it for my demos, and use their credentials in the
>> environment file.
>> This means Juju has "full power" on my AWS environment, to the extend it
>> could create additional users. Furthermore, if I share my environment with
>> someone, I am "giving" my AWS account away essentially. Not cool.
>> Hence I tried to find the minimum policy (or group of policies) I should
>> apply to make it work without giving away too much power.
>>
>> Juju seems to work fine with PowerUser perms, which is everything minus
>> user management. A good start, but still too much for me.
>>
>> Then when I tried to restrict further,
>> * FullEC2Access: not sufficient, fails to bootstrap
>> * FullEC2 + FullS3: not sufficient, fails to bootstrap
>> The error I get is :
>> ERROR failed to bootstrap environment: cannot start bootstrap instance:
>> recording instance in provider-state: cannot write file "provider-state" to
>> control bucket: The specified bucket does not exist
>>
>> ==> Is there a recommended set of policies somewhere? I'd love to see
>> that in the docs as well, with advice for each cloud.
>>
>> Thanks,
>> Sam
>>
>>
>> --
>> Juju mailing list
>> Juju@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju
>>
>>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

[Juju] Minimum policies for Juju to work on public clouds

[Samuel Cozannet]

Hi All,

I have been setting up many different environments on AWS, GCE, Azure
(...), but my most used cloud by far until now has been AWS.

The way I have operated until now is to create an admin group in IAM, then
adding users in it for my demos, and use their credentials in the
environment file.
This means Juju has "full power" on my AWS environment, to the extend it
could create additional users. Furthermore, if I share my environment with
someone, I am "giving" my AWS account away essentially. Not cool.
Hence I tried to find the minimum policy (or group of policies) I should
apply to make it work without giving away too much power.

Juju seems to work fine with PowerUser perms, which is everything minus
user management. A good start, but still too much for me.

Then when I tried to restrict further,
* FullEC2Access: not sufficient, fails to bootstrap
* FullEC2 + FullS3: not sufficient, fails to bootstrap
The error I get is :
ERROR failed to bootstrap environment: cannot start bootstrap instance:
recording instance in provider-state: cannot write file "provider-state" to
control bucket: The specified bucket does not exist

==> Is there a recommended set of policies somewhere? I'd love to see that
in the docs as well, with advice for each cloud.

Thanks,
Sam
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Bitnami wordpress charm development

[Samuel Cozannet]

I think what you'd actually want to do is a layer for that. If you create
configuration in the charm, then you'd have a generic charm, and not expose
what is actually into it.
As Juju wants to model, what actually runs must be explicitly described.

With a layer, you could enable people to use a remote script at install,
while creating one charm per app essentially.

That would then give you the ability to extend the bitnami behavior, which
focuses on deploying 1 unit of everything. For example, in your wordpress
deployment, you could essentially add a mysql relationship that would
transfer the local DB to a remote SQL DB, which would then enable scaling
out the front end.

Interested in following up on this as well, please keep me in the loop.
++
Samuel


--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Tue, Mar 1, 2016 at 6:23 PM, Ney Moura <neymou...@gmail.com> wrote:

> Hi Samuel!
>
> Well! It's a good idea! But since I'm to new in shell scripting I don't
> think I could handle such thing. On the other hand I think I could create a
> template charm that, with minimal effort, the community would be able to
> use it on another bitnami stacks. Since stack commands should be the same,
> I think I'ts possible =)
>
> The only thing the developer has to do is to change the url witch the run
> script is downloaded and specify installation parameters, both in
> installation hook.
>
>
> 2016-03-01 4:47 GMT-03:00 Samuel Cozannet <samuel.cozan...@canonical.com>:
>
>> Hi Ney,
>>
>> I've got myself in the same issue in the past, and found this pretty
>> useful: dos2unix. You can apt-get install it, then you get a dos2unix and a
>> unix2dos commands.
>>
>> If you succeed with the bitnami -> charm, do you think you could create a
>> generic bitnami converter? That would be interesting.
>>
>> Good charming!
>> Best,
>> Sam
>>
>>
>> --
>> Samuel Cozannet
>> Cloud, Big Data and IoT Strategy Team
>> Business Development - Cloud and ISV Ecosystem
>> Changing the Future of Cloud
>> Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> /
>> Juju <https://jujucharms.com>
>> samuel.cozan...@canonical.com
>> mob: +33 616 702 389
>> skype: samnco
>> Twitter: @SaMnCo_23
>> [image: View Samuel Cozannet's profile on LinkedIn]
>> <https://es.linkedin.com/in/scozannet>
>>
>> On Tue, Mar 1, 2016 at 1:46 AM, Ney Moura <neymou...@gmail.com> wrote:
>>
>>> I was thinking that i missed some configurations to ran those commands
>>> properly only.
>>>
>>> Since I use nano, I wasn't able to see those.
>>>
>>> Thanks for the tip about hook environment and the command!
>>>
>>> Mainly for the quick response =D
>>>
>>> 2016-02-29 21:31 GMT-03:00 José Antonio Rey <j...@ubuntu.com>:
>>>
>>>> Glad that was it! When I opened it on vim, it said [dos] at the bottom,
>>>> and after googling the error I saw what it was.
>>>>
>>>> About the command, what happens is right. You should not be able to
>>>> execute any of those commands. The commands are only executable in a 'hook
>>>> environment', meaning only hooks will be able to run them. If you really
>>>> want to execute one of them, you can use 'juju run --service
>>>> wordpress-single "command goes here"'.
>>>>
>>>> Why do you want to run that manually? Anything missing?
>>>>
>>>> --
>>>> José Antonio Rey
>>>>
>>>> On Mon, Feb 29, 2016, 19:22 Ney Moura <neymou...@gmail.com> wrote:
>>>>
>>>>> Thanks everyone!
>>>>>
>>>>> José Antonio, you were right!
>>>>>
>>>>> How did you discover that?
>>>>>
>>>>>
>>>>> But let me challenge you guys even more!
>>>>>
>>>>> I need to install something in the VM before using the juju commands,
>>>>> like status-set? I'm asking this because I wasn't able to run these
>>>>> commands in there.
>>>>>
>>>>>
>>>>> Thanks again!
>>>>>
>>>

Re: Bitnami wordpress charm development

[Samuel Cozannet]

Hi Ney,

I've got myself in the same issue in the past, and found this pretty
useful: dos2unix. You can apt-get install it, then you get a dos2unix and a
unix2dos commands.

If you succeed with the bitnami -> charm, do you think you could create a
generic bitnami converter? That would be interesting.

Good charming!
Best,
Sam


--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Tue, Mar 1, 2016 at 1:46 AM, Ney Moura <neymou...@gmail.com> wrote:

> I was thinking that i missed some configurations to ran those commands
> properly only.
>
> Since I use nano, I wasn't able to see those.
>
> Thanks for the tip about hook environment and the command!
>
> Mainly for the quick response =D
>
> 2016-02-29 21:31 GMT-03:00 José Antonio Rey <j...@ubuntu.com>:
>
>> Glad that was it! When I opened it on vim, it said [dos] at the bottom,
>> and after googling the error I saw what it was.
>>
>> About the command, what happens is right. You should not be able to
>> execute any of those commands. The commands are only executable in a 'hook
>> environment', meaning only hooks will be able to run them. If you really
>> want to execute one of them, you can use 'juju run --service
>> wordpress-single "command goes here"'.
>>
>> Why do you want to run that manually? Anything missing?
>>
>> --
>> José Antonio Rey
>>
>> On Mon, Feb 29, 2016, 19:22 Ney Moura <neymou...@gmail.com> wrote:
>>
>>> Thanks everyone!
>>>
>>> José Antonio, you were right!
>>>
>>> How did you discover that?
>>>
>>>
>>> But let me challenge you guys even more!
>>>
>>> I need to install something in the VM before using the juju commands,
>>> like status-set? I'm asking this because I wasn't able to run these
>>> commands in there.
>>>
>>>
>>> Thanks again!
>>>
>>> 2016-02-29 19:15 GMT-03:00 José Antonio Rey <j...@ubuntu.com>:
>>>
>>>> This is because you are using Windows line breaks instead of Linux/Unix
>>>> line breaks. Try saving your file with Linux/Unix line breaks and give it a
>>>> go.
>>>>
>>>>
>>>> On 02/29/2016 05:12 PM, David Britton wrote:
>>>>
>>>>> On Mon, Feb 29, 2016 at 07:02:08PM -0300, Ney Moura wrote:
>>>>>
>>>>>>
>>>>>> But I keep having erros with the install hook. It says file not found.
>>>>>>
>>>>>>
>>>>> Maybe I missed it in the tarball -- could you attach a juju debug-log
>>>>> capture while you deploy it?
>>>>>
>>>>> Or, /var/log/all-machines.log from the bootstrap node is the same
>>>>> thing.
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>> --
>>>> José Antonio Rey
>>>>
>>>
>>>
>>>
>>> --
>>> *Ney Moura Conceição*
>>>
>>
>
>
> --
> *Ney Moura Conceição*
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Mesos / Marathon with Juju

[Samuel Cozannet]

Hi John,

We love your use case already. The current status is the following:

* We have a legacy charm for Mesos, that scales out with slaves, and that
runs Marathon. You can see it here:
https://jujucharms.com/u/jonnybgod/mesos-master/trusty/5
* This charm has not yet been ported to the new charming methodology
(layers)

As explained in the documentation, the charm can also optionally install
Docker and other mesos services. You should therefore be able to run
Cassandra as a Mesos framework, and other micro services as Docker
containers into it.

There is also a partner effort to standardize a Mesos based stack. You can
see it demoed in this video <https://www.youtube.com/watch?v=Bxit-a3_yL4>.
They are looking for help to charm up this new stack. If you are interested
in helping out, send me an off list email and I'll introduce you to them.

Let me know how your tests go and if you need further details,
All the best,
Samuel


On Wed, Jan 27, 2016 at 11:15 PM, John Weldon <johnweld...@gmail.com> wrote:

> Can anyone provide me information if that's feasible today, and what I'd
> need to know to get started





--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Juju, LXD, Snappy, Raspberry pi

[Samuel Cozannet]

Hi Matthew,

Great stuff, thanks for sharing :)

Quick question, if you set the LXD provider to work remotely, do you really
need to compile Juju to run on the rpi? Can't it run from the laptop, with
a rpi2 being just the target?

Thx,
Sam


--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Thu, Dec 3, 2015 at 2:19 PM, Matthew Williams <
matthew.willi...@canonical.com> wrote:

> Me again folks,
>
> Just for fun:
>
> Now that lxd provider support is in juju, and there is a snap for lxd. I
> wondered if it would be possible to use juju to deploy lxd units on a
> raspberry pi 2.
>
> Here's the result:
> http://blog.mattyw.net/blog/2015/12/02/juju-lxd-snappy-pi/
>
> Matty
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Juju and cluster managers (like mesos)

[Samuel Cozannet]

Adding Artyom from DataArt who has a lot of very clever ideas about this
and recently created a cool autoscaling demo.

I'm interested in any follow up given to your work. I share your
frustration at containers systems using a so called "orchestration" when
the orchestration is really some basic hooks.
The consequence of these systems is a total absence of portability between
techs (moving from k8s to Swarm or worse Mesos requires a lot of rewriting
the core and even sometimes rebuild some of the containers to adapt to the
service discovery APIs). Something that Juju wants to address really well.

My path so far is to create specific injection charms for k8s and others
(Swarm so far). By talking only to the current leader, you kind of create
this abstraction you are talking about.
That means you can then expose configuration to scale out & in the service
by calling the Juju API to reconfigure the service itself.
Not a complete solution, but a starting point. The issue with it is that to
comply with Juju models, I have to create an injection charm per app, which
is additional work on top of containerizing for example.
The LXD provider will certainly help in that space, even more when/if LXD
become first class citizen in "Container Orchestration Tools".

As for monitoring, some charms expose monitoring hooks that can be consumed
by other specialized services. As a consequence you can easily integrate
not only with service spawned by Juju, but also external systems. However,
monitoring is an expression of the operation and not the model, therefore
"can not" be operated by Juju.

Scaling should not depend on Juju either in our current vision. It's not an
expression of a model, but rather of how to operate the model. Therefore,
this task should stay outside of Juju, even if it can be operated via
Juju's APIs (scale out / in, potentially rolling upgrades in the future).

++
Sam




--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Mon, Nov 30, 2015 at 2:47 PM, Merlijn Sebrechts <
merlijn.sebrec...@gmail.com> wrote:

> Hi all
>
>
> I'd like to start a discussion about how Juju can fit into cluster
> managers like Apache Mesos and Kubernetes.
>
> Currently, Juju fits nicely into this story as a way to setup these
> cluster managers. Payloads continue on that idea with Juju as a manager of
> a cluster manager. However, I'm a lot more interested in Juju on top of a
> cluster manager, where the cluster manager would be a provider Juju uses to
> deploy services on.
>
> Juju provides an awesome way to model complex services in a modular and
> re-usable way. The relationships allow for much more complex interactions
> between services than what the "service discovery" in Kubernetes and Mesos
> allows. Service discovery allows for a service to say "I need the IP's of
> these services" but that's pretty much it. No flexible adaptable
> infrastructure where services change their behavior depending on what they
> are connected to. It basically stems from the same mindset that brought us
> tools like Chef and Puppet: One company with a big dev team that creates
> services for internal use only.
>
>
>- Cluster managers are very good at scheduling "dumb" workloads.
>They're a datacenter kernel, they don't care what runs in the container. At
>best, they provide a way for two containers to communicate (service
>discovery).
>
>
>- Juju is very good at configuring applications. It changes services
>depending on how they are connected. Juju for the most part doesn't care
>where services run, only how they are connected.
>
>
> Combine these two and you could get an awesome PaaS that can run a lot
> more than "dumb" 12-factor apps. It bothers me to see frameworks like
> Kubernetes use the terminology "service orchestration" when all they do is
> connect static services. Juju is on to something with its dynamic
> relations, but it seems not many people have caught on...
>
> I see two possible paths to integrate Service Orchestration with Cluster
> management (scheduling):
>
>
>- *Support cluster managers in Juju as providers.* This might be hard
>to do since Juju's units require an OS-level container, not a process
>container. Step 1 would be LXD support in Mesos/Kubernetes?
>
>
>- *Write cluster management extensions on top of Juju.* Basically
>recreate the scheduli

Re: Juju service commands or communitation protocol

[Samuel Cozannet]

Hi André,

Not sure about your use case, but we've used this integration (python
native I am afraid) to do some Juju autoscaling with a metrology tool
(Zabbix). Have a look if you're doing something similar :)

Repo (a bit outdated now): https://github.com/SaMnCo/ob-zabbix/
video: https://www.youtube.com/watch?v=IF7bUgDCYMM

++
Sam




--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Thu, Nov 5, 2015 at 12:19 AM, André Moreira <andre@gmail.com> wrote:

> Thank you, Adam. This is exactly what I was looking for.
> André
>
> 2015-11-04 14:21 GMT-02:00 Adam Stokes <adam.sto...@canonical.com>:
>
>> You'll have to check out some of the bindings code:
>>
>>
>> http://bazaar.launchpad.net/~juju-deployers/python-jujuclient/trunk/view/head:/jujuclient.py
>>
>> That should give you a good indication of what the juju api server
>> requires. For example the credentials need to be in a form of
>>
>>
>> {'Type': 'Admin',
>>   'Request': 'Login',
>>   'RequestId': 1,
>>   'Params': {'AuthTag': 'user-admin',
>>  'Password': None}}
>>
>> The above library is a single file which is pretty easy to grok and has
>> good documentation on what the functions do.
>>
>> On Wed, Nov 4, 2015 at 11:04 AM, André Moreira <andre@gmail.com>
>> wrote:
>>
>>> Hi Adam,
>>> What I mean is how do I construct the json requests and how should I
>>> expect the json response to be provided. Do you know where do I find this
>>> kind of documentation? I mean, is there a doc for it without needing to
>>> read the source code?
>>> André
>>>
>>> 2015-11-04 11:34 GMT-02:00 Adam Stokes <adam.sto...@canonical.com>:
>>>
>>>> If you're going to use Go or creating bindings of your own the api
>>>> documentation is https://godoc.org/github.com/juju/juju/api.
>>>>
>>>> On Wed, Nov 4, 2015 at 8:30 AM, André Moreira <andre@gmail.com>
>>>> wrote:
>>>>
>>>>> Thank you, Mark, for the reply.
>>>>> Nice to know there is this library and there will be an official one!
>>>>> But, if I want to use the REST api without the python library, where
>>>>> can I find documentation for it?
>>>>> Thanks in advance,
>>>>> André
>>>>>
>>>>> 2015-11-04 6:32 GMT-02:00 Mark Shuttleworth <m...@ubuntu.com>:
>>>>>
>>>>>> On 04/11/15 09:02, Merlijn Sebrechts wrote:
>>>>>> > I'm very interested to hear more about that python library! Is this
>>>>>> > something that will be discussed at the summit?
>>>>>>
>>>>>> It could be. There have been a couple of informal variations on this
>>>>>> theme from various sources, we recently decided it was worth making an
>>>>>> official one. Someone closer to that work would be able to provide
>>>>>> more
>>>>>> insight, including whether it will be based on one like
>>>>>> http://python-jujuclient.readthedocs.org/ or whether it's a fresh
>>>>>> start.
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Le doux charme de maint songe
>>>>> Par leur bel art inventé
>>>>> Sous les habits du mensonge
>>>>> Nous offre la vérité.
>>>>> -La Fontaine
>>>>>
>>>>> --
>>>>> Juju mailing list
>>>>> Juju@lists.ubuntu.com
>>>>> Modify settings or unsubscribe at:
>>>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Le doux charme de maint songe
>>> Par leur bel art inventé
>>> Sous les habits du mensonge
>>> Nous offre la vérité.
>>> -La Fontaine
>>>
>>
>>
>
>
> --
>
> Le doux charme de maint songe
> Par leur bel art inventé
> Sous les habits du mensonge
> Nous offre la vérité.
> -La Fontaine
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: UOS Sessions, anyone?

[Samuel Cozannet]

I have a few meaningful demos for big data with datasets ready and
everything scripted on my github (
HTTPS://github.com/SaMnCo/juju-strata-demos, pick saiku or SpagoBI branch)

But I will be off during the sessions so someone else would have to run
them.

I can and would be happy to train or help whoever volunteers prior to the
30th of October.

Best,
Sam
On Oct 23, 2015 3:35 PM, "Rick Harding"  wrote:

> Definitely think it'd be great to get some sessions going. Some ideas:
>
> reactive framework from ben/cory and maybe some talk through how to join
> into the community/guide folks to submitting new layers/stubs
>
> using the big-data solutions to do something interesting, work on the
> reuse of existing work story there
>
> the UI Eng folks could walk through the plans for the new charm upload
> process and possibly recruit some beta users as that moves forward
>
> might also get them to show off some of the GUI 2.0 goodness coming and
> discuss the roadmap there a bit
>
> Alexis, do you think someone could pull together a core demo of new stuff
> from the lightning talks we had recently? Storage, networking, lxd provider
> demos? It'd be great to double dip and use the recorded sessions as
> potential anchors to blog/announcements about new features in 1.25 perhaps?
>
> Jose, we'll chat with folks and see what we can get pulled together.
> Thanks for reaching out!
>
> Rick
>
>
> On Fri, Oct 23, 2015 at 3:13 AM José Antonio Rey  wrote:
>
>> Hey guys,
>>
>> UOS is just around the corner. Do you have anything to tell people
>> about, anything you'd like to discuss about Juju or the Cloud and
>> Ubuntu? Let me know, and let's get your session on UOS!
>>
>> Yes, we accept cool ideas. If you're working and want to show it to the
>> world - it's the perfect time.
>>
>> If you have any other questions about UOS or sessions/the schedule,
>> don't hesitate to send me an email. I'll be more than glad to help you
>> out!
>>
>> --
>> José Antonio Rey
>>
>> --
>> Juju mailing list
>> Juju@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju
>>
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Designing for Success: Juju and Charm architecture overview

[Samuel Cozannet]

+1, great presentation. We should publish this as a post with some
wordsmithing + marketing (images).

++
Sam


--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu <http://ubuntu.com>  / Canonical UK LTD <http://canonical.com> / Juju
<https://jujucharms.com>
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
<https://es.linkedin.com/in/scozannet>

On Sun, Sep 20, 2015 at 5:46 PM, James Beedy <jamesbe...@gmail.com> wrote:

> Marco -
>
> Your slides and presentation couldn't be more clear and comprehensive! Thanks
> a ton (to you and everyone else) for putting the time and effort in to this
> event to make it the huge success that it was! Knowing the "hows" and
> "whys" is what really matters and what really counts in the end. Having a
> clear understanding of the environment architecture is what will really
> help aspiring juju developers (like myself) to write and develop better
> charms! Thanks again!
>
> James
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: [Juju] Multiple sub service with a relation

[Samuel Cozannet]

Thanks John!

Allow me to rephrase as I am not sure I am following. Are saying we could
have a declarative format in the configuration of the charm that would
apply to relations created like:

datasources : [
  { name: mysql,
datasets: [ ds1, ds2, ... ],
  },
  { name: mongodb,
datasets: [ ds3, ds4, ... ],
  },
  ...
  ]

and then config-changed hook would replay the creation of relations for any
new datasets?

@Alberto, Ana: if my understanding is correct, this would be a way to
expose the configuration of the datasets / schemas from the charm config.
Some code to generate to manage errors (absence of dataset, absence of
relation to datasource, wrong datasource definition...) but it seems closer
to Juju's philosophy than using actions.

++
Sam

On Thu, Aug 20, 2015 at 1:30 AM, John Meinel j...@arbash-meinel.com wrote:

 Would this just be config on the charm that would eventually become config
 on the relation?




Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
https://es.linkedin.com/in/scozannet
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: [Juju] Multiple sub service with a relation

[Samuel Cozannet]

Hi Ana,

If you have multiple servers, you'll have multiple relations.

The administrator running the environment will have a knowledge of all the
relations, thus of the servers and names.

Your charm shall be able to match relation counterparts and config options
and log the potential errors without exiting, so that in case of mistake
the admin can observe what's wrong and fix it.

Now I guess what John was saying would be that the settings in the config
would be a json string such as
juju set charmname datasources=[
  { name: mysql,
server: server.domain,
datasets: [ ds1, ds2, ... ],
  },
  { name: mongodb,
server: mongo.domain,
datasets: [ ds3, ds4, ... ],
  },
  ...
  ]

Or another format... But basically putting the unknown in something that
you manipulate as an object in your code.

Hope it's clearer.
++
Samuel

Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
https://es.linkedin.com/in/scozannet

On Thu, Aug 20, 2015 at 1:54 PM, Ana Tomic ana.to...@mht.net wrote:

 Hi everyone,



 Thank you for your suggestion.
 If I have understood well, if we use config options the user should be
 able to use something like:

 juju set charmname database-server=server_name

 We want to connect to various numbers of servers at the same time, so we
 need an option for every server.

 Like database_server1, database_server2... but the problem is that we can
 not specify the exact number of servers.
 Is there a way to have variate number of options?



 Thanks in advance,

 Ana



 *From:* Samuel Cozannet [mailto:samuel.cozan...@canonical.com]
 *Sent:* Thursday, August 20, 2015 11:06 AM
 *To:* John Meinel
 *Cc:* Amir Sanjar; Ana Tomic; Juju email list; Alberto Ghedin
 *Subject:* Re: [Juju] Multiple sub service with a relation



 Thanks John!

 Allow me to rephrase as I am not sure I am following. Are saying we could
 have a declarative format in the configuration of the charm that would
 apply to relations created like:

 datasources : [

   { name: mysql,

 datasets: [ ds1, ds2, ... ],
   },

   { name: mongodb,

 datasets: [ ds3, ds4, ... ],
   },

   ...
   ]



 and then config-changed hook would replay the creation of relations for
 any new datasets?

 @Alberto, Ana: if my understanding is correct, this would be a way to
 expose the configuration of the datasets / schemas from the charm config.
 Some code to generate to manage errors (absence of dataset, absence of
 relation to datasource, wrong datasource definition...) but it seems closer
 to Juju's philosophy than using actions.

 ++

 Sam



 On Thu, Aug 20, 2015 at 1:30 AM, John Meinel j...@arbash-meinel.com
 wrote:

 Would this just be config on the charm that would eventually become config
 on the relation?




 Best,

 Samuel



 --

 Samuel Cozannet

 Cloud, Big Data and IoT Strategy Team

 Business Development - Cloud and ISV Ecosystem

 Changing the Future of Cloud
 Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com /
 Juju https://jujucharms.com

 samuel.cozan...@canonical.com

 mob: +33 616 702 389

 skype: samnco

 Twitter: @SaMnCo_23
 [image: Image removed by sender. View Samuel Cozannet's profile on
 LinkedIn] https://es.linkedin.com/in/scozannet

-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

[Juju] Proxy Charm examples

[Samuel Cozannet]

Hi!

Still working with SpagoBI, Alberto and Ana are looking for examples of
proxy charms, or best practices to expose external services.

I think vzzualmobileapi is one of these, but are there documented best
practices about them somewhere and/or example? I couldn't help on that one.

Many thanks!

Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
https://es.linkedin.com/in/scozannet
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

[Juju] Multiple sub service with a relation

[Samuel Cozannet]

Hi All!

Our friends at Engineering are working on the SpagoBI charm set and it's
going very well.

They are facing an issue that I could not answer, and I guess some of you
will have ideas about it.

SpagoBI is (as the name suggests) a BI suite. It connects to many data
sources, and provides tools to analyse them, aggregate results and display
them nicely.
That means that when SpagoBI relates to a data source such as MongoDB,
there is a sub relation to the actual dataset within the data source. For
example, it would be mongodb-user-document-set and
mongodb-actions-document-set.

In the current design of relations, this doesn't seem to fit.

My answer has been to propose to work with actions. If SpagoBI connects to
MongoDB globally, then something like

juju action do spagobi/0 connect-dataset datasource=mongodb dataset=user

would be parsable, testable, and would work. It has the drawback of not
being visible from a Juju perspective as the actions will show up in the
logs, but not in the GUI or through the API.

Are there other options? What would be the best practice in this case?

Many thanks in advance!
Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
https://es.linkedin.com/in/scozannet
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Juju Eco Dashboard

[Samuel Cozannet]

Good one, thanks!

Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
https://es.linkedin.com/in/scozannet

On Mon, Jul 20, 2015 at 12:42 PM, Charles Butler 
charles.but...@canonical.com wrote:

 Greetings,

 I whipped up a quick dashboard last year to help keep track of the Eco
 battlefield view, that quickly grew long in the tooth. I've spent a
 couple hours in the code this morning to update it and make some meaningful
 widgets.

 I need to invest a bit more time translating the Launchpad API into a
 RubyGem https://github.com/chuckbutler/rlaunchpadlib to get the values
 to build meaningful widgets for the LP based issue trackers, charms, and
 contributors. But you can track the progress here:
 http://dash.juju.solutions/wide

 The old narrow dash has been refactored for 1080p optimization.

 If you have suggestions, bugs, commentary - You can find the source over
 on the GitHub repository here:
 https://github.com/juju-solutions/juju-is-dashing

 A readme is forthcoming, because... #lazy

 Charles Butler charles.but...@canonical.com - Juju Charmer
 Come see the future of datacenter orchestration: http://jujucharms.com

 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju


-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: [Review Queue] gluster, docker, and new big data (spark/zeppelin/pig/hive)

[Samuel Cozannet]

Congrats guys, great work! Happy to see Spark there!
On Jul 3, 2015 12:16 AM, Kevin Monroe kevin.mon...@canonical.com wrote:

 Hi folks!  Cory and I hit a couple review queue items today:

 + gluster
 https://bugs.launchpad.net/charms/+bug/1469213

 This was a real treat.  Not only have I been looking for a reason to try
 out this fancy network filesystem, but I've also been wondering what's so
 great about rust.  This charm helped me on both fronts (yup, it really has
 a rust component [1]).  I'm happy to say we didn't hit any segfaults, but
 unfortunately can't +1 it at this time.

 Though the deployment was successful, we had reservations on a missing
 config-changed hook and would like to see the readme and tests polished a
 bit more prior to hitting the store.  Overall, this is a great submission
 and I look forward to working more with Chris to see this through
 promulgation.

 + docker

 https://code.launchpad.net/~lazypower/charms/trusty/docker/trunk/+merge/260867

 This was a refresh to v0.1.3 of the docker charm.  We noted some missing
 hooks in the review, but overall deployment and amulet tests looked good.

 We were stumped by a mysterious docker_opts.py module in this ansible
 charm, but luckily had Whit to explain this was a mechanism to translate
 hook events into docker runtime options. We requested this be documented a
 bit better in the class and have some unit tests to make its utility more
 obvious [2].

 + Big 'ol Data
 I forgot to fire off a note to the list about our Q time last week.  For
 those that don't know, Amir, Cory, and I make up the Big Data charmers team
 at Canonical.  Last week we spent some time polishing our own bits and
 we're excited to shepherd these into the store:
   - apache-spark: a fast big data processing engine
 https://bugs.launchpad.net/charms/+bug/1463012
   - apache-zeppelin: notebook for interactive data analytics
 https://bugs.launchpad.net/charms/+bug/1463026
   - apache-hive: data warehouse for query and analytics (think sql)
 https://bugs.launchpad.net/charms/+bug/1468769
   - apache-pig: data analysis with the high-level Pig language
 https://bugs.launchpad.net/charms/+bug/1468768


 That's a wrap from us.  It's been a pleasure!  Find us on freenode #juju
 (asanjar, cory_fu, kwmonroe) with any questions/comments.

 Refs:
 [1]
 http://bazaar.launchpad.net/~xfactor973/charms/trusty/gluster/trunk/view/head:/src/main.rs
 [2] https://github.com/chuckbutler/docker-charm/issues/54

 Thanks!
 -Kevin Monroe

 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju


-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: [Juju] Azure issues fix?

[Samuel Cozannet]

Hey,

Thx, that would help.
What form of documentation / specs are you expecting exactly?

BTW, if that could also be included in Fan Networking, we would have a set
of cool networking features for container workloads.
On Jun 26, 2015 8:22 AM, Charles Butler charles.but...@canonical.com
wrote:

 Hey Sam,

 Back when I was working on the big data eco we ran into this problem
 consistently on many substrates. There is work there that we can probably
 extrapolate into a helper function to eliminate this behavior on even the
 hackiest of situations. I'm not 100% a fan of just tossing a bunch of DNS
 data in /etc/hosts - but it does work.

 We should work together w/ the new work thats been landed in the DNS charm
 to try and offer a proper solution to this that doesn't involve a static
 list in /etc/hosts that is subject to environment changes, and let a proper
 DNS host offer that name resolution in our deployments. I know that I've
 gotten this to work with minmal effort on my internal LAN, which is a far
 cry from a data center, but this works extremely well.

 LMK if you're interested, I'd like to get some requirements established
 and extend the charm to do what you need it to, so we can call this a
 finished line item by adding 1 additional charm to the toplogy and some
 relations.

 All the best,



 Charles Butler charles.but...@canonical.com - Juju Charmer
 Come see the future of datacenter orchestration: http://jujucharms.com

 On Thu, Jun 25, 2015 at 8:17 PM, Samuel Cozannet 
 samuel.cozan...@canonical.com wrote:

 Hi All,

 Been struggling all week to get a decent Sentiment Analysis bundle on
 Azure, while it used to work for ages.

 Apparently, the new Azure after ARM templates is managing DNS a bit
 differently than in the past. There is no (more?) resolution of hostnames
 in the private network, but of the local machine.

 If you have any charm/bundle that uses (private) DNS resolution, I
 encourage you to test them against MS Azure, and eventually enforce charm
 relations to update /etc/hosts on all units, or whatever method to enable
 DNS resolution on, to and from every unit in the deployment.
 This should not break deployments on other clouds, and will re-enable
 Azure if it suddenly started to fail.

 If anyone has more / other info on this matter, or on specifics of Azure,
 I'd be happy hear your feedback.

 Thanks,
 Best,
 Samuel

 --
 Samuel Cozannet
 Cloud, Big Data and IoT Strategy Team
 Business Development - Cloud and ISV Ecosystem
 Changing the Future of Cloud
 Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com /
 Juju https://jujucharms.com
 samuel.cozan...@canonical.com
 mob: +33 616 702 389
 skype: samnco
 Twitter: @SaMnCo_23
 [image: View Samuel Cozannet's profile on LinkedIn]
 https://es.linkedin.com/in/scozannet

 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju



-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Monitoring physical cluster

[Samuel Cozannet]

the agent is subordinate to other workloads. Say you want to monitor mysql:

juju deploy mysql
juju deploy zabbix-agent
juju add-relation zabbix-agent mysql

so if you have 100 nodes, zabbix will automatically be installed on the 100
nodes.

I hope it answers your question,

Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
https://es.linkedin.com/in/scozannet

On Fri, Jun 19, 2015 at 3:29 PM, Daniel Gonzalez danielg...@gmail.com
wrote:

 Thank you for your answer.

 How can I deploy the agent to the nodes? Might I launch it like: juju
 deploy zabbix-agent --to 5?

 Thanks in advance,


 Daniel

 2015-06-19 13:25 GMT+01:00 Samuel Cozannet samuel.cozan...@canonical.com
 :

 One solution would be to use Zabbix + agent, for which we have charms
 that would work on your basis.

 You could derive work from https://github.com/SaMnCo/ob-zabbix where I
 and Nicolas made the integration pretty seamless. See example of use
 integrated with Juju API here :
 https://www.youtube.com/watch?v=Tsa0QgffZ28

 Best,
 Sam

 Best,
 Samuel

 --
 Samuel Cozannet
 Cloud, Big Data and IoT Strategy Team
 Business Development - Cloud and ISV Ecosystem
 Changing the Future of Cloud
 Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com /
 Juju https://jujucharms.com
 samuel.cozan...@canonical.com
 mob: +33 616 702 389
 skype: samnco
 Twitter: @SaMnCo_23
 [image: View Samuel Cozannet's profile on LinkedIn]
 https://es.linkedin.com/in/scozannet

 On Fri, Jun 19, 2015 at 2:21 PM, Daniel Gonzalez danielg...@gmail.com
 wrote:

 Hi everyone,

 I have a small physical cluster running with MaaS and Juju and I was
 thinking how could I deploy a Nagios solution but I want to monitor the
 physical nodes, not the services running in top of them.

 I hope this is the right place to place my question.

 Thanks,

 Daniel

 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju




-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Monitoring physical cluster

[Samuel Cozannet]

One solution would be to use Zabbix + agent, for which we have charms that
would work on your basis.

You could derive work from https://github.com/SaMnCo/ob-zabbix where I and
Nicolas made the integration pretty seamless. See example of use integrated
with Juju API here :https://www.youtube.com/watch?v=Tsa0QgffZ28

Best,
Sam

Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
[image: View Samuel Cozannet's profile on LinkedIn]
https://es.linkedin.com/in/scozannet

On Fri, Jun 19, 2015 at 2:21 PM, Daniel Gonzalez danielg...@gmail.com
wrote:

 Hi everyone,

 I have a small physical cluster running with MaaS and Juju and I was
 thinking how could I deploy a Nagios solution but I want to monitor the
 physical nodes, not the services running in top of them.

 I hope this is the right place to place my question.

 Thanks,

 Daniel

 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju


-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

[Friday Labs] Mesos, ElasticSearch, MongoDB, Cassandra, HDFS are in a box...

[Samuel Cozannet]

Hi All,

Have you ever dreamt of running a full *Mesos* cluster but never had the
time to play with it?
Better, a Mesos cluster running Spark code on *Cassandra*?
Or wait. Maybe I'd also like to have it in *ElasticSearch AND Cassandra*.
Hmm, now we're talking!
*Aerospike* is really fast, I'd also like to have it.
At the same time, for the data lake, you're running *Hadoop* in prod, so
you'd also like to dump your stuff there.
While we are at it, *MongoDB* is nice as well, and our devs could use that
for the node.js apps.
Or maybe you don't want to choose, and you want them ALL at the same time.

You're 20 lines away from having your dream become true. Keep reading.
Spoiler: As you're all Juju users, you're also 35min away from having it
running for real in HA in your preferred cloud

A month ago I attended Strata+Hadoop in London, and I discovered a pretty
awesome piece of technology called Stratio (www.stratio.com)

Stratio is an open source Big Data analytics platform based on Spark. It
uses a data pipeline built on Kafka and Flume, backed by one or more of
Cassandra, MongoDB, ElasticSearch, HDFS or Aerospike (WIP) for the
resilient storage.
Analytics is provided by running Spark either in Standalone or in a Mesos
cluster, managed by ZooKeeper.

The ultimate version of Stratio is called Sparkta, and offers the ability
to describe data processing with a very simple JSON language that tells
input, output, processing to apply etc... (6 words only). Sparkta is due
for GA sometime this month.

Stratio deployer is based on Chef running from a specific node (Stratio
Admin). Hence charming the whole thing was pretty easy as the charm is a
wrapper around the chef based deployer, as if Juju was only managing the
resources and specifying them to Chef Server. Each node is built depending
on the relation that's created with the admin node (ZK, Mesos...).

I also designed 4 reference architectures based on each of the storage
backends. Each reference arch has:
* 1x Stratio Admin (there is no HA yet)
* 3x ZooKeeper
* 2x Mesos Master
* 3 instances of storage, also running Mesos Slaves for data locality. For
HDFS, it's actually 8 nodes (3x data, 3x journal, 2x name)

The code repositories lie in GitHub, but I push version to Launchpad at the
same time in my personal namespace (samuel-cozannet)

* Bundles: https://github.com/SaMnCo/bundle-stratio
* Charms:
  * Admin: https://github.com/SaMnCo/charm-stratio-admin
  * Node: https://github.com/SaMnCo/charm-stratio-node
* Discussion tracker:
https://groups.google.com/forum/?hl=fr#!topic/stratio-admin/KCth-xqZdM4

Next Steps:
* Clean up the code, make it faster (~35min deployment for now, should use
the framework to fasten that up)
* Add a demo use case, with Spark code that runs out of the box
* Charm Sparkta when it's ready. There is little documentation yet as the
project itself if really young. I'll be working with Stratio to make it
happen, hopefully supported by them over time.
* Charm Sparkta dashboard that shows results of analytics

Any feedback/questions more than welcome. I hope you'll find this platform
or some of its components useful. Stratio people are very nice and answer
quickly to questions, don't hesitate to reach out to them.

Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: [Review Queue] New charm: trusty/cassandra

[Samuel Cozannet]

Thanks for sharing Tim.

I want to add that Datastax has recently joined our Charm Partner
Programme. This charm will be the base of their work. They will support it
and port it to support the upcoming DSE 4.7 due this month.

Feel free to ask any question about this,
Thanks,
Samuel

Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23

On Mon, May 18, 2015 at 2:51 PM, Tim Van Steenburgh 
tim.van.steenbu...@canonical.com wrote:

 This is a total rewrite of the Cassandra charm. It supports Apache
 Cassandra 2.0, 2.1 and DataStax Enterprise 4.6.

 It is not backwards compatible with the old precise/cassandra charm that
 only supported earlier versions of Cassandra.

 Many thanks to Stuart Bishop for his work on this charm!

 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju


-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Availability of alpha Easy Juju on Azure disk image

[Samuel Cozannet]

Hi All,

Following Andrew's test, I just released an upgrade. The changelog is:

* Timer on page refresh is now 5sec instead of 1, which gives more time to
upload the file.
* There is a lock mechanism. It's no more possible to have 2 concurrent
bootstrap processes
* Also, even if the bootstrap process is done an you upload a new file, it
will not bootstrap another environment and change your configuration. You
need to delete the previous Azure Settings before you can spin a second
environment.

The upload of the new image is at ~50%, and already completed for North and
West Europe.

I hope you like it!
Best,
Sam


Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23

On Wed, Feb 11, 2015 at 9:10 AM, Samuel Cozannet 
samuel.cozan...@canonical.com wrote:

 Hi Andrew,

 First of all, thanks for having taken the time to try it out! Much
 appreciated.

 Thx for the comments. Point taken. As Marco mentioned this alpha will now
 be productized and maintained properly on Azure. They will get a whole lot
 better very soon.

 In the meantime, if you add a comment on the vm depot to notify the user
 not to try twice times, I think that would help and avoid similar issues
 for others.

 A quick win I can think of is also
 - timer to 5s
 - add a lock file to avoid double execution.

 Best,
 Samuel
 Le 11 févr. 2015 03:37, Andrew Wilkins andrew.wilk...@canonical.com a
 écrit :

 Hi Samuel,

 Looks neat. A few things:
  1. Once the VM is ready, the entire landing page keeps refreshing at 1s
 intervals. That doesn't leave a lot of time to add the .publishsettings
 file and upload.
  2. There's no feedback to say whether or not the .publishsetttings file
 has been uploaded, and whether bootstrapping is underway. I guess I did it
 twice because of point 1, because now I have two juju bootstrap processes
 on the machine :)
  3. When the GUI eventually came up, it wouldn't accept the password that
 the page displayed. This is related to point 2: the page gave me the link
 to the GUI for one env, and the password for the other.

 It'd be great if the upload credentials bit greyed out once uploaded,
 and then under 3. Wait for a few minutes... there was something
 describing the current status. e.g. Installing Juju, Deploying Juju GUI.

 Cheers,
 Andrew

 On Tue, Feb 10, 2015 at 11:01 PM, Samuel Cozannet 
 samuel.cozan...@canonical.com wrote:

 Dear All,

 Yesterday we released an alpha of a new Ubuntu image on MS Azure VM
 Depot: https://vmdepot.msopentech.com/Vhd/Show?vhdId=50248

 If you use that image and spin a VM with it, you'll be able to upload
 your Azure .publishsettings file to its web interface. From that moment,
 the VM will boostrap a Juju environment, install the Juju GUI.
 The main web page auto refreshes, and presents the link to the Juju GUI
 with a password when it's ready.

 The whole process from upload to Juju GUI takes about 10min to complete,
 and does not require any knowledge of Juju or Ubuntu to start playing.
 It leaves you with a fully functional Juju environment using your
 default subscription as your main Azure provider.

 We hope you enjoy that new and easy way to start with Juju. We aim at
 making this image available from the Marketplace when we gather enough
 feedback and fix bugs that remain.

 The whole code and explanations are available on
 https://github.com/SaMnCo/juju-azure. Please use that repo to send
 feedback while we make it better. You can also answer to this thread on the
 list to ask any question or feature request you may have.

 Best,
 Samuel

 --
 Samuel Cozannet
 Cloud, Big Data and IoT Strategy Team
 Business Development - Cloud and ISV Ecosystem
 Changing the Future of Cloud
 Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com /
 Juju https://jujucharms.com
 samuel.cozan...@canonical.com
 mob: +33 616 702 389
 skype: samnco
 Twitter: @SaMnCo_23

 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju



-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Availability of alpha Easy Juju on Azure disk image

[Samuel Cozannet]

Hi Kapil!!

Thanks for the comment! I am glad you like it :)
I actually don't know if it's possible with the other clouds as access to
the APIs is really different (that publishsettings file is really a cool
feature of MS Azure)

We'll be looking at it anyway over the upcoming weeks, to make the
experience better (see Marco's comment up in the chain) and get it beyond
alpha stage.

Best,
Samuel

Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23

On Wed, Feb 11, 2015 at 1:35 PM, Kapil Thangavelu kap...@gmail.com wrote:

 That's pretty awesome for a cloud integrated quickstart. nice work.
 Sounds like it work for aws marketplace and gce click to deploy as well.

 cheers,

 Kapil

 On Wed, Feb 11, 2015 at 3:16 AM, Samuel Cozannet 
 samuel.cozan...@canonical.com wrote:

 Thanks Marco, that is awesome :)

 For those interested, this image comes with a blog post on our insights
 site
 http://insights.ubuntu.com/2015/02/10/create-devops-magic-on-azure-with-canonicals-juju/

 Also available on the Azure channel9 blog
 http://channel9.msdn.com/Blogs/AzurePartner/Guest-Post-Create-DevOps-Magic-on-Azure-with-Canonicals-Juju

 Have fun reading and don't forget to share the love! :)

 Best,
 Sam
  Le 11 févr. 2015 04:26, Marco Ceppi ma...@ondina.co a écrit :

 Hey Andrew,

 I agree with you on the feedback, since it's an HTML page at the moment
 it's hard to incorporate those items, but I've forked the repo and started
 building a lightweight Python app to provide that feedback to the user.

 Marco

 On Tue Feb 10 2015 at 9:37:48 PM Andrew Wilkins 
 andrew.wilk...@canonical.com wrote:

 Hi Samuel,

 Looks neat. A few things:
  1. Once the VM is ready, the entire landing page keeps refreshing at
 1s intervals. That doesn't leave a lot of time to add the .publishsettings
 file and upload.
  2. There's no feedback to say whether or not the .publishsetttings
 file has been uploaded, and whether bootstrapping is underway. I guess I
 did it twice because of point 1, because now I have two juju bootstrap
 processes on the machine :)
  3. When the GUI eventually came up, it wouldn't accept the password
 that the page displayed. This is related to point 2: the page gave me the
 link to the GUI for one env, and the password for the other.

 It'd be great if the upload credentials bit greyed out once uploaded,
 and then under 3. Wait for a few minutes... there was something
 describing the current status. e.g. Installing Juju, Deploying Juju 
 GUI.

 Cheers,
 Andrew

 On Tue, Feb 10, 2015 at 11:01 PM, Samuel Cozannet 
 samuel.cozan...@canonical.com wrote:

 Dear All,

 Yesterday we released an alpha of a new Ubuntu image on MS Azure VM
 Depot: https://vmdepot.msopentech.com/Vhd/Show?vhdId=50248

 If you use that image and spin a VM with it, you'll be able to upload
 your Azure .publishsettings file to its web interface. From that moment,
 the VM will boostrap a Juju environment, install the Juju GUI.
 The main web page auto refreshes, and presents the link to the Juju
 GUI with a password when it's ready.

 The whole process from upload to Juju GUI takes about 10min to
 complete, and does not require any knowledge of Juju or Ubuntu to start
 playing.
 It leaves you with a fully functional Juju environment using your
 default subscription as your main Azure provider.

 We hope you enjoy that new and easy way to start with Juju. We aim at
 making this image available from the Marketplace when we gather enough
 feedback and fix bugs that remain.

 The whole code and explanations are available on
 https://github.com/SaMnCo/juju-azure. Please use that repo to send
 feedback while we make it better. You can also answer to this thread on 
 the
 list to ask any question or feature request you may have.

 Best,
 Samuel

 --
 Samuel Cozannet
 Cloud, Big Data and IoT Strategy Team
 Business Development - Cloud and ISV Ecosystem
 Changing the Future of Cloud
 Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com /
 Juju https://jujucharms.com
 samuel.cozan...@canonical.com
 mob: +33 616 702 389
 skype: samnco
 Twitter: @SaMnCo_23

 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju


 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at: https://lists.ubuntu.com/
 mailman/listinfo/juju


 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju



-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Availability of alpha Easy Juju on Azure disk image

[Samuel Cozannet]

Thanks Marco, that is awesome :)

For those interested, this image comes with a blog post on our insights
site
http://insights.ubuntu.com/2015/02/10/create-devops-magic-on-azure-with-canonicals-juju/

Also available on the Azure channel9 blog
http://channel9.msdn.com/Blogs/AzurePartner/Guest-Post-Create-DevOps-Magic-on-Azure-with-Canonicals-Juju

Have fun reading and don't forget to share the love! :)

Best,
Sam
 Le 11 févr. 2015 04:26, Marco Ceppi ma...@ondina.co a écrit :

 Hey Andrew,

 I agree with you on the feedback, since it's an HTML page at the moment
 it's hard to incorporate those items, but I've forked the repo and started
 building a lightweight Python app to provide that feedback to the user.

 Marco

 On Tue Feb 10 2015 at 9:37:48 PM Andrew Wilkins 
 andrew.wilk...@canonical.com wrote:

 Hi Samuel,

 Looks neat. A few things:
  1. Once the VM is ready, the entire landing page keeps refreshing at 1s
 intervals. That doesn't leave a lot of time to add the .publishsettings
 file and upload.
  2. There's no feedback to say whether or not the .publishsetttings file
 has been uploaded, and whether bootstrapping is underway. I guess I did it
 twice because of point 1, because now I have two juju bootstrap processes
 on the machine :)
  3. When the GUI eventually came up, it wouldn't accept the password that
 the page displayed. This is related to point 2: the page gave me the link
 to the GUI for one env, and the password for the other.

 It'd be great if the upload credentials bit greyed out once uploaded,
 and then under 3. Wait for a few minutes... there was something
 describing the current status. e.g. Installing Juju, Deploying Juju GUI.

 Cheers,
 Andrew

 On Tue, Feb 10, 2015 at 11:01 PM, Samuel Cozannet 
 samuel.cozan...@canonical.com wrote:

 Dear All,

 Yesterday we released an alpha of a new Ubuntu image on MS Azure VM
 Depot: https://vmdepot.msopentech.com/Vhd/Show?vhdId=50248

 If you use that image and spin a VM with it, you'll be able to upload
 your Azure .publishsettings file to its web interface. From that moment,
 the VM will boostrap a Juju environment, install the Juju GUI.
 The main web page auto refreshes, and presents the link to the Juju GUI
 with a password when it's ready.

 The whole process from upload to Juju GUI takes about 10min to complete,
 and does not require any knowledge of Juju or Ubuntu to start playing.
 It leaves you with a fully functional Juju environment using your
 default subscription as your main Azure provider.

 We hope you enjoy that new and easy way to start with Juju. We aim at
 making this image available from the Marketplace when we gather enough
 feedback and fix bugs that remain.

 The whole code and explanations are available on
 https://github.com/SaMnCo/juju-azure. Please use that repo to send
 feedback while we make it better. You can also answer to this thread on the
 list to ask any question or feature request you may have.

 Best,
 Samuel

 --
 Samuel Cozannet
 Cloud, Big Data and IoT Strategy Team
 Business Development - Cloud and ISV Ecosystem
 Changing the Future of Cloud
 Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com /
 Juju https://jujucharms.com
 samuel.cozan...@canonical.com
 mob: +33 616 702 389
 skype: samnco
 Twitter: @SaMnCo_23

 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju


 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at: https://lists.ubuntu.com/
 mailman/listinfo/juju


-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Availability of alpha Easy Juju on Azure disk image

[Samuel Cozannet]

Hi Andrew,

First of all, thanks for having taken the time to try it out! Much
appreciated.

Thx for the comments. Point taken. As Marco mentioned this alpha will now
be productized and maintained properly on Azure. They will get a whole lot
better very soon.

In the meantime, if you add a comment on the vm depot to notify the user
not to try twice times, I think that would help and avoid similar issues
for others.

A quick win I can think of is also
- timer to 5s
- add a lock file to avoid double execution.

Best,
Samuel
Le 11 févr. 2015 03:37, Andrew Wilkins andrew.wilk...@canonical.com a
écrit :

 Hi Samuel,

 Looks neat. A few things:
  1. Once the VM is ready, the entire landing page keeps refreshing at 1s
 intervals. That doesn't leave a lot of time to add the .publishsettings
 file and upload.
  2. There's no feedback to say whether or not the .publishsetttings file
 has been uploaded, and whether bootstrapping is underway. I guess I did it
 twice because of point 1, because now I have two juju bootstrap processes
 on the machine :)
  3. When the GUI eventually came up, it wouldn't accept the password that
 the page displayed. This is related to point 2: the page gave me the link
 to the GUI for one env, and the password for the other.

 It'd be great if the upload credentials bit greyed out once uploaded,
 and then under 3. Wait for a few minutes... there was something
 describing the current status. e.g. Installing Juju, Deploying Juju GUI.

 Cheers,
 Andrew

 On Tue, Feb 10, 2015 at 11:01 PM, Samuel Cozannet 
 samuel.cozan...@canonical.com wrote:

 Dear All,

 Yesterday we released an alpha of a new Ubuntu image on MS Azure VM
 Depot: https://vmdepot.msopentech.com/Vhd/Show?vhdId=50248

 If you use that image and spin a VM with it, you'll be able to upload
 your Azure .publishsettings file to its web interface. From that moment,
 the VM will boostrap a Juju environment, install the Juju GUI.
 The main web page auto refreshes, and presents the link to the Juju GUI
 with a password when it's ready.

 The whole process from upload to Juju GUI takes about 10min to complete,
 and does not require any knowledge of Juju or Ubuntu to start playing.
 It leaves you with a fully functional Juju environment using your default
 subscription as your main Azure provider.

 We hope you enjoy that new and easy way to start with Juju. We aim at
 making this image available from the Marketplace when we gather enough
 feedback and fix bugs that remain.

 The whole code and explanations are available on
 https://github.com/SaMnCo/juju-azure. Please use that repo to send
 feedback while we make it better. You can also answer to this thread on the
 list to ask any question or feature request you may have.

 Best,
 Samuel

 --
 Samuel Cozannet
 Cloud, Big Data and IoT Strategy Team
 Business Development - Cloud and ISV Ecosystem
 Changing the Future of Cloud
 Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com /
 Juju https://jujucharms.com
 samuel.cozan...@canonical.com
 mob: +33 616 702 389
 skype: samnco
 Twitter: @SaMnCo_23

 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju



-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Availability of alpha Easy Juju on Azure disk image

[Samuel Cozannet]

Dear All,

Yesterday we released an alpha of a new Ubuntu image on MS Azure VM Depot:
https://vmdepot.msopentech.com/Vhd/Show?vhdId=50248

If you use that image and spin a VM with it, you'll be able to upload your
Azure .publishsettings file to its web interface. From that moment, the VM
will boostrap a Juju environment, install the Juju GUI.
The main web page auto refreshes, and presents the link to the Juju GUI
with a password when it's ready.

The whole process from upload to Juju GUI takes about 10min to complete,
and does not require any knowledge of Juju or Ubuntu to start playing.
It leaves you with a fully functional Juju environment using your default
subscription as your main Azure provider.

We hope you enjoy that new and easy way to start with Juju. We aim at
making this image available from the Marketplace when we gather enough
feedback and fix bugs that remain.

The whole code and explanations are available on
https://github.com/SaMnCo/juju-azure. Please use that repo to send feedback
while we make it better. You can also answer to this thread on the list to
ask any question or feature request you may have.

Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Problem deploying bundle data-analytics-with-sql-like

[Samuel Cozannet]

Hey Ken,

My advice if you fall into an error state with a bundle like this is to
deploy it charm by charm (or use the deployment script provided), as those
may manage race conditions between the charms.

Let me explain this a little more what we call a race condition.
When deploying complex solutions, it happens that several charms try to
change a relation at the same time. If setting the relation takes too
long, it sometimes fails the next relation changes.
In short, there is a race between 2 services trying to access the same
resource at the same time and that specific resource can only be consumed
once at a time.
This is what is happening to you.


How do we fix that?
* First of all, we have a juju-deployer for bundles that is a lot more
intelligent as the default deployment for bundles (
http://pythonhosted.org/juju-deployer/config.html). I encourage you to use
this for bundles
* You can also retry the hook by typing juju resolved -r yarn-hdfs-master/0
which will tell Juju you fixed the issue and it should retry the previously
failed hook.
* If you're not familiar with the workload and a service broke, maybe the
old reboot trick will fix your issue... juju run --unit yarn-hdfs-master/0
sudo reboot will do... Not guaranteed to work but...
* In the end, this is all because of the charm not managing the status of
his fellow charm in a relation. You can fill a bug vs. that charm and
explain under which conditions you saw the issues, and/or you can also hack
into the code yourself and try to fix it. In your case here, the problem is
probably that the service is still down while trying to join. You may check
the service port is up and running, and wait until it is before actually
firing the hook. (This is a basic but common example).

Hope it helps!
Best,
Samuel





Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23

On Thu, Feb 5, 2015 at 1:55 PM, Ken Williams ke...@theasi.co wrote:


 Hi all,

I'm currently using the 'data-analytics-with-sql-like' bundle.
Sometimes it works fine but sometimes it deploys with an error
and I don't know how to fix it without doing 'destroy-environment' and
deploy again (which takes time).

The error (in 'juju stat') is,

   yarn-hdfs-master/0:
 agent-state: error
 agent-state-info: 'hook failed: namenode-relation-joined for
 compute-node:datanode'
 agent-version: 1.21.1
 machine: 4


 When this error occurs, I can ssh onto yarn-hdfs-master but
 I cannot 'hdfs dfs -put' any data onto hdfs.

 Is there any way I can fix this without destroying the environment
 and deploying again?

 Thankyou for any help, please,


 Ken



 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju


-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: How best to install Spark?

[Samuel Cozannet]

Indeed...

I actually use that in other cases but for some reason I didn't get it
right this time :/. Thanks for catching this!

Best,
Sam

Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23

On Tue, Feb 3, 2015 at 1:43 AM, Andrew Wilkins andrew.wilk...@canonical.com
 wrote:

 On Mon, Feb 2, 2015 at 9:57 PM, Samuel Cozannet 
 samuel.cozan...@canonical.com wrote:

 Excellent! Happy to help you through your discovery of awesomeness with
 Juju :)

 Note that, if you have jq installed (which I advise, sudo apt-get install
 jq)
 juju stat | python -c 'import sys, yaml, json;
 json.dump(yaml.load(sys.stdin), sys.stdout, indent=4)' | jq
 '.services.SERVICE NAME.units.SERVICE NAME/0.machine' | tr -d
 \


 FYI, you can do juju status --format=json and skip the Python bit.

 will return the ID of the machine for SERVICE NAME (replace that by
 yarn-master or what ever the name you gave), which saves you the browsing
 of several pages of juju status...

 Let us know how your testing goes!

 Best,
 Sam


 Best,
 Samuel

 --
 Samuel Cozannet
 Cloud, Big Data and IoT Strategy Team
 Business Development - Cloud and ISV Ecosystem
 Changing the Future of Cloud
 Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com /
 Juju https://jujucharms.com
 samuel.cozan...@canonical.com
 mob: +33 616 702 389
 skype: samnco
 Twitter: @SaMnCo_23

 On Mon, Feb 2, 2015 at 2:52 PM, Ken Williams ke...@theasi.co wrote:

 Hi Sam,

 Just to confirm that deploying the spark-master and the
 yarn-hdfs-master to
 the same machine seems to have worked !  :-)

 // use 'juju status' to find which machine yarn-hdfs-master is on
 juju status
 [ etc...]
 // say...machine: 4

 // deploy spark-master to same machine
 juju deploy --to 4 spark-master

 // add relations
 juju add-relation yarn-hdfs-master:resourcemanager spark-master
 juju add-relation yarn-hdfs-master:namenode spark-master


 // run test
 root@ip-172-31-21-92:~# spark-submit --class
 org.apache.spark.examples.SparkPi /tmp/spark-examples-1.2.0-hadoop2.4.0.jar
 25 --master yarn   --num-executors 3 --driver-memory 1g
 --executor-memory 1g --executor-cores 1 --deploy-mode cluster
 --queue thequeue /tmp/spark-examples-1.2.0-hadoop2.4.0.jar
 Spark assembly has been built with Hive, including Datanucleus jars on
 classpath
 15/02/02 13:40:45 WARN NativeCodeLoader: Unable to load native-hadoop
 library for your platform... using builtin-java classes where applicable
 Pi is roughly 3.1405888


 Many thanks again for all your help,

 Best Regards,

 Ken



 On 30 January 2015 at 18:11, Ken Williams ke...@theasi.co wrote:


 Ok - Sam, I'll try this and let you know.

 Thanks again for all your help,

 Best Regards,

 Ken



 On 30 January 2015 at 18:09, Samuel Cozannet 
 samuel.cozan...@canonical.com wrote:

 I'll have a look asap, but probably not before Tuesday.

 This may be my guts tell me that but, if you have the time, try to
 collocate YARN and Spark, that will guarantee you have the YARN_CONF_DIR
 set. I am 90% sure it will fix your problem.

 YARN itself will not eat much resources, you should be alright and it
 may allow you to move forward instead of being stuck.

 Best,
 Sam






 Best,
 Samuel

 --
 Samuel Cozannet
 Cloud, Big Data and IoT Strategy Team
 Business Development - Cloud and ISV Ecosystem
 Changing the Future of Cloud
 Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com /
 Juju https://jujucharms.com
 samuel.cozan...@canonical.com
 mob: +33 616 702 389
 skype: samnco
 Twitter: @SaMnCo_23

 On Fri, Jan 30, 2015 at 7:01 PM, Ken Williams ke...@theasi.co wrote:

 Hi Sam,

 Attached is my bundles.yaml file.

 Also, there is no file 'directories.sh' on my spark-master/0
 machine (see below),

 ubuntu@ip-172-31-54-245:~$ ls -l /etc/profile.d/
 total 12
 -rw-r--r-- 1 root root 1559 Jul 29  2014 Z97-byobu.sh
 -rwxr-xr-x 1 root root 2691 Oct  6 13:19 Z99-cloud-locale-test.sh
 -rw-r--r-- 1 root root  663 Apr  7  2014 bash_completion.sh
 ubuntu@ip-172-31-54-245:~$


 Many thanks again your help,

 Ken


 On 30 January 2015 at 15:45, Samuel Cozannet 
 samuel.cozan...@canonical.com wrote:

 Hey,

 can you send the bundle you're using (in the GUI, bottom right,
 export button should give you a bundles.yaml file, please send that to
 me, so I can bootstrap the same environment as you are playing with.

 also
 * can you let me know if you have a file
 /etc/profile.d/directories.sh?
 * if yes, can you execute it from your command line, then do the
 spark command again, and let me know?

 Thx,
 Sam









 Best,
 Samuel

 --
 Samuel Cozannet
 Cloud, Big Data and IoT Strategy Team
 Business Development - Cloud and ISV Ecosystem
 Changing the Future of Cloud
 Ubuntu http

Re: flight-delay-demo problem

[Samuel Cozannet]

Hi Ken,

Yeah, the bundle fails because there is a race condition happening. Instead
of quickstarting it, use the deploy script (00-deploy.sh) and you should be
alright.

The most important thing is that you need the Hadoop cluster up  running
(YARN + 4 compute nodes) up and running before you start deploying PIG. By
up and running, I mean everything green on the GUI, including relations.
The script takes care of that as it checks and waits until this is done
before it moves to the next stage.

Note that the deployment takes ~75min to complete, and you need a LOT of
resources (like 5x quad core, 16GB RAM machines). Better do that on a
cloud.

BTW, your command is not correct, remove the -n flight* and only keep the
juju-quickstart bundles.yaml.

Best,
Sam


Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23

On Fri, Jan 30, 2015 at 5:58 PM, Ken Williams ke...@theasi.co wrote:


 Hi,

 I'm trying to install the flight-delay-demo from here


 https://demo.jujucharms.com/~samuel-cozannet/trusty/flight-delay-demo-7

 If I follow the instructions in the Readme file to git clone the code
 and then juju-quickstart the bundle, I get this error,

 root@adminuser-VirtualBox:~#
 root@adminuser-VirtualBox:~# git clone
 https://github.com/SaMnCo/bundle-flight-delay-demo.git flight-delay-demo
 Cloning into 'flight-delay-demo'...
 remote: Counting objects: 75, done.
 remote: Total 75 (delta 0), reused 0 (delta 0)
 Unpacking objects: 100% (75/75), done.
 Checking connectivity... done.
 root@adminuser-VirtualBox:~#
 root@adminuser-VirtualBox:~# cd flight-delay-demo/
 root@adminuser-VirtualBox:~/flight-delay-demo# juju-quickstart -n
 flight-delay-demo bundles.yaml
 WARNING:root:ignoring invalid jenv file amazon.jenv: no state-servers found
 usage: juju-quickstart [-h] [-e ENV_NAME] [-n BUNDLE_NAME] [-i]
[--environments-file ENV_FILE]
[--gui-charm-url CHARM_URL] [--no-browser]
[--distro-only] [--ppa] [--version] [--debug]
[--description] [--upload-tools]
[--upload-series UPLOAD_SERIES]
[--constraints CONSTRAINTS]
[bundle]
 juju-quickstart: error: bundle flight-delay-demo not found in the provided
 list of bundles (envExport)
 root@adminuser-VirtualBox:~/flight-delay-demo#

 Is this bundle ready to be used, or not ?

 Thanks for help,

 Ken




 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju


-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: How best to install Spark?

[Samuel Cozannet]

I'll have a look asap, but probably not before Tuesday.

This may be my guts tell me that but, if you have the time, try to
collocate YARN and Spark, that will guarantee you have the YARN_CONF_DIR
set. I am 90% sure it will fix your problem.

YARN itself will not eat much resources, you should be alright and it may
allow you to move forward instead of being stuck.

Best,
Sam






Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23

On Fri, Jan 30, 2015 at 7:01 PM, Ken Williams ke...@theasi.co wrote:

 Hi Sam,

 Attached is my bundles.yaml file.

 Also, there is no file 'directories.sh' on my spark-master/0 machine
 (see below),

 ubuntu@ip-172-31-54-245:~$ ls -l /etc/profile.d/
 total 12
 -rw-r--r-- 1 root root 1559 Jul 29  2014 Z97-byobu.sh
 -rwxr-xr-x 1 root root 2691 Oct  6 13:19 Z99-cloud-locale-test.sh
 -rw-r--r-- 1 root root  663 Apr  7  2014 bash_completion.sh
 ubuntu@ip-172-31-54-245:~$


 Many thanks again your help,

 Ken


 On 30 January 2015 at 15:45, Samuel Cozannet 
 samuel.cozan...@canonical.com wrote:

 Hey,

 can you send the bundle you're using (in the GUI, bottom right, export
 button should give you a bundles.yaml file, please send that to me, so I
 can bootstrap the same environment as you are playing with.

 also
 * can you let me know if you have a file /etc/profile.d/directories.sh?
 * if yes, can you execute it from your command line, then do the spark
 command again, and let me know?

 Thx,
 Sam









 Best,
 Samuel

 --
 Samuel Cozannet
 Cloud, Big Data and IoT Strategy Team
 Business Development - Cloud and ISV Ecosystem
 Changing the Future of Cloud
 Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com /
 Juju https://jujucharms.com
 samuel.cozan...@canonical.com
 mob: +33 616 702 389
 skype: samnco
 Twitter: @SaMnCo_23

 On Fri, Jan 30, 2015 at 3:46 PM, Ken Williams ke...@theasi.co wrote:

 Ok - I have been able to add the relation using this,

 juju add-relation yarn-hdfs-master:resourcemanager
 spark-master

 But I still cannot see a /etc/hadoop/conf directory on the spark-master
 machine
 so I still get the same error about HADOOP_CONF_DIR and YARN_CONF_DIR
 (below),


 root@ip-172-31-60-53:~# spark-submit --class
 org.apache.spark.examples.SparkPi --master yarn-client
 --num-executors 3 --driver-memory 1g --executor-memory 1g
 --executor-cores 1 --queue thequeue lib/spark-examples*.jar 10
 Spark assembly has been built with Hive, including Datanucleus jars on
 classpath
 Exception in thread main java.lang.Exception: When running with master
 'yarn-client' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the
 environment.
 at
 org.apache.spark.deploy.SparkSubmitArguments.checkRequiredArguments(SparkSubmitArguments.scala:177)
 at
 org.apache.spark.deploy.SparkSubmitArguments.init(SparkSubmitArguments.scala:81)
 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:70)
 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
 root@ip-172-31-60-53:~#

 Should there be a /etc/hadoop/conf directory ?

 Thanks for any help,

 Ken


 On 30 January 2015 at 12:59, Samuel Cozannet 
 samuel.cozan...@canonical.com wrote:

 Have you tried without ':master:

 juju add-relation yarn-hdfs-master:resourcemanager spark-master

 I think Spark master consumes the relationship but doesn't have to
 expose its master relationship.

 Rule of thumb, when a relation is non ambiguous on one of its ends,
 there is no requirement to specify it when adding it.

 Another option if this doesn't work is to use the GUI to create the
 relation. It will give you a dropdown of available relationships between
 entities.

 Let me know how it goes,
 Thx,
 Sam


 Best,
 Samuel

 --
 Samuel Cozannet
 Cloud, Big Data and IoT Strategy Team
 Business Development - Cloud and ISV Ecosystem
 Changing the Future of Cloud
 Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com /
 Juju https://jujucharms.com
 samuel.cozan...@canonical.com
 mob: +33 616 702 389
 skype: samnco
 Twitter: @SaMnCo_23

 On Fri, Jan 30, 2015 at 1:09 PM, Ken Williams ke...@theasi.co wrote:

 Hi Sam,

 I understand what you are saying but when I try to add the 2
 relations I get this error,

 root@adminuser-VirtualBox:~# juju add-relation
 yarn-hdfs-master:resourcemanager spark-master:master
 ERROR no relations found
 root@adminuser-VirtualBox:~# juju add-relation
 yarn-hdfs-master:namenode spark-master:master
 ERROR no relations found

   Am I adding the relations right ?

   Attached is my 'juju status' file.

   Thanks for all your help,

 Ken





 On 30 January 2015 at 11:16, Samuel Cozannet 
 samuel.cozan...@canonical.com wrote:

 Hey Ken,

 Yes, you need to create

Re: How best to install Spark?

[Samuel Cozannet]

Have you tried without ':master:

juju add-relation yarn-hdfs-master:resourcemanager spark-master

I think Spark master consumes the relationship but doesn't have to expose
its master relationship.

Rule of thumb, when a relation is non ambiguous on one of its ends, there
is no requirement to specify it when adding it.

Another option if this doesn't work is to use the GUI to create the
relation. It will give you a dropdown of available relationships between
entities.

Let me know how it goes,
Thx,
Sam


Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23

On Fri, Jan 30, 2015 at 1:09 PM, Ken Williams ke...@theasi.co wrote:

 Hi Sam,

 I understand what you are saying but when I try to add the 2 relations
 I get this error,

 root@adminuser-VirtualBox:~# juju add-relation
 yarn-hdfs-master:resourcemanager spark-master:master
 ERROR no relations found
 root@adminuser-VirtualBox:~# juju add-relation yarn-hdfs-master:namenode
 spark-master:master
 ERROR no relations found

   Am I adding the relations right ?

   Attached is my 'juju status' file.

   Thanks for all your help,

 Ken





 On 30 January 2015 at 11:16, Samuel Cozannet 
 samuel.cozan...@canonical.com wrote:

 Hey Ken,

 Yes, you need to create the relationship between the 2 entities to they
 know about each other.

 Looking at the list of hooks for the charm
 https://github.com/Archethought/spark-charm/tree/master/hooks you can
 see there are 2 hooks named namenode-relation-changed
 https://github.com/Archethought/spark-charm/blob/master/hooks/namenode-relation-changed
  and resourcemanager-relation-changed
 https://github.com/Archethought/spark-charm/blob/master/hooks/resourcemanager-relation-changed
  which
 are related to YARN/Hadoop.
 Looking deeper in the code, you'll notice they reference a function found
 in bdutils.py called setHadoopEnvVar(), which based on its name should
 set the HADOOP_CONF_DIR.

 There are 2 relations, so add both of them.

 Note that I didn't test this myself, but I expect this should fix the
 problem. If it doesn't please come back to us...

 Thanks!
 Sam


 Best,
 Samuel

 --
 Samuel Cozannet
 Cloud, Big Data and IoT Strategy Team
 Business Development - Cloud and ISV Ecosystem
 Changing the Future of Cloud
 Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com /
 Juju https://jujucharms.com
 samuel.cozan...@canonical.com
 mob: +33 616 702 389
 skype: samnco
 Twitter: @SaMnCo_23

 On Fri, Jan 30, 2015 at 11:51 AM, Ken Williams ke...@theasi.co wrote:


 Thanks, Kapil - this works :-)

 I can now run the SparkPi example successfully.
 root@ip-172-31-60-53:~# spark-submit --class
 org.apache.spark.examples.SparkPi /tmp/spark-examples-1.2.0-hadoop2.4.0.jar
 Spark assembly has been built with Hive, including Datanucleus jars on
 classpath
 15/01/30 10:29:33 WARN NativeCodeLoader: Unable to load native-hadoop
 library for your platform... using builtin-java classes where applicable
 Pi is roughly 3.14318

 root@ip-172-31-60-53:~#

 I'm now trying to run the same example with the spark-submit '--master'
 option set to either 'yarn-cluster' or 'yarn-client'
 but I keep getting the same error :

 root@ip-172-31-60-53:~# spark-submit --class
 org.apache.spark.examples.SparkPi --master yarn-client
 --num-executors 3 --driver-memory 1g --executor-memory 1g
 --executor-cores 1 --queue thequeue lib/spark-examples*.jar 10
 Spark assembly has been built with Hive, including Datanucleus jars on
 classpath
 Exception in thread main java.lang.Exception: When running with master
 'yarn-client' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the
 environment.

 But on my spark-master/0 machine there is no /etc/hadoop/conf directory.
 So what should the HADOOP_CONF_DIR or YARN_CONF_DIR value be ?
 Do I need to add a juju relation between spark-master and ...
 yarn-hdfs-master to make them aware of each other ?

 Thanks for any help,

 Ken





 On 28 January 2015 at 19:32, Kapil Thangavelu 
 kapil.thangav...@canonical.com wrote:



 On Wed, Jan 28, 2015 at 1:54 PM, Ken Williams ke...@theasi.co wrote:


 Hi Sam/Amir,

 I've been able to 'juju ssh spark-master/0' and I successfully ran
 the two
 simple examples for pyspark and spark-shell,

 ./bin/pyspark
  sc.parallelize(range(1000)).count()
 1000

 ./bin/spark-shell
  scala sc.parallelize(1 to 1000).count()
 1000


 Now I want to run some of the spark examples in the spark-exampes*.jar
 file, which I have on my local machine. How do I copy the jar file from
 my local machine to the AWS machine ?

 I have tried 'scp' and 'juju scp' from the local command-line but both
 fail (below),

 root@adminuser:~# scp /tmp/spark-examples-1.2.0

Re: How best to install Spark?

[Samuel Cozannet]

Hey Ken,

Yes, you need to create the relationship between the 2 entities to they
know about each other.

Looking at the list of hooks for the charm
https://github.com/Archethought/spark-charm/tree/master/hooks you can see
there are 2 hooks named namenode-relation-changed
https://github.com/Archethought/spark-charm/blob/master/hooks/namenode-relation-changed
 and resourcemanager-relation-changed
https://github.com/Archethought/spark-charm/blob/master/hooks/resourcemanager-relation-changed
which
are related to YARN/Hadoop.
Looking deeper in the code, you'll notice they reference a function found
in bdutils.py called setHadoopEnvVar(), which based on its name should
set the HADOOP_CONF_DIR.

There are 2 relations, so add both of them.

Note that I didn't test this myself, but I expect this should fix the
problem. If it doesn't please come back to us...

Thanks!
Sam


Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23

On Fri, Jan 30, 2015 at 11:51 AM, Ken Williams ke...@theasi.co wrote:


 Thanks, Kapil - this works :-)

 I can now run the SparkPi example successfully.
 root@ip-172-31-60-53:~# spark-submit --class
 org.apache.spark.examples.SparkPi /tmp/spark-examples-1.2.0-hadoop2.4.0.jar
 Spark assembly has been built with Hive, including Datanucleus jars on
 classpath
 15/01/30 10:29:33 WARN NativeCodeLoader: Unable to load native-hadoop
 library for your platform... using builtin-java classes where applicable
 Pi is roughly 3.14318

 root@ip-172-31-60-53:~#

 I'm now trying to run the same example with the spark-submit '--master'
 option set to either 'yarn-cluster' or 'yarn-client'
 but I keep getting the same error :

 root@ip-172-31-60-53:~# spark-submit --class
 org.apache.spark.examples.SparkPi --master yarn-client
 --num-executors 3 --driver-memory 1g --executor-memory 1g
 --executor-cores 1 --queue thequeue lib/spark-examples*.jar 10
 Spark assembly has been built with Hive, including Datanucleus jars on
 classpath
 Exception in thread main java.lang.Exception: When running with master
 'yarn-client' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the
 environment.

 But on my spark-master/0 machine there is no /etc/hadoop/conf directory.
 So what should the HADOOP_CONF_DIR or YARN_CONF_DIR value be ?
 Do I need to add a juju relation between spark-master and ...
 yarn-hdfs-master to make them aware of each other ?

 Thanks for any help,

 Ken





 On 28 January 2015 at 19:32, Kapil Thangavelu 
 kapil.thangav...@canonical.com wrote:



 On Wed, Jan 28, 2015 at 1:54 PM, Ken Williams ke...@theasi.co wrote:


 Hi Sam/Amir,

 I've been able to 'juju ssh spark-master/0' and I successfully ran
 the two
 simple examples for pyspark and spark-shell,

 ./bin/pyspark
  sc.parallelize(range(1000)).count()
 1000

 ./bin/spark-shell
  scala sc.parallelize(1 to 1000).count()
 1000


 Now I want to run some of the spark examples in the spark-exampes*.jar
 file, which I have on my local machine. How do I copy the jar file from
 my local machine to the AWS machine ?

 I have tried 'scp' and 'juju scp' from the local command-line but both
 fail (below),

 root@adminuser:~# scp /tmp/spark-examples-1.2.0-hadoop2.4.0.jar
 ubuntu@ip-172-31-59:/tmp
 ssh: Could not resolve hostname ip-172-31-59: Name or service not known
 lost connection
 root@adminuser:~# juju scp /tmp/spark-examples-1.2.0-hadoop2.4.0.jar
 ubuntu@ip-172-31-59:/tmp
 ERROR exit status 1 (nc: getaddrinfo: Name or service not known)

 Any ideas ?


 juju scp /tmp/spark-examples-1.2.0-hadoop2.4.0.jar spark-master/0:/tmp


 Ken











 On 28 January 2015 at 17:29, Samuel Cozannet 
 samuel.cozan...@canonical.com wrote:

 Glad it worked!

 I'll make a merge request to the upstream so that it works natively
 from the store asap.

 Thanks for catching that!
 Samuel

 Best,
 Samuel

 --
 Samuel Cozannet
 Cloud, Big Data and IoT Strategy Team
 Business Development - Cloud and ISV Ecosystem
 Changing the Future of Cloud
 Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com /
 Juju https://jujucharms.com
 samuel.cozan...@canonical.com
 mob: +33 616 702 389
 skype: samnco
 Twitter: @SaMnCo_23

 On Wed, Jan 28, 2015 at 6:15 PM, Ken Williams ke...@theasi.co wrote:


 Hi Sam (and Maarten),

 Cloning Spark 1.2.0 from github seems to have worked!
 I can install the Spark examples afterwards.

 Thanks for all your help!

 Yes - Andrew and Angie both say 'hi'  :-)

 Best Regards,

 Ken


 On 28 January 2015 at 16:43, Samuel Cozannet 
 samuel.cozan...@canonical.com wrote:

 Hey Ken,

 So I had a closer look to your Spark problem and found out what went
 wrong.

 The charm available on the charmstore

Re: How best to install Spark?

[Samuel Cozannet]

Glad it worked!

I'll make a merge request to the upstream so that it works natively from
the store asap.

Thanks for catching that!
Samuel

Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23

On Wed, Jan 28, 2015 at 6:15 PM, Ken Williams ke...@theasi.co wrote:


 Hi Sam (and Maarten),

 Cloning Spark 1.2.0 from github seems to have worked!
 I can install the Spark examples afterwards.

 Thanks for all your help!

 Yes - Andrew and Angie both say 'hi'  :-)

 Best Regards,

 Ken


 On 28 January 2015 at 16:43, Samuel Cozannet 
 samuel.cozan...@canonical.com wrote:

 Hey Ken,

 So I had a closer look to your Spark problem and found out what went
 wrong.

 The charm available on the charmstore is trying to download Spark 1.0.2,
 and the versions available on the Apache website are 1.1.0, 1.1.1 and
 1.2.0.

 There is another version of the charm available on GitHub that actually
 will deploy 1.2.0

 1. On your computer, the below folders  get there:

 cd ~
 mkdir charms
 mkdir charms/trusty
 cd charms/trusty

 2. Branch the Spark charm.

 git clone https://github.com/Archethought/spark-charm spark

 3. Deploy Spark from local repository

 juju deploy --repository=~/charms local:trusty/spark spark-master
 juju deploy --repository=~/charms local:trusty/spark spark-slave
 juju add-relation spark-master:master spark-slave:slave

 Worked on AWS for me just minutes ago. Let me know how it goes for you.
 Note that this version of the charm does NOT install the Spark examples.
 The files are present though, so you'll find them in
 /var/lib/juju/agents/unit-spark-master-0/charm/files/archive

 Hope that helps...
 Let me know if it works for you!

 Best,
 Sam


 Best,
 Samuel

 --
 Samuel Cozannet
 Cloud, Big Data and IoT Strategy Team
 Business Development - Cloud and ISV Ecosystem
 Changing the Future of Cloud
 Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com /
 Juju https://jujucharms.com
 samuel.cozan...@canonical.com
 mob: +33 616 702 389
 skype: samnco
 Twitter: @SaMnCo_23

 On Wed, Jan 28, 2015 at 4:44 PM, Ken Williams ke...@theasi.co wrote:


 Hi folks,

 I'm completely new to juju so any help is appreciated.

 I'm trying to create a hadoop/analytics-type platform.

 I've managed to install the 'data-analytics-with-sql-like' bundle
 (using this command)

 juju quickstart
 bundle:data-analytics-with-sql-like/data-analytics-with-sql-like

 This is very impressive, and gives me virtually everything that I want
 (hadoop, hive, etc) - but I also need Spark.

 The Spark charm (http://manage.jujucharms.com/~asanjar/trusty/spark)
 and bundle (
 http://manage.jujucharms.com/bundle/~asanjar/spark/spark-cluster)
 however do not seem stable or available and I can't figure out how to
 install them.

 Should I just download and install the Spark tar-ball on the nodes
 in my AWS cluster, or is there a better way to do this ?

 Thanks in advance,

 Ken


 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju




-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: How best to install Spark?

[Samuel Cozannet]

Hi Ken!

Good to know you like our charms and bundles! Are you working with Andrew 
Angie?

I have been talking with them several times, so I have a little bit of
background on your use cases. Let me know if you want to do a short hangout
to discuss your specific workload.

Specifically, if you want to use Spark in conjunction with Hadoop, you
probably want to deploy it on the same node as your YARN master. So
assuming you deployed it and named it yarn-master, you can do: (install jq
first with sudo apt-get install jq)

 TARGET_MACHINE=$(juju stat | python -c 'import sys, yaml, json;
json.dump(yaml.load(sys.stdin), sys.stdout, indent=4)' | jq
'.services.yarn-master.units.yarn-master/0.machine' | tr -d \ )

== This command will output the ID of the machine running the YARN master.

Then

juju deploy --to $TARGET_MACHINE cs:~asanjar/trusty/spark spark-master

Then you'll be able to read from Hadoop into Spark.



Best,
Samuel





Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23

On Wed, Jan 28, 2015 at 4:44 PM, Ken Williams ke...@theasi.co wrote:


 Hi folks,

 I'm completely new to juju so any help is appreciated.

 I'm trying to create a hadoop/analytics-type platform.

 I've managed to install the 'data-analytics-with-sql-like' bundle
 (using this command)

 juju quickstart
 bundle:data-analytics-with-sql-like/data-analytics-with-sql-like

 This is very impressive, and gives me virtually everything that I want
 (hadoop, hive, etc) - but I also need Spark.

 The Spark charm (http://manage.jujucharms.com/~asanjar/trusty/spark)
 and bundle (
 http://manage.jujucharms.com/bundle/~asanjar/spark/spark-cluster)
 however do not seem stable or available and I can't figure out how to
 install them.

 Should I just download and install the Spark tar-ball on the nodes
 in my AWS cluster, or is there a better way to do this ?

 Thanks in advance,

 Ken


 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju


-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: How best to install Spark?

[Samuel Cozannet]

Hey Ken,

So I had a closer look to your Spark problem and found out what went wrong.

The charm available on the charmstore is trying to download Spark 1.0.2,
and the versions available on the Apache website are 1.1.0, 1.1.1 and
1.2.0.

There is another version of the charm available on GitHub that actually
will deploy 1.2.0

1. On your computer, the below folders  get there:

cd ~
mkdir charms
mkdir charms/trusty
cd charms/trusty

2. Branch the Spark charm.

git clone https://github.com/Archethought/spark-charm spark

3. Deploy Spark from local repository

juju deploy --repository=~/charms local:trusty/spark spark-master
juju deploy --repository=~/charms local:trusty/spark spark-slave
juju add-relation spark-master:master spark-slave:slave

Worked on AWS for me just minutes ago. Let me know how it goes for you.
Note that this version of the charm does NOT install the Spark examples.
The files are present though, so you'll find them in
/var/lib/juju/agents/unit-spark-master-0/charm/files/archive

Hope that helps...
Let me know if it works for you!

Best,
Sam


Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Business Development - Cloud and ISV Ecosystem
Changing the Future of Cloud
Ubuntu http://ubuntu.com  / Canonical UK LTD http://canonical.com / Juju
https://jujucharms.com
samuel.cozan...@canonical.com
mob: +33 616 702 389
skype: samnco
Twitter: @SaMnCo_23

On Wed, Jan 28, 2015 at 4:44 PM, Ken Williams ke...@theasi.co wrote:


 Hi folks,

 I'm completely new to juju so any help is appreciated.

 I'm trying to create a hadoop/analytics-type platform.

 I've managed to install the 'data-analytics-with-sql-like' bundle
 (using this command)

 juju quickstart
 bundle:data-analytics-with-sql-like/data-analytics-with-sql-like

 This is very impressive, and gives me virtually everything that I want
 (hadoop, hive, etc) - but I also need Spark.

 The Spark charm (http://manage.jujucharms.com/~asanjar/trusty/spark)
 and bundle (
 http://manage.jujucharms.com/bundle/~asanjar/spark/spark-cluster)
 however do not seem stable or available and I can't figure out how to
 install them.

 Should I just download and install the Spark tar-ball on the nodes
 in my AWS cluster, or is there a better way to do this ?

 Thanks in advance,

 Ken


 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju


-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

[New Charm] iPython Notebook available for testing

[Samuel Cozannet]

Hi All,

Following a request from a friend doing data science training I took some
time to release an early and very simple iPython Notebook Server charm.

I welcome additions, bugs from both Launchpad and GitHub:
* https://github.com/SaMnCo
*
https://code.launchpad.net/~samuel-cozannet/charms/trusty/ipython-notebook/trunk

I only tested it locally and on AWS so far. If someone is happy to deploy
on Rackspace and tell me if OK I could use that help. Don't hesitate to
fire an email to tell me about features you'd like or missing libraries.

I just uploaded it to the charmstore to my personal namespace so give it a
few minutes for the GUI.
You can already install it with :

juju deploy cs:~samuel-cozannet/trusty/ipython-notebook
juju expose ipython-notebook

then browse to https://unit-ip

Let me know your thoughts,

Enjoy,
Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Strategic Program Manager
Changing the Future of Cloud
Ubuntu http://ubuntu.com / Canonical http://canonical.com UK LTD
samuel.cozan...@canonical.com
+33 616 702 389
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Recommendation engine on top hdp-hadoop-cluster and mahout

[Samuel Cozannet]

Hi All,

Following the announcement of the latest HDP Hadoop bundle by LazyPower and
Amir, I built a demo to run a recommendation engine on top of it.

You can find the code here :
https://github.com/SaMnCo/juju-hadoop-recommendation-engine

Notes:
* The script will install Mahout and Snappy compression lib. This will in
the end be included in the charm.
* It has to run once to learn the recommendation from the source file
* The output is a big matrix and there is a small python script provided to
extract meaningful results
* Wait until the YARN resource manager is started before installing and
running
* Tested on AWS only
* The initial workload uses 100k movie recommendations but there are 1M and
10M files available (see config in the demo script)
* The inspiration for this was built for Hadoop Vanilla. Most things can be
ported to that environment as well.

If anyone is willing to help on a small web GUI, that could make a nice
demo for our beloved SEs.

Hope you like it!
Best,
Sam

-- 
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Strategic Program Manager
Changing the Future of Cloud
Ubuntu http://ubuntu.com / Canonical http://canonical.com UK LTD
samuel.cozan...@canonical.com
+33 616 702 389
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Juju devel 1.21-alpha1 is released

[Samuel Cozannet]

, set the
 'disable-network-management' option in environments.yaml to true:

 disable-network-management: true

 This tells Juju not to create a network bridge or bringing eth0
 up and down during cloudinit. Juju will not make changes to the
 network config when its agents start.


 Performing autopsies on failed bootstraps

 The juju 'bootstrap' command has a new option for testers and anyone
 examining why a bootstrap failed. Use the '--keep-broken' option to
 keep the machine up. You can then use ssh to gather logs and
 investigate the cause of the failure.


 Resolved issues

 * Maas provider assumes machine uses dhcp for eth0
   Lp 1361374

 * Relation-get with invalid relation name panics agent
   Lp 1365412

 * We should remove direct db access for clients
   Lp 1253652

 * Allow specifying a key when doing manual provisioning
   Lp 1270466

 * Juju doesn't use maas' knowledge of system architecture when picking
   tools
   Lp 1303853

 * Juju add-machine still assumes precise (maas)
   Lp 1315473

 * Local provider is very slow to tranistion from agent-status: pending
   Lp 1322302

 * Juju should wrap apt-get invocations with eatmydata when
   provisioning cloud instances
   Lp 1335822

 * Juju 1.21-alpha1 local provider does not create all-machines.log
   Lp 1339715

 * Cloudinit does not use ssh client
   Lp 1339976

 * Provisioner-safe-mode is undocumented
   Lp 1342729

 * Networker restarts every 3 seconds with the local provider (missing
   /etc/network/interfaces)
   Lp 1343219

 * Describe harvesting strategy rather than using safe mode name
   Lp 1345553

 * Configstore: if the size of the serialised jenv decreases the .jenv
   file will be corrupt
   Lp 1348458

 * Juju-core client panics with juju set empty string
   Lp 1348829

 * Juju ignores environments.yaml on failed bootstrap if $provider.jenv
   exists
   Lp 1361680

 * Saved addresses should omit link-local addresses
   Lp 1362453

 * Add-machine containers should default to latest lts
   Lp 1363971

 * Blobstore's hashing needs improvement
   Lp 1364750

 * --keep-broken option still allows instance to be stopped
   Lp 1365772

 * Removing a unit on an unclean machine should remove that machine
   Lp 1206532

 * Juju log files should not be world readable
   Lp 1286518

 * Juju uses hard-coded regions
   Lp 1319474

 * Cmd/juju: deploy --to a non existent machine fails too late in the
   process
   Lp 1212538

 * Cmd/juju: add-machine should take a -n param
   Lp 1214209

 * Missing @ syntax for reading config setting from file content
   Lp 1216967

 * Container provisioner may choose bad tools
   Lp 1347984

 * Juju set help is written but not shown
   Lp 1359187


 Finally

 We encourage everyone to subscribe the mailing list at
 juju-...@lists.canonical.com, or join us on #juju-dev on freenode.

 --
 Curtis Hovey
 Canonical Cloud Development and Operations
 http://launchpad.net/~sinzui

 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju




-- 
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Strategic Program Manager
Changing the Future of Cloud
Ubuntu http://ubuntu.com / Canonical http://canonical.com UK LTD
samuel.cozan...@canonical.com
+33 616 702 389
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju