Re: Best practices for fat charms
Resending this to the list, rather than just Jorge, sorry. On 1 April 2014 20:07, Jorge O. Castro jo...@ubuntu.com wrote: Hi everyone, Matt Bruzek and I have been doing some charm testing on a machine that does not have general access to the internet. So charms that pull from PPAs, github, etc. do not work. We've been able to fatten the charms by doing things like creating a /files directory in the charm itself and putting the package/tarball/jar file in there, and given the networking issues that we might face in production environments that we should start thinking about best practices for having charms with payloads instead of pulling from a network source. Great question :) So we have been deploying fat charms in a restricted environment like this since we started using juju. Our build machines have no internet access, but they *can* access the ubuntu archives, a private archive, and launchpad, and other internal services. For us there are two kinds of charms that need different work to support these restrictions. One type is the regular charmstore charms than make up your infrastructure glue (like apache2, haproxy, squid, postgresql, etc). These kinds of packages don't change their core software payload often, usually on stable releases. For charms like these that install things that are not in the ubuntu archives, we fork and modify them as necessary to support installing from a custom archive, for which we build a package ourselves. A good example of a charm that works well this way is elastic search, which uses the vendor packages by default, but allows you to specify a ppa or archive to install from instead in the config. Many other charms also do this, but not all, so it's worth noting I think. The other type is application charms, which are typically private charms where the payload is your application, and you can change the core software payload multiple times a day. For these charms (our core workload) we do a fat charm as you suggest above. 1) check out pinned version of the charms into a local repository 2) run a build step over the local repository that looks for a Makefile with a charm-payload target in each local charm's dir. 3) if found, it is run, which pulls deps and builds. Source deps have to be mirrored in lp, package deps not in main need to be added to the private archive. A build will fail if it tries to reach something it can't. 4) this produces some artifact(s) in the charms files/ directory (tarballs, executable, scripts, etc). 5) a subsequent deploy/upgrade then ships these out to the charm dir on the unit, and the hooks unpack as appropriate for that project. This works reasonably well. It has a few downsides: a) conflates charm upgrade with code deployment. This required some hacks with the charm revision (now fixed I think). b) some build deps on the build machine need manual management. c) the build step is repeated at each stage: dev, CI, staging, production, which is wasteful and error prone. One of the things we are looking at at the moment is to deploy the build artifacts from an internal trusted url. For us, this would be an openstack swift bucket. The ideal goal is that if the CI build/tests succeed, it would deposit the build artifacts in a versioned swift url. A charm's config can then be updated to the new url. This potentially allows us to a) deploy code separately to charm upgrades b) reuse the same build artifact for CI/staging/dev, c) roll back easily with a config change d) manually do rolling upgrades via juju run if needed. Michael Nelson's been working on this, I expect he'll have more to add. So in essence we are thinking of trying to move away from fat charms, and use build artifacts at trusted urls to get the payload on the units, for the reasons above. Some final thoughts: We're still looking to simultaneously support the fat charm approach of bundling payload with upgrade-charm as well, as it's really nice for upgrading code and charm in a single juju action, which the url approach doesn't do. Our build artifacts are usually tarballs for python projects, and binaries for go projects, plus an assortment of scripts. I am planning to look at PEX files for python, as well as maybe docker images, to see if this can be simplified further and made more robust. HTH On 1 April 2014 20:07, Jorge O. Castro jo...@ubuntu.com wrote: Hi everyone, Matt Bruzek and I have been doing some charm testing on a machine that does not have general access to the internet. So charms that pull from PPAs, github, etc. do not work. We've been able to fatten the charms by doing things like creating a /files directory in the charm itself and putting the package/tarball/jar file in there, and given the networking issues that we might face in production environments that we should start thinking about best practices for having charms with payloads instead of pulling from a network source. Marco has some ideas on how we can
Re: Best practices for fat charms
fwiw. deployer/bundles have support for an explicit build phase for this reason.. basically a build hook in charms is run prior to deploying. i'd like to push it a bit further to deployer bundles as an archive format that can be completely self-contained for an app. On Tue, Apr 1, 2014 at 3:07 PM, Jorge O. Castro jo...@ubuntu.com wrote: Hi everyone, Matt Bruzek and I have been doing some charm testing on a machine that does not have general access to the internet. So charms that pull from PPAs, github, etc. do not work. We've been able to fatten the charms by doing things like creating a /files directory in the charm itself and putting the package/tarball/jar file in there, and given the networking issues that we might face in production environments that we should start thinking about best practices for having charms with payloads instead of pulling from a network source. Marco has some ideas on how we can generalize this and he will respond to this thread. -- Jorge Castro Canonical Ltd. http://juju.ubuntu.com/ - Automate your Cloud Infrastructure -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Best practices for fat charms
Thanks Jorge, Not sure we want to call them fat charms, maybe enterprise charms. Here is my approach when making a charm work on the enterprise or limited networks. 1) Find out what hook downloads the packages that we are unable to access (wget, curl, or special ppa repositories). The enterprise network will block these requests often resulting in a charm hook failing. 2) Download the necessary packages from system that has access. 3) Upload the packages to the locked down system, copying the packages to a directory on the local charm. 4) Edit the local charm hooks to check for the package in the local directory first and if that does not exist, the charm would continue to download the files (using wget, or curl, or custom ppa). I believe we could provide a charm-tools method that does something like this and we could use this in charms to create enterprise charms that are able to be used on limited network environments. However this creates an interesting problem that I have not figured out a good way to resolve yet (your feedback requested). If the packages exist within the charm the URL will never be used. Some charms allow the user to configure the download URL and sha1sum of the package. Other charms do not allow this level of customization. For charms that have a config option for URL we could change it to also accept a file:// transport and some kind of $CHARM_DIR variable and use http:// or https:// for normal URLs. But what to do with the charms that do not allow the URLs to be configurable, or the charms that use a custom ppa repository? - Matthew Bruzek matthew.bru...@canonical.com On Tue, Apr 1, 2014 at 2:07 PM, Jorge O. Castro jo...@ubuntu.com wrote: Hi everyone, Matt Bruzek and I have been doing some charm testing on a machine that does not have general access to the internet. So charms that pull from PPAs, github, etc. do not work. We've been able to fatten the charms by doing things like creating a /files directory in the charm itself and putting the package/tarball/jar file in there, and given the networking issues that we might face in production environments that we should start thinking about best practices for having charms with payloads instead of pulling from a network source. Marco has some ideas on how we can generalize this and he will respond to this thread. -- Jorge Castro Canonical Ltd. http://juju.ubuntu.com/ - Automate your Cloud Infrastructure -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Best practices for fat charms
If your trying to do this in automated fashion, juju supports proxies, and possibily with intelligent proxy you could do something a bit more automated. else its going to require alot of auditing. you could even skip the additional steps of modifying all the charms have the intelligent proxy work in offline mode with its cache to serve out the files back when deploying in an offline setup. the issue is going to be https url then. On Tue, Apr 1, 2014 at 4:05 PM, Matt Bruzek matthew.bru...@canonical.comwrote: Thanks Jorge, Not sure we want to call them fat charms, maybe enterprise charms. Here is my approach when making a charm work on the enterprise or limited networks. 1) Find out what hook downloads the packages that we are unable to access (wget, curl, or special ppa repositories). The enterprise network will block these requests often resulting in a charm hook failing. 2) Download the necessary packages from system that has access. 3) Upload the packages to the locked down system, copying the packages to a directory on the local charm. 4) Edit the local charm hooks to check for the package in the local directory first and if that does not exist, the charm would continue to download the files (using wget, or curl, or custom ppa). I believe we could provide a charm-tools method that does something like this and we could use this in charms to create enterprise charms that are able to be used on limited network environments. However this creates an interesting problem that I have not figured out a good way to resolve yet (your feedback requested). If the packages exist within the charm the URL will never be used. Some charms allow the user to configure the download URL and sha1sum of the package. Other charms do not allow this level of customization. For charms that have a config option for URL we could change it to also accept a file:// transport and some kind of $CHARM_DIR variable and use http:// or https:// for normal URLs. But what to do with the charms that do not allow the URLs to be configurable, or the charms that use a custom ppa repository? - Matthew Bruzek matthew.bru...@canonical.com On Tue, Apr 1, 2014 at 2:07 PM, Jorge O. Castro jo...@ubuntu.com wrote: Hi everyone, Matt Bruzek and I have been doing some charm testing on a machine that does not have general access to the internet. So charms that pull from PPAs, github, etc. do not work. We've been able to fatten the charms by doing things like creating a /files directory in the charm itself and putting the package/tarball/jar file in there, and given the networking issues that we might face in production environments that we should start thinking about best practices for having charms with payloads instead of pulling from a network source. Marco has some ideas on how we can generalize this and he will respond to this thread. -- Jorge Castro Canonical Ltd. http://juju.ubuntu.com/ - Automate your Cloud Infrastructure -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Best practices for fat charms
just to be clear the https url thing is solvable with the intelligent proxy thing just a bit more work and client/library support isn't always great. http://wiki.squid-cache.org/Features/HTTPS On Tue, Apr 1, 2014 at 4:11 PM, Kapil Thangavelu kapil.thangav...@canonical.com wrote: If your trying to do this in automated fashion, juju supports proxies, and possibily with intelligent proxy you could do something a bit more automated. else its going to require alot of auditing. you could even skip the additional steps of modifying all the charms have the intelligent proxy work in offline mode with its cache to serve out the files back when deploying in an offline setup. the issue is going to be https url then. On Tue, Apr 1, 2014 at 4:05 PM, Matt Bruzek matthew.bru...@canonical.comwrote: Thanks Jorge, Not sure we want to call them fat charms, maybe enterprise charms. Here is my approach when making a charm work on the enterprise or limited networks. 1) Find out what hook downloads the packages that we are unable to access (wget, curl, or special ppa repositories). The enterprise network will block these requests often resulting in a charm hook failing. 2) Download the necessary packages from system that has access. 3) Upload the packages to the locked down system, copying the packages to a directory on the local charm. 4) Edit the local charm hooks to check for the package in the local directory first and if that does not exist, the charm would continue to download the files (using wget, or curl, or custom ppa). I believe we could provide a charm-tools method that does something like this and we could use this in charms to create enterprise charms that are able to be used on limited network environments. However this creates an interesting problem that I have not figured out a good way to resolve yet (your feedback requested). If the packages exist within the charm the URL will never be used. Some charms allow the user to configure the download URL and sha1sum of the package. Other charms do not allow this level of customization. For charms that have a config option for URL we could change it to also accept a file:// transport and some kind of $CHARM_DIR variable and use http:// or https:// for normal URLs. But what to do with the charms that do not allow the URLs to be configurable, or the charms that use a custom ppa repository? - Matthew Bruzek matthew.bru...@canonical.com On Tue, Apr 1, 2014 at 2:07 PM, Jorge O. Castro jo...@ubuntu.com wrote: Hi everyone, Matt Bruzek and I have been doing some charm testing on a machine that does not have general access to the internet. So charms that pull from PPAs, github, etc. do not work. We've been able to fatten the charms by doing things like creating a /files directory in the charm itself and putting the package/tarball/jar file in there, and given the networking issues that we might face in production environments that we should start thinking about best practices for having charms with payloads instead of pulling from a network source. Marco has some ideas on how we can generalize this and he will respond to this thread. -- Jorge Castro Canonical Ltd. http://juju.ubuntu.com/ - Automate your Cloud Infrastructure -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
