Re: An Open Question: Charm Dependency Management
On Wed, Jan 21, 2015 at 8:24 AM, Michael Nelson < michael.nel...@canonical.com> wrote: > > For eg., if it was a python charm that needed some python packages: > > > .virtualenv: > virtualenv .virtualenv > .virtualenv/bin/pip install -r test_requirements.txt > > test: .virtualenv > ./virtualenv/bin/python ./unit-tests > My python charm makefiles looks like this too, but one minor improvement is to suppress output from pip with '-q' which makes running dependant targets like 'test' and 'lint' much less noisy, e.g. .virtualenv: @virtualenv .virtualenv @.virtualenv/bin/pip install -q -r requirements.txt --upgrade -- Kit Randel Canonical - Ubuntu Engineering - Continuous Integration Team -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: An Open Question: Charm Dependency Management
We are striving to test charms against all substrates and architectures juju supports; and are nearing completion for that goal. Manual provider is not currently tested AFAIK but will likely be in the future. On Tue Jan 20 2015 at 7:41:27 PM Andrew Wilkins < andrew.wilk...@canonical.com> wrote: > On Wed, Jan 21, 2015 at 1:36 AM, Charles Butler < > charles.but...@canonical.com> wrote: > >> Greetings, >> >> If you work on charms in any capacity: this affects you, and I would love >> to have your feedback. >> >> While working the review queue I've encountered a few charm merges that >> are failing our testing infrastructure due to missing dependencies. This >> also has implications that reach beyond our testing infrastructure: Anyone >> that is submitting a new charm, Patches being accepted to existing charms, >> and even our documentation efforts over on the Charm Authorship Docs. >> > > On this topic, does the automated testing cover testing charms with the > manual provider? There have been charms that failed (and were fixed) > because they did not explicitly install dependencies. Those charms were > using packages that are automatically installed as a result of cloud-init > being on the machine; since the manual provider does not use cloud-init, > those packages don't get automatically installed and the charms fail. IIRC, > python-yaml fits into this category. > > >> There seems to be a bit of confusion about what the recommended process >> is to ensure all our dependencies are encapsulated in the charm. >> >> Having spoken with various members of the development community, I feel >> like our dependency encapsulation process for charms is still very much a >> grey area with several different ideologies on how to manage them, thus far >> I've seen: >> >>- A Virtualenv per project to manage python dependencies >>- make targets that sudo install packages on the host system >>- Zero Dependency management >> >> This is indeed a difficult topic to approach and digest as we're >> supporting basically everything out there. Not everyone uses the same >> tool-chain to accomplish the goal of dependency isolation - and several >> different Config Management tools have a different approach to this that >> assume it is installed on the host. this leaves a broken experience for: >> >>- new charm authors >>- CI >>- Anyone that comes along and runs the test targets or bundletester >> >> If we're going to ask our community to contribute to charms, is it fair >> to make them run down dependencies that may or may not exist on their host? >> It seems like we can do a better job of highlighting this, and providing a >> quick start style development target to install these pre-deps which would >> satisfy CI and Development. With this being the proposal, I follow it with >> some questions: >> >>- How have *we* solved this problem in other areas of our ecosystem? >>- How have other platforms solved this problem? >>- Can we emulate / improve on that pattern? >>- If a package management solution exists for a technology (eg: >>virtualenv for python, bundler for ruby, npm for node, berkshelf for chef) >>- can we adopt these and get started by templating in the dependency >>management into the charm generator template? >> >> >> I'm hoping this email is seen more as a conversation starter vs me being >> pedantic - I'm more concerned with getting the right set of information to >> our users/community than I am in solving some meta problem of packaging >> charms and their Development Environments. >> >> -- >> All the best, >> >> Charles Butler - Juju Charmer >> Come see the future of datacenter orchestration: http://jujucharms.com >> >> -- >> Juju mailing list >> Juju@lists.ubuntu.com >> Modify settings or unsubscribe at: >> https://lists.ubuntu.com/mailman/listinfo/juju >> >> -- > Juju mailing list > Juju@lists.ubuntu.com > Modify settings or unsubscribe at: https://lists.ubuntu.com/ > mailman/listinfo/juju > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: An Open Question: Charm Dependency Management
On Wed, Jan 21, 2015 at 1:36 AM, Charles Butler < charles.but...@canonical.com> wrote: > Greetings, > > If you work on charms in any capacity: this affects you, and I would love > to have your feedback. > > While working the review queue I've encountered a few charm merges that > are failing our testing infrastructure due to missing dependencies. This > also has implications that reach beyond our testing infrastructure: Anyone > that is submitting a new charm, Patches being accepted to existing charms, > and even our documentation efforts over on the Charm Authorship Docs. > On this topic, does the automated testing cover testing charms with the manual provider? There have been charms that failed (and were fixed) because they did not explicitly install dependencies. Those charms were using packages that are automatically installed as a result of cloud-init being on the machine; since the manual provider does not use cloud-init, those packages don't get automatically installed and the charms fail. IIRC, python-yaml fits into this category. > There seems to be a bit of confusion about what the recommended process > is to ensure all our dependencies are encapsulated in the charm. > > Having spoken with various members of the development community, I feel > like our dependency encapsulation process for charms is still very much a > grey area with several different ideologies on how to manage them, thus far > I've seen: > >- A Virtualenv per project to manage python dependencies >- make targets that sudo install packages on the host system >- Zero Dependency management > > This is indeed a difficult topic to approach and digest as we're > supporting basically everything out there. Not everyone uses the same > tool-chain to accomplish the goal of dependency isolation - and several > different Config Management tools have a different approach to this that > assume it is installed on the host. this leaves a broken experience for: > >- new charm authors >- CI >- Anyone that comes along and runs the test targets or bundletester > > If we're going to ask our community to contribute to charms, is it fair to > make them run down dependencies that may or may not exist on their host? It > seems like we can do a better job of highlighting this, and providing a > quick start style development target to install these pre-deps which would > satisfy CI and Development. With this being the proposal, I follow it with > some questions: > >- How have *we* solved this problem in other areas of our ecosystem? >- How have other platforms solved this problem? >- Can we emulate / improve on that pattern? >- If a package management solution exists for a technology (eg: >virtualenv for python, bundler for ruby, npm for node, berkshelf for chef) >- can we adopt these and get started by templating in the dependency >management into the charm generator template? > > > I'm hoping this email is seen more as a conversation starter vs me being > pedantic - I'm more concerned with getting the right set of information to > our users/community than I am in solving some meta problem of packaging > charms and their Development Environments. > > -- > All the best, > > Charles Butler - Juju Charmer > Come see the future of datacenter orchestration: http://jujucharms.com > > -- > Juju mailing list > Juju@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju > > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: An Open Question: Charm Dependency Management
On Wed, Jan 21, 2015 at 5:17 AM, David Britton wrote: > On Tue, Jan 20, 2015 at 05:58:24PM +, Marco Ceppi wrote: >> I don't see how a Makefile in a charm doesn't resolve this issue. > > +1 on some standard published Makefile targets. We already have some > that are highly recommended: > > - test > - lint +1 (and -1000 on any Makefile targets which run as sudo) > > Maybe: > > - test-depends or depends # to install/update dependencies needed for > testing It doesn't hurt to have an extra target like test-depends, but it should be enough to just have the test target depend on some other target that's required for whatever technology the author is using. For eg., if it was a python charm that needed some python packages: .virtualenv: virtualenv .virtualenv .virtualenv/bin/pip install -r test_requirements.txt test: .virtualenv ./virtualenv/bin/python ./unit-tests Yep, you could make that DRYer, but it might be less readable. -Michael > > Are there others that are needed/missing or that I forgot we already > have as standard? > > -- > David Britton > > -- > Juju mailing list > Juju@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: An Open Question: Charm Dependency Management
Lots of good feedback here with regard to how we want to manage it. I'm personally +1 on having a Makefile in my charm that handles these things, but was unsure if this was our defacto recommended path to completion. Thanks for such an active and rapid response on the thread. On Tue, Jan 20, 2015 at 1:20 PM, Marco Ceppi wrote: > Well there are two notions of testing, unit_test and functional_test one > is largely more expensive than the other. Outside of that test-depends is a > good one. Whatever it is we should identify those so we can make sure > bundletester is updated to sniff these targets out (if this is the route we > chose). > > > On Tue Jan 20 2015 at 1:18:05 PM David Britton < > david.brit...@canonical.com> wrote: > >> On Tue, Jan 20, 2015 at 05:58:24PM +, Marco Ceppi wrote: >> > I don't see how a Makefile in a charm doesn't resolve this issue. >> >> +1 on some standard published Makefile targets. We already have some >> that are highly recommended: >> >> - test >> - lint >> >> Maybe: >> >> - test-depends or depends # to install/update dependencies needed for >> testing >> >> Are there others that are needed/missing or that I forgot we already >> have as standard? >> >> -- >> David Britton >> > -- All the best, Charles Butler - Juju Charmer Come see the future of datacenter orchestration: http://jujucharms.com -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
[Review Queue] wildfly charms
I spent some time in the review queue today on the wildfly-ha-master and wildfly-ha-slave charms. https://bugs.launchpad.net/charms/+bug/1399239 https://bugs.launchpad.net/charms/+bug/1399228 Both are now in the recommended section of the charm store. Many thanks to *Saurabh Kumar* from Techblue for working with the review comments and landing the charms! - Matt Bruzek -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: An Open Question: Charm Dependency Management
Well there are two notions of testing, unit_test and functional_test one is largely more expensive than the other. Outside of that test-depends is a good one. Whatever it is we should identify those so we can make sure bundletester is updated to sniff these targets out (if this is the route we chose). On Tue Jan 20 2015 at 1:18:05 PM David Britton wrote: > On Tue, Jan 20, 2015 at 05:58:24PM +, Marco Ceppi wrote: > > I don't see how a Makefile in a charm doesn't resolve this issue. > > +1 on some standard published Makefile targets. We already have some > that are highly recommended: > > - test > - lint > > Maybe: > > - test-depends or depends # to install/update dependencies needed for > testing > > Are there others that are needed/missing or that I forgot we already > have as standard? > > -- > David Britton > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: An Open Question: Charm Dependency Management
On Tue, Jan 20, 2015 at 05:58:24PM +, Marco Ceppi wrote: > I don't see how a Makefile in a charm doesn't resolve this issue. +1 on some standard published Makefile targets. We already have some that are highly recommended: - test - lint Maybe: - test-depends or depends # to install/update dependencies needed for testing Are there others that are needed/missing or that I forgot we already have as standard? -- David Britton -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
juju devel 1.21-rc1 is released
juju-core 1.21-rc1 A new development release of Juju, juju-core 1.21-rc1, is now available. This release replaces 1.21-beta5. Getting Juju juju-core 1.21-rc1 is available for utopic and backported to earlier series in the following PPA: https://launchpad.net/~juju/+archive/devel The devel packages in this archive use the devel simple-streams. You must configure the 'agent-stream' option in your environments.yaml to use the matching juju agents. agent-stream: devel Upgrading from stable releases to development releases is not supported. You can upgrade test environments to development releases to test new features and fixes, but it is not advised to upgrade production environments to 1.21-rc1. Notable Changes This releases addresses stability and performance issues. Resolved issues * Error upgrade in progress - juju functionality is limited Lp 1411502 Finally We encourage everyone to subscribe the mailing list at juju-...@lists.canonical.com, or join us on #juju-dev on freenode. -- Curtis Hovey Canonical Cloud Development and Operations http://launchpad.net/~sinzui -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: An Open Question: Charm Dependency Management
A Makefile that has a target to install dependencies suffices, but I think suggested conventions are still helpful. For example, in my case I prefer python virtualenvs over system packages. Once you establish some conventions (perhaps even just documentation conventions), A charmer can document the approach taken so that devs will be aware and can take measures. e.g. if I want to make a MR to a charm that needs system packages, I will know to do my development and testing inside of a container. On Tue, Jan 20, 2015 at 11:58 AM, Marco Ceppi wrote: > I don't see how a Makefile in a charm doesn't resolve this issue. As long > as we define what targets a user should create in the Makefile, the > Makefile can then do everything required: create a virtualenv and install > deps, install ruby and execute bundler, npm for node, etc. Since charms are > so varies in how they're written and what language they're implemented in > this seems to make the most sense. > > Marco > > On Tue Jan 20 2015 at 12:37:00 PM Charles Butler < > charles.but...@canonical.com> wrote: > >> Greetings, >> >> If you work on charms in any capacity: this affects you, and I would love >> to have your feedback. >> >> While working the review queue I've encountered a few charm merges that >> are failing our testing infrastructure due to missing dependencies. This >> also has implications that reach beyond our testing infrastructure: Anyone >> that is submitting a new charm, Patches being accepted to existing charms, >> and even our documentation efforts over on the Charm Authorship Docs. >> >> There seems to be a bit of confusion about what the recommended process >> is to ensure all our dependencies are encapsulated in the charm. >> >> Having spoken with various members of the development community, I feel >> like our dependency encapsulation process for charms is still very much a >> grey area with several different ideologies on how to manage them, thus far >> I've seen: >> >>- A Virtualenv per project to manage python dependencies >>- make targets that sudo install packages on the host system >>- Zero Dependency management >> >> This is indeed a difficult topic to approach and digest as we're >> supporting basically everything out there. Not everyone uses the same >> tool-chain to accomplish the goal of dependency isolation - and several >> different Config Management tools have a different approach to this that >> assume it is installed on the host. this leaves a broken experience for: >> >>- new charm authors >>- CI >>- Anyone that comes along and runs the test targets or bundletester >> >> If we're going to ask our community to contribute to charms, is it fair >> to make them run down dependencies that may or may not exist on their host? >> It seems like we can do a better job of highlighting this, and providing a >> quick start style development target to install these pre-deps which would >> satisfy CI and Development. With this being the proposal, I follow it with >> some questions: >> >>- How have *we* solved this problem in other areas of our ecosystem? >>- How have other platforms solved this problem? >>- Can we emulate / improve on that pattern? >>- If a package management solution exists for a technology (eg: >>virtualenv for python, bundler for ruby, npm for node, berkshelf for chef) >>- can we adopt these and get started by templating in the dependency >>management into the charm generator template? >> >> >> I'm hoping this email is seen more as a conversation starter vs me being >> pedantic - I'm more concerned with getting the right set of information to >> our users/community than I am in solving some meta problem of packaging >> charms and their Development Environments. >> >> -- >> All the best, >> >> Charles Butler - Juju Charmer >> Come see the future of datacenter orchestration: http://jujucharms.com >> -- >> Juju mailing list >> Juju@lists.ubuntu.com >> Modify settings or unsubscribe at: https://lists.ubuntu.com/ >> mailman/listinfo/juju >> > > -- > Juju mailing list > Juju@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju > > -- she...@pobox.com -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: An Open Question: Charm Dependency Management
I don't see how a Makefile in a charm doesn't resolve this issue. As long as we define what targets a user should create in the Makefile, the Makefile can then do everything required: create a virtualenv and install deps, install ruby and execute bundler, npm for node, etc. Since charms are so varies in how they're written and what language they're implemented in this seems to make the most sense. Marco On Tue Jan 20 2015 at 12:37:00 PM Charles Butler < charles.but...@canonical.com> wrote: > Greetings, > > If you work on charms in any capacity: this affects you, and I would love > to have your feedback. > > While working the review queue I've encountered a few charm merges that > are failing our testing infrastructure due to missing dependencies. This > also has implications that reach beyond our testing infrastructure: Anyone > that is submitting a new charm, Patches being accepted to existing charms, > and even our documentation efforts over on the Charm Authorship Docs. > > There seems to be a bit of confusion about what the recommended process > is to ensure all our dependencies are encapsulated in the charm. > > Having spoken with various members of the development community, I feel > like our dependency encapsulation process for charms is still very much a > grey area with several different ideologies on how to manage them, thus far > I've seen: > >- A Virtualenv per project to manage python dependencies >- make targets that sudo install packages on the host system >- Zero Dependency management > > This is indeed a difficult topic to approach and digest as we're > supporting basically everything out there. Not everyone uses the same > tool-chain to accomplish the goal of dependency isolation - and several > different Config Management tools have a different approach to this that > assume it is installed on the host. this leaves a broken experience for: > >- new charm authors >- CI >- Anyone that comes along and runs the test targets or bundletester > > If we're going to ask our community to contribute to charms, is it fair to > make them run down dependencies that may or may not exist on their host? It > seems like we can do a better job of highlighting this, and providing a > quick start style development target to install these pre-deps which would > satisfy CI and Development. With this being the proposal, I follow it with > some questions: > >- How have *we* solved this problem in other areas of our ecosystem? >- How have other platforms solved this problem? >- Can we emulate / improve on that pattern? >- If a package management solution exists for a technology (eg: >virtualenv for python, bundler for ruby, npm for node, berkshelf for chef) >- can we adopt these and get started by templating in the dependency >management into the charm generator template? > > > I'm hoping this email is seen more as a conversation starter vs me being > pedantic - I'm more concerned with getting the right set of information to > our users/community than I am in solving some meta problem of packaging > charms and their Development Environments. > > -- > All the best, > > Charles Butler - Juju Charmer > Come see the future of datacenter orchestration: http://jujucharms.com > -- > Juju mailing list > Juju@lists.ubuntu.com > Modify settings or unsubscribe at: https://lists.ubuntu.com/ > mailman/listinfo/juju > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
[Review Queue]
Reviewed the python rewrite of the jenkins charm. Generally looks good but needs some minor fixes for testing and an intermittent race condition. https://code.launchpad.net/~hopem/charms/trusty/jenkins/python-redux/+merge/245769 -w -- --- D. Whit Morriss Developer, Juju Ecosystem Canonical USA -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
An Open Question: Charm Dependency Management
Greetings, If you work on charms in any capacity: this affects you, and I would love to have your feedback. While working the review queue I've encountered a few charm merges that are failing our testing infrastructure due to missing dependencies. This also has implications that reach beyond our testing infrastructure: Anyone that is submitting a new charm, Patches being accepted to existing charms, and even our documentation efforts over on the Charm Authorship Docs. There seems to be a bit of confusion about what the recommended process is to ensure all our dependencies are encapsulated in the charm. Having spoken with various members of the development community, I feel like our dependency encapsulation process for charms is still very much a grey area with several different ideologies on how to manage them, thus far I've seen: - A Virtualenv per project to manage python dependencies - make targets that sudo install packages on the host system - Zero Dependency management This is indeed a difficult topic to approach and digest as we're supporting basically everything out there. Not everyone uses the same tool-chain to accomplish the goal of dependency isolation - and several different Config Management tools have a different approach to this that assume it is installed on the host. this leaves a broken experience for: - new charm authors - CI - Anyone that comes along and runs the test targets or bundletester If we're going to ask our community to contribute to charms, is it fair to make them run down dependencies that may or may not exist on their host? It seems like we can do a better job of highlighting this, and providing a quick start style development target to install these pre-deps which would satisfy CI and Development. With this being the proposal, I follow it with some questions: - How have *we* solved this problem in other areas of our ecosystem? - How have other platforms solved this problem? - Can we emulate / improve on that pattern? - If a package management solution exists for a technology (eg: virtualenv for python, bundler for ruby, npm for node, berkshelf for chef) - can we adopt these and get started by templating in the dependency management into the charm generator template? I'm hoping this email is seen more as a conversation starter vs me being pedantic - I'm more concerned with getting the right set of information to our users/community than I am in solving some meta problem of packaging charms and their Development Environments. -- All the best, Charles Butler - Juju Charmer Come see the future of datacenter orchestration: http://jujucharms.com -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
[Review Queue] - BigData, squid-reverse-proxy
hdp-storm-zookeeper bundle: *pending* - The bundle did not pass proof as is - but given the age of the branch it looks like bitrot from recent changes in how the big data charms are organized. I submitted a patch inline with the review to fix up and pass - once submitted will promote to the store post haste. https://bugs.launchpad.net/charms/+bug/1373006 squid-reverse-proxy: *merged* - Add's an NRPE fixup for changing ports and monitoring with NRPE. The CI test failure and dependency management is being tracked in a different issue: https://code.launchpad.net/~jacekn/charms/trusty/squid-reverseproxy/squid-reverseproxy-nrpe-fix/+merge/245312 https://bugs.launchpad.net/charms/+source/squid-reverseproxy/+bug/1401323 -- All the best, Charles Butler - Juju Charmer Come see the future of datacenter orchestration: http://jujucharms.com -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: bind mount support for local provider
On Sun, Jan 18, 2015 at 8:01 PM, Andrew Wilkins < andrew.wilk...@canonical.com> wrote: > On Sat, Jan 17, 2015 at 11:47 PM, Corey Bryant > wrote: > >> Excellent, thanks Ian. Do you have an estimate on when this will be >> available? >> > > Hi Corey, > > Not yet; we are working towards getting a single provider fully functional > before delving too deeply into others. I'll ping you when we've got a > better idea. > > Ok, thanks Andrew. > Cheers, > Andrew > > >> On Fri, Jan 16, 2015 at 10:42 PM, Ian Booth >> wrote: >> >>> Hi Cory >>> >>> This is part of what we are planning to deliver for this cycle as part >>> of the >>> storage work. We also plan on being able to provide the container with >>> access to >>> block devices eg loopback, either in container's filesystem or on the >>> host machine. >>> >>> >>> On 17/01/15 02:11, Corey Bryant wrote: >>> > Hi all, >>> > >>> > Do there happen to be any plans for juju bind mount support for the >>> local >>> > provider? >>> > >>> > For example: juju deploy mysql --bind "/shared/mysql /shared" >>> > >>> > which would bind mount the host /shared/mysql directory to /shared in >>> the >>> > deployed container. >>> > >>> > >>> > >>> >>> >> >> >> -- >> Regards, >> Corey >> >> -- >> Juju mailing list >> Juju@lists.ubuntu.com >> Modify settings or unsubscribe at: >> https://lists.ubuntu.com/mailman/listinfo/juju >> >> > -- Regards, Corey -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju