Fwd: Re: Juju Stable Update Policy
I just realized that Curtis didn't reply to the list. Forwarding. Martin - Forwarded message from Curtis Hovey-Canonical cur...@canonical.com - Date: Mon, 28 Jul 2014 12:47:11 -0400 From: Curtis Hovey-Canonical cur...@canonical.com To: Alexis Bruemmer alexis.bruem...@canonical.com, martin.p...@ubuntu.com Subject: Re: Juju Stable Update Policy X-Spam-Status: No, score=-1.9 required=3.4 tests=BAYES_00 autolearn=no version=3.3.2 From: Martin Pitt martin.p...@ubuntu.com Date: Tue, Jul 22, 2014 at 9:05 AM Subject: Re: Juju Stable Update Policy Alexis Bruemmer [2014-07-08 14:05 -0700]: Juju developers and have a specific QA review, and a variety of fresh install and upgrade combinations must be explicitly tested on every cloud that Juju supports (for details see the “Testing a revision” section of the Juju CI Design and Operation document linked below) This really seems to be the crux of regression testing as far as SRUs and existing cloud deployments are concerned. Whenever I try to open the google docs I just get a File unavailable. Sorry, there's a problem with this file. Please reload., so I can't look at the details. Does that mean that newer clients are tested on existing deployments with older servers and agents? Which combinations does that cover? I have updated the sharing policies for the Juju Ci doc, let me know if you still have issues accessing the file. I can see it now, but it still doesn't explain how you test backwards compatibility to existing deployments? Or I can't find it. I can see teh google doc now, but it still crashes on the first mouse click. I think for long-term documentation we don't want to bury that in google docs, but perhaps underneath https://wiki.ubuntu.com/StableReleaseUpdates ? So in summary, I have reasonably good confidence in the currently documented CI, but I need some more convincing about backwards compatibility testing. We don't have automated tests of minor-to-minor-client-to-server yet. We are not scheduled to start work on this feature for a a few weeks. We are added infrastructure to Juju CI to support this and we have done some manual testing to prove how we will automate this. 1. CI will have a store of stable minor jujus: 1.18.4, 1.20.1, 1.21.0... 2. For each new revision of juju, * For each stable minor juju an env will be bootstrapped. * The juju client under test will be run through a series of commands that demonstrate the env can be maintained. * The test will culminate in a destroy environment to show the env can reach its EOL. * An env will be bootstrapped with the new juju * For each stable minor juju a series of commands will test that old client can maintain the new env. * reports.vapour.ws will have a report showing the all combinations pass test suite. 3. Only revisions that pass all tests are considered for release, CI will provide the actual release tarball. 4. CI will gain a mechanism to test a packages built for Ubuntu (ubuntu's packaging rules) that will test the packages pass all the tests the release tarball passed. Demonstrating that juju and the ubuntu packaging are compatible with the versions published in Ubuntu. -- Curtis Hovey Canonical Cloud Development and Operations http://launchpad.net/~sinzui -- technical-board mailing list technical-board@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/technical-board - End forwarded message - -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) -- technical-board mailing list technical-board@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/technical-board
Re: Juju Stable Update Policy
Hello all, Curtis Hovey-Canonical [2014-07-28 12:47 -0400]: We are added infrastructure to Juju CI to support this and we have done some manual testing to prove how we will automate this. 1. CI will have a store of stable minor jujus: 1.18.4, 1.20.1, 1.21.0... 2. For each new revision of juju, * For each stable minor juju an env will be bootstrapped. * The juju client under test will be run through a series of commands that demonstrate the env can be maintained. * The test will culminate in a destroy environment to show the env can reach its EOL. * An env will be bootstrapped with the new juju * For each stable minor juju a series of commands will test that old client can maintain the new env. Perfect, thanks! That's what I was looking for. Provisional MRE granted (after quick discussion in today's TB meeting), I'll update https://wiki.ubuntu.com/StableReleaseUpdates/MicroReleaseExceptions as soon as this mail hits the archive. Thanks, Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) signature.asc Description: Digital signature -- technical-board mailing list technical-board@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/technical-board
Re: Juju Stable Update Policy
From: Martin Pitt martin.p...@ubuntu.com Date: Tue, Jul 22, 2014 at 9:05 AM Subject: Re: Juju Stable Update Policy Alexis Bruemmer [2014-07-08 14:05 -0700]: Juju developers and have a specific QA review, and a variety of fresh install and upgrade combinations must be explicitly tested on every cloud that Juju supports (for details see the “Testing a revision” section of the Juju CI Design and Operation document linked below) This really seems to be the crux of regression testing as far as SRUs and existing cloud deployments are concerned. Whenever I try to open the google docs I just get a File unavailable. Sorry, there's a problem with this file. Please reload., so I can't look at the details. Does that mean that newer clients are tested on existing deployments with older servers and agents? Which combinations does that cover? I have updated the sharing policies for the Juju Ci doc, let me know if you still have issues accessing the file. I can see it now, but it still doesn't explain how you test backwards compatibility to existing deployments? Or I can't find it. I can see teh google doc now, but it still crashes on the first mouse click. I think for long-term documentation we don't want to bury that in google docs, but perhaps underneath https://wiki.ubuntu.com/StableReleaseUpdates ? So in summary, I have reasonably good confidence in the currently documented CI, but I need some more convincing about backwards compatibility testing. We don't have automated tests of minor-to-minor-client-to-server yet. We are not scheduled to start work on this feature for a a few weeks. We are added infrastructure to Juju CI to support this and we have done some manual testing to prove how we will automate this. 1. CI will have a store of stable minor jujus: 1.18.4, 1.20.1, 1.21.0... 2. For each new revision of juju, * For each stable minor juju an env will be bootstrapped. * The juju client under test will be run through a series of commands that demonstrate the env can be maintained. * The test will culminate in a destroy environment to show the env can reach its EOL. * An env will be bootstrapped with the new juju * For each stable minor juju a series of commands will test that old client can maintain the new env. * reports.vapour.ws will have a report showing the all combinations pass test suite. 3. Only revisions that pass all tests are considered for release, CI will provide the actual release tarball. 4. CI will gain a mechanism to test a packages built for Ubuntu (ubuntu's packaging rules) that will test the packages pass all the tests the release tarball passed. Demonstrating that juju and the ubuntu packaging are compatible with the versions published in Ubuntu. -- Curtis Hovey Canonical Cloud Development and Operations http://launchpad.net/~sinzui -- technical-board mailing list technical-board@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/technical-board
Re: Juju Stable Update Policy
Hello Alexis, Alexis Bruemmer [2014-07-08 14:05 -0700]: Juju developers and have a specific QA review, and a variety of fresh install and upgrade combinations must be explicitly tested on every cloud that Juju supports (for details see the “Testing a revision” section of the Juju CI Design and Operation document linked below) This really seems to be the crux of regression testing as far as SRUs and existing cloud deployments are concerned. Whenever I try to open the google docs I just get a File unavailable. Sorry, there's a problem with this file. Please reload., so I can't look at the details. Does that mean that newer clients are tested on existing deployments with older servers and agents? Which combinations does that cover? I have updated the sharing policies for the Juju Ci doc, let me know if you still have issues accessing the file. I can see it now, but it still doesn't explain how you test backwards compatibility to existing deployments? Or I can't find it. I can see teh google doc now, but it still crashes on the first mouse click. I think for long-term documentation we don't want to bury that in google docs, but perhaps underneath https://wiki.ubuntu.com/StableReleaseUpdates ? So in summary, I have reasonably good confidence in the currently documented CI, but I need some more convincing about backwards compatibility testing. Thanks, Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) -- technical-board mailing list technical-board@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/technical-board
Re: Juju Stable Update Policy
Hello Alexis, thanks for writing down the summary, and the initial adjustments after our PMs. Sorry for responding so late, this slipped through the cracks :( In general this seems fine to me, I just have two questions: Alexis Bruemmer [2014-06-26 13:00 -0700]: It is critical that the state server and agents be kept in sync, and both of these sets of machines are created, booted, and managed by Juju. Out of interest, how is that enforced? If you dist-upgrade one VM but not the other, will bad things happen? Or should that be worded differently? Tests are required for all changes, each change must be signed off by two Juju developers and have a specific QA review, and a variety of fresh install and upgrade combinations must be explicitly tested on every cloud that Juju supports (for details see the “Testing a revision” section of the Juju CI Design and Operation document linked below) This really seems to be the crux of regression testing as far as SRUs and existing cloud deployments are concerned. Whenever I try to open the google docs I just get a File unavailable. Sorry, there's a problem with this file. Please reload., so I can't look at the details. Does that mean that newer clients are tested on existing deployments with older servers and agents? Which combinations does that cover? Thanks, Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) -- technical-board mailing list technical-board@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/technical-board
Re: Juju Stable Update Policy
Hi Alexis, Given the lack of response so far, I guess that this is fairly uncontroversial. It's clear to me that the client-server nature of juju warrants an exception to the SRU rules, analogous to the exception already in place for Landscape. I am also satisfied by the general policies in place for Juju stable updates, though as you suggest it would be a good idea for the SRU team to review the specifics in depth with Curtis. Does anyone else on the TB have any feedback here, or is everyone happy to ratify this plan as-is? On Thu, Jun 26, 2014 at 01:00:20PM -0700, Alexis Bruemmer wrote: Hello Ubuntu Technical Board, The Juju Core team requests your review and approval of the Juju Stable Update Policy outlined below: Introduction Juju is a open source service from Canonical for service orchestration. A typical juju install consists of a “cloud environment” which consists of a set of central state servers and any number of managed services and machines. Each managed machine in the environment runs “agents” which talk to the central state server(s) for that environment. The package in ubuntu is the “juju client” and is what is used to spin up cloud servers, bootstrap a set of state servers, and then communicate via a stable API with one or more state servers about what workloads to deploy and how to configure them. The juju client also allows for SSH access (tunneled through the state servers) to managed machines, as well as the ability to run actions on sets of machines in the environment. The juju client can also run a simulated cloud environment on containers directly on the user’s host box (via the ‘juju-local’ package). Plans for this cycle include a centralized multi-tenant state servers, which will serve multiple environments, and be hosted by Canonical, or our clients. They also include juju deployed workloads running on a wide variety of “Host OS’s” including Ubuntu of course, but also Windows and other versions of Linux/Unix. As might be expected from this kind of design, there are well defined API’s that manage the interactions between the state servers and the agents that run on individual managed hosts, as well as the juju clients that might be installed on individual user’s machines. It is critical that the state server and agents be kept in sync, and both of these sets of machines are created, booted, and managed by Juju. The code for the state servers and host machine agents is pulled down by Juju at install time, and could be running on a completely different architecture, and even a completely different OS than the juju client (which is supported on Windows and Mac OS as well as a variety of linux/unix platforms). These installs are managed by Juju itself (since it is a service orchestration and configuration management tool) and do not use the ubuntu packages -- instead they pull the appropriate juju binary from the same place they get OS images (simple streams). It is desirable to update both clients and servers to provide newer features, but many features of newer servers will be unavailable with older clients. And it is possible that security updates on the server will require compatible client updates. At present, users of the Juju service are typically instructed to use a PPA to work around this. However, since the `juju` package is part of Ubuntu, the juju team needs the ability to push updates into the LTS, otherwise it is unlikely the version we ship will remain actually useful to juju users. Rationale Canonical produces another system management tool, Landscape, which received a similar exception in 2009. Like Landscape, we believe Juju’s requirements as a system management tool make it a candidate for a stable release update. In some ways the juju client is subject to less possibility for destabilizing the OS than Landscape. As of today it consists of a single binary (though discussions with the Ubuntu team are in progress to break that down) which neither has dependencies on any third party libraries, nor is depended on by any other Ubuntu packages. Prior to making this request, the Juju Core update process has been updated to include an exceptionally strict QA process which closely matches that required of the Landscape client. Tests are required for all changes, each change must be signed off by two Juju developers and have a specific QA review, and a variety of fresh install and upgrade combinations must be explicitly tested on every cloud that Juju supports (for details see the “Testing a revision” section of the Juju CI Design and Operation document linked below) . Furthermore we are working to create a full set of functional tests of juju deployed workloads that will be used to gate any stable release. The Juju team requests that the ubuntu-sru team review these processes in detail with Curtis (our release and QA manager) and the juju team will update these test
