On Sun, May 11, 2014 at 11:08:31PM +0100, Colin Watson wrote: > [This is a Foundations/CI project; I haven't posted a status update on > it before, but a few people have asked about it in private mail, so it > might be helpful to have a public update to refer to.] > > We've been working on reorganising the infrastructure for the live > filesystem parts of our image builds. At the moment there's a single > dedicated builder for each architecture that the central image build > system (lp:ubuntu-cdimage, running on nusakan.canonical.com) can ask to > produce various live filesystems; ubuntu-cdimage then fetches the > results from it, performs some final polishing steps (for ISO images, > this involves building a pool of .debs among other things, so is a > fairly complex procedure requiring debian-cd and a local mirror; for > phone images it's relatively simple), and publishes the output on > cdimage.ubuntu.com. In the case of phone images, system-image then > steps in to turn these into the system-image format and compute deltas > etc. > > This is serviceable for what we're doing with it right now, but there > are a few motivations for improving things which have been on the > wishlist for some time: > > * It's not at all horizontally scalable. We notice this most around > milestone releases when we try to do lots of image builds at once, > and we can only do one at a time per architecture. This is basically > why respin cycles for milestones take eight hours or whatever > ridiculous amount it is. > > * We can't sensibly extend this to experimental builds. > > * To a first approximation, ubuntu-cdimage has only been deployed once > (although I rewrote it a while back, and one of the goals of the > rewrite was to make deployment easier). > > We discussed this at the release engineering sprint last year, and the > general consensus was that we should move live filesystem building into > Launchpad. This has a number of advantages: both build farms are > normally well within capacity except for occasional bursts that tend to > be at complementary times; we would be able to run many more image > builds at once and thus drastically reduce the time required for a full > respin; it would be much easier to monitor builds in progress; builds > will be cancellable without sysadmin intervention; and we'll get one or > two extra builders on most architectures by merging the build farms, at > least where the hardware in use is new enough to make that worthwhile. > > I've been working on this on and off for a while, and essentially > full-time since 14.04 released. The current status is as follows: > > * The launchpad-buildd package in production already has most of the > code required to build live filesystems; this was done towards the > end of last year. More recent testing uncovered a few small but > critical problems with this code with various real-world image types > (buggy handling of symlinks and of duplicate files in build output); > one fix is in trunk and the other is awaiting review. > > * The bulk of the required changes to Launchpad itself have been > through one review pass and are awaiting re-review. The required > extensions to the web UI are still only on my local disk, as I have a > few more tests to write first, but I expect to finish that in the > first half of the coming week. > > * I have a local installation that is capable of building full live > filesystem images, tested with both Ubuntu Desktop and Ubuntu Touch > (i386). > > * I have a local ubuntu-cdimage branch that can request live filesystem > builds from a local Launchpad instance and wait for them to complete. > It cannot yet download the results. > > There's still a fair bit to do, but my aim is to have this landed and in > production in time for the core sprint in Malta in the last week of May. > > The engineers developing the CI Airline need to be able to build live > filesystems against PPAs as a component of their work, and so I've been > in discussions with them to make sure that we will be able to use the > same software for this, and where possible the same infrastructure, > rather than duplicating work. In anticipation of this requirement, the > database modelling for live filesystems in Launchpad includes the > facility to associate a build with a PPA, although the pieces required > to pass that through to the sources.list used to construct the live > filesystem aren't quite in place yet. The CI Airline will need its own > cut-down ubuntu-cdimage and system-image instances, and I'll be working > with Paul Larson early this week to get the ubuntu-cdimage part of that > up and running; phone image builds are simple enough from this point of > view that I don't expect that to be difficult. Although I can't speak > for system-image, I think we are well on course to have all the rest of > this either done before Malta or easy to finish off there.
I'm mostly off until Malta, only around this Friday and Monday-Tuesday of next week (16, 19 and 20th of May) so if you have cdimage setup by then, it's possible I can help setting things up for system-image builds, if you need more time, then it's just a matter of sitting together with Paul in Malta and getting the remaining bits sorted out (I expect we'll need some minor tweaks to the code but nothing too scary). > > Cheers, > > -- > Colin Watson [[email protected]] > > -- > Mailing list: https://launchpad.net/~ubuntu-phone > Post to : [email protected] > Unsubscribe : https://launchpad.net/~ubuntu-phone > More help : https://help.launchpad.net/ListHelp -- Stéphane Graber Ubuntu developer http://www.ubuntu.com
signature.asc
Description: Digital signature
-- Mailing list: https://launchpad.net/~ubuntu-phone Post to : [email protected] Unsubscribe : https://launchpad.net/~ubuntu-phone More help : https://help.launchpad.net/ListHelp
