Re: [Launchpad-users] Launchpad builder VMs upgraded to bionic
On Tue, Sep 8, 2020 at 1:48 PM Colin Watson wrote: > The simplest and IMO best way to do this would be to SRU the systemd > change that bumped it to 64M [1] to bionic; we'd then pick that up in > the natural course of events by way of new cloud image builds. Has that > been considered? It feels as though what's happened here is simply that > the Launchpad build farm upgrade has surfaced a bug in bionic, so the > most appropriate thing to do would be to fix it in bionic. > Thanks Colin, it is a good idea! Although, I've seen uploads from my teammates take like 2-3 months to get merged/released on systemd package, due to security fixes on such package. It might take a while to get the fix organically if we follow such approach... > Failing that, can somebody advise on whether there's an appropriate way > to configure this in an image without having to maintain a fork of > systemd? The workflow here is that we consume Ubuntu cloud images and > make a few small changes to them, mostly things like installing > launchpad-buildd, before publishing them to Glance for use when starting > new builder VMs. > I think the cloud image folks can chime in with good ideas, but I'd consider cloud-init - likely it's possible to configure it in a way to apply this change on boot time. Another idea: would it be possible to change launchpad-buildd to accomplish the ulimit change ? Cheers, Guilherme ___ Mailing list: https://launchpad.net/~launchpad-users Post to : launchpad-users@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-users More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-users] Launchpad builder VMs upgraded to bionic
On Tue, Aug 4, 2020 at 11:50 AM Colin Watson wrote: > > The VMs in Launchpad's build farm have been on Ubuntu 16.04 (xenial) for > some time. We're intentionally fairly conservative about upgrading the > base VMs, but it's about time to have something newer, so we've just > upgraded them to Ubuntu 18.04 (bionic). The actual builds still run in > chroots or LXD containers of the appropriate Ubuntu series just as > before, so most builds shouldn't notice any difference, but the kernel > version is now 4.15 rather than 4.4. (As I type this, some VMs are > still on xenial, but they'll be replaced with bionic once they're reset > at the end of their next build.) Hi Colin et.al., first of all thanks for the builder update and heads-up! We've noticed a failure in building cryptsetup from source, reported in LP #1891473 [0]. The reason for such failure is detailed in the LP, but a summary is: - cryptsetup might make use of mlock() syscall and so, restrict its memory usage by the memlock limit (ulimit -l), if it succeeds the mlock() call; - in Xenial, the memlock limit was extremely low, so mlock() failed and the cryptsetup test hereby failing (luks2-metadata validation) succeeded, since cryptsetup continues with no locked memory; -in Bionic, such limit was bumped to 16M [1], and cryptsetup succeeds in the memory locking procedure, but...the limit is low enough in order the luks2-metadata-validation-4M exceeds that and fails - worth to notice that chroot/containers inherit those limits from host; - in Focal such a limit was quite increased (to 64M), so the tests do not fail anymore. In my understanding, we have 2 ways of resolving that: (a) We could bump the memlock limit in PPA builders to 64M (to match real-life cases of Focal, specially useful for Focal packages being built); (b) Alternatively, we could *reduce* the memlock limit in the cryptsetup package (we have enough privilege to decrease the limit in the container), so it mimics the old behavior of Xenial and so cryptsetup doesn't lock memory anymore, and its tests gladly succeed. I've managed to have a build succeeding in my PPA using this approach. I vote for approach (a), since by doing (b) we're "masking off" the mlock() in the cryptsetup packages, which in Bionic+ versions will be successful in real-life scenario (due to the larger memlock limits), but wouldn't be tested in build time. Opinions are much appreciated, I currently have a fix stuck due to the FTBFS in cryptsetup, so I'd like to move-on with that in order to move-on with my upload. Cheers, Guilherme [0] https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/1891473 [1] https://github.com/systemd/systemd/commit/fb3ae275cb ___ Mailing list: https://launchpad.net/~launchpad-users Post to : launchpad-users@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-users More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-users] Launchpad builder VMs upgraded to bionic
On Tue, Sep 08, 2020 at 12:57:28PM -0300, Guilherme Piccoli wrote: > Hi Colin et.al., first of all thanks for the builder update and > heads-up! We've noticed a failure in building cryptsetup from source, > reported in LP #1891473 [0]. The reason for such failure is detailed > in the LP, but a summary is: > - cryptsetup might make use of mlock() syscall and so, restrict its > memory usage by the memlock limit (ulimit -l), if it succeeds the > mlock() call; > - in Xenial, the memlock limit was extremely low, so mlock() failed > and the cryptsetup test hereby failing (luks2-metadata validation) > succeeded, since cryptsetup continues with no locked memory; > -in Bionic, such limit was bumped to 16M [1], and cryptsetup succeeds > in the memory locking procedure, but...the limit is low enough in > order the luks2-metadata-validation-4M exceeds that and fails - worth > to notice that chroot/containers inherit those limits from host; > - in Focal such a limit was quite increased (to 64M), so the tests do > not fail anymore. > > In my understanding, we have 2 ways of resolving that: > > (a) We could bump the memlock limit in PPA builders to 64M (to match > real-life cases of Focal, specially useful for Focal packages being > built); The simplest and IMO best way to do this would be to SRU the systemd change that bumped it to 64M [1] to bionic; we'd then pick that up in the natural course of events by way of new cloud image builds. Has that been considered? It feels as though what's happened here is simply that the Launchpad build farm upgrade has surfaced a bug in bionic, so the most appropriate thing to do would be to fix it in bionic. Failing that, can somebody advise on whether there's an appropriate way to configure this in an image without having to maintain a fork of systemd? The workflow here is that we consume Ubuntu cloud images and make a few small changes to them, mostly things like installing launchpad-buildd, before publishing them to Glance for use when starting new builder VMs. [1] https://github.com/systemd/systemd/commit/91cfdd8d29 -- Colin Watson (he/him) [cjwat...@canonical.com] ___ Mailing list: https://launchpad.net/~launchpad-users Post to : launchpad-users@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-users More help : https://help.launchpad.net/ListHelp
