Re: [Launchpad-users] Launchpad builder VMs upgraded to bionic
On 9/9/20 5:31 PM, Colin Watson wrote: > On Tue, Sep 08, 2020 at 02:51:45PM -0300, Guilherme Piccoli wrote: >> On Tue, Sep 8, 2020 at 1:48 PM Colin Watson wrote: >>> Failing that, can somebody advise on whether there's an appropriate way >>> to configure this in an image without having to maintain a fork of >>> systemd? The workflow here is that we consume Ubuntu cloud images and >>> make a few small changes to them, mostly things like installing >>> launchpad-buildd, before publishing them to Glance for use when starting >>> new builder VMs. >> >> I think the cloud image folks can chime in with good ideas, but I'd >> consider cloud-init - likely it's possible to configure it in a way to >> apply this change on boot time. Another idea: would it be possible to >> change launchpad-buildd to accomplish the ulimit change ? > > Both of these are likely to be harder than just changing the image to > add the appropriate bit of configuration, IMO. cloud-init makes sense > when you're using unmodified cloud images, but we already have a > pipeline for modifying images for our use; > https://bazaar.launchpad.net/~canonical-sysadmins/canonical-is-charms/launchpad-buildd-image-modifier/view/head:/files/scripts/setup-ppa-buildd > is the key bit here. (But I'd still need to know what exactly we need > to set there.) > > I'd still like it to be fixed in bionic's systemd, even if we need to > work around it in the short term. just seen during a groovy test rebuild: https://bugs.launchpad.net/launchpad/+bug/1897461 ___ Mailing list: https://launchpad.net/~launchpad-users Post to : launchpad-users@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-users More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-users] Launchpad builder VMs upgraded to bionic
On Wed, Sep 16, 2020 at 12:02:48PM +0100, Dimitri John Ledkov wrote: > Are we using PAM today already? Cause on riscv64 builder I started to see > > E: PAM error: Authentication token is no longer valid; new one required > Chroot setup failed > E: Error creating chroot session: skipping openssl schroot may happen to use PAM, but it's not something we rely on as part of its interface - from our perspective that's an implementation detail. -- Colin Watson (he/him) [cjwat...@canonical.com] ___ Mailing list: https://launchpad.net/~launchpad-users Post to : launchpad-users@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-users More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-users] Launchpad builder VMs upgraded to bionic
On Wed, 16 Sep 2020 at 11:35, Colin Watson wrote: > > On Wed, Sep 16, 2020 at 10:22:05AM +0100, Dimitri John Ledkov wrote: > > chroot builders do call ~= `sudo chroot` at some point. > > The main work is done by sbuild, which uses schroot, not "sudo chroot". > > (We used to use sbuild's sudo session mode, but I changed that to > schroot in 2017.) > > > would changing limits.d + adding pam_limits to sudo pam config be > > enough for launchpad-buildd? > > I think schroot does support PAM. However, I don't think it's good > design for launchpad-buildd to rely on PAM, since after all it's not > doing any kind of interactive authentication. I would rather not pursue > this option any further. There are other less complicated ways to set > resource limits. Are we using PAM today already? Cause on riscv64 builder I started to see E: PAM error: Authentication token is no longer valid; new one required Chroot setup failed E: Error creating chroot session: skipping openssl from time to time. Or is it just a timing issue? -- Regards, Dimitri. buildlog_ubuntu-groovy-riscv64.openssl_1.1.1f-1ubuntu4_BUILDING.txt.gz Description: application/gzip ___ Mailing list: https://launchpad.net/~launchpad-users Post to : launchpad-users@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-users More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-users] Launchpad builder VMs upgraded to bionic
On Wed, Sep 16, 2020 at 10:22:05AM +0100, Dimitri John Ledkov wrote: > chroot builders do call ~= `sudo chroot` at some point. The main work is done by sbuild, which uses schroot, not "sudo chroot". (We used to use sbuild's sudo session mode, but I changed that to schroot in 2017.) > would changing limits.d + adding pam_limits to sudo pam config be > enough for launchpad-buildd? I think schroot does support PAM. However, I don't think it's good design for launchpad-buildd to rely on PAM, since after all it's not doing any kind of interactive authentication. I would rather not pursue this option any further. There are other less complicated ways to set resource limits. -- Colin Watson (he/him) [cjwat...@canonical.com] ___ Mailing list: https://launchpad.net/~launchpad-users Post to : launchpad-users@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-users More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-users] Launchpad builder VMs upgraded to bionic
Thanks Dimitri, Steve and Colin for the good discussion. The consensus was to update Bionic's systemd to bump such a limit and make it on-par with Focal and newer releases, so I went ahead and worked on such SRU. I found an "old" Launchpad bug [0] proposing this, so used that and backported the upstream patch proposed by Colin to Bionic systemd. Would you Dimitri or Steve take the patch and release the package to -proposed with that? The debdiff is present on Launchpad and I've added you both to the LP. Thanks in advance, Guilherme [0] https://bugs.launchpad.net/bugs/1830746 ___ Mailing list: https://launchpad.net/~launchpad-users Post to : launchpad-users@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-users More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-users] Launchpad builder VMs upgraded to bionic
[Restoring CC of launchpad-users@ - not sure why this was dropped.] On Wed, Sep 09, 2020 at 03:25:04PM -0700, Steve Langasek wrote: > On Wed, Sep 09, 2020 at 10:33:00AM +0100, Dimitri John Ledkov wrote: > > I have not tried this, but i think one should be able to create a > > snippet in /etc/security/limits.d/ with like > > > * soft memlock unlimited > > * hard memlock unlimited > > > Or to the appropriate value of 64*1024 instead of unlimited. > > Which should only take effect for things which are part of PAM sessions that > have invoked pam_limits. I don't think this would be true for the builders? Correct - pam_limits isn't going to be involved here. Would something involving DefaultLimitMEMLOCK= do the job, maybe? -- Colin Watson (he/him) [cjwat...@canonical.com] ___ Mailing list: https://launchpad.net/~launchpad-users Post to : launchpad-users@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-users More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-users] Launchpad builder VMs upgraded to bionic
On Tue, Sep 08, 2020 at 02:51:45PM -0300, Guilherme Piccoli wrote: > On Tue, Sep 8, 2020 at 1:48 PM Colin Watson wrote: > > Failing that, can somebody advise on whether there's an appropriate way > > to configure this in an image without having to maintain a fork of > > systemd? The workflow here is that we consume Ubuntu cloud images and > > make a few small changes to them, mostly things like installing > > launchpad-buildd, before publishing them to Glance for use when starting > > new builder VMs. > > I think the cloud image folks can chime in with good ideas, but I'd > consider cloud-init - likely it's possible to configure it in a way to > apply this change on boot time. Another idea: would it be possible to > change launchpad-buildd to accomplish the ulimit change ? Both of these are likely to be harder than just changing the image to add the appropriate bit of configuration, IMO. cloud-init makes sense when you're using unmodified cloud images, but we already have a pipeline for modifying images for our use; https://bazaar.launchpad.net/~canonical-sysadmins/canonical-is-charms/launchpad-buildd-image-modifier/view/head:/files/scripts/setup-ppa-buildd is the key bit here. (But I'd still need to know what exactly we need to set there.) I'd still like it to be fixed in bionic's systemd, even if we need to work around it in the short term. -- Colin Watson (he/him) [cjwat...@canonical.com] ___ Mailing list: https://launchpad.net/~launchpad-users Post to : launchpad-users@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-users More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-users] Launchpad builder VMs upgraded to bionic
On Tue, Sep 8, 2020 at 1:48 PM Colin Watson wrote: > The simplest and IMO best way to do this would be to SRU the systemd > change that bumped it to 64M [1] to bionic; we'd then pick that up in > the natural course of events by way of new cloud image builds. Has that > been considered? It feels as though what's happened here is simply that > the Launchpad build farm upgrade has surfaced a bug in bionic, so the > most appropriate thing to do would be to fix it in bionic. > Thanks Colin, it is a good idea! Although, I've seen uploads from my teammates take like 2-3 months to get merged/released on systemd package, due to security fixes on such package. It might take a while to get the fix organically if we follow such approach... > Failing that, can somebody advise on whether there's an appropriate way > to configure this in an image without having to maintain a fork of > systemd? The workflow here is that we consume Ubuntu cloud images and > make a few small changes to them, mostly things like installing > launchpad-buildd, before publishing them to Glance for use when starting > new builder VMs. > I think the cloud image folks can chime in with good ideas, but I'd consider cloud-init - likely it's possible to configure it in a way to apply this change on boot time. Another idea: would it be possible to change launchpad-buildd to accomplish the ulimit change ? Cheers, Guilherme ___ Mailing list: https://launchpad.net/~launchpad-users Post to : launchpad-users@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-users More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-users] Launchpad builder VMs upgraded to bionic
On Tue, Aug 4, 2020 at 11:50 AM Colin Watson wrote: > > The VMs in Launchpad's build farm have been on Ubuntu 16.04 (xenial) for > some time. We're intentionally fairly conservative about upgrading the > base VMs, but it's about time to have something newer, so we've just > upgraded them to Ubuntu 18.04 (bionic). The actual builds still run in > chroots or LXD containers of the appropriate Ubuntu series just as > before, so most builds shouldn't notice any difference, but the kernel > version is now 4.15 rather than 4.4. (As I type this, some VMs are > still on xenial, but they'll be replaced with bionic once they're reset > at the end of their next build.) Hi Colin et.al., first of all thanks for the builder update and heads-up! We've noticed a failure in building cryptsetup from source, reported in LP #1891473 [0]. The reason for such failure is detailed in the LP, but a summary is: - cryptsetup might make use of mlock() syscall and so, restrict its memory usage by the memlock limit (ulimit -l), if it succeeds the mlock() call; - in Xenial, the memlock limit was extremely low, so mlock() failed and the cryptsetup test hereby failing (luks2-metadata validation) succeeded, since cryptsetup continues with no locked memory; -in Bionic, such limit was bumped to 16M [1], and cryptsetup succeeds in the memory locking procedure, but...the limit is low enough in order the luks2-metadata-validation-4M exceeds that and fails - worth to notice that chroot/containers inherit those limits from host; - in Focal such a limit was quite increased (to 64M), so the tests do not fail anymore. In my understanding, we have 2 ways of resolving that: (a) We could bump the memlock limit in PPA builders to 64M (to match real-life cases of Focal, specially useful for Focal packages being built); (b) Alternatively, we could *reduce* the memlock limit in the cryptsetup package (we have enough privilege to decrease the limit in the container), so it mimics the old behavior of Xenial and so cryptsetup doesn't lock memory anymore, and its tests gladly succeed. I've managed to have a build succeeding in my PPA using this approach. I vote for approach (a), since by doing (b) we're "masking off" the mlock() in the cryptsetup packages, which in Bionic+ versions will be successful in real-life scenario (due to the larger memlock limits), but wouldn't be tested in build time. Opinions are much appreciated, I currently have a fix stuck due to the FTBFS in cryptsetup, so I'd like to move-on with that in order to move-on with my upload. Cheers, Guilherme [0] https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/1891473 [1] https://github.com/systemd/systemd/commit/fb3ae275cb ___ Mailing list: https://launchpad.net/~launchpad-users Post to : launchpad-users@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-users More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-users] Launchpad builder VMs upgraded to bionic
On Tue, Sep 08, 2020 at 12:57:28PM -0300, Guilherme Piccoli wrote: > Hi Colin et.al., first of all thanks for the builder update and > heads-up! We've noticed a failure in building cryptsetup from source, > reported in LP #1891473 [0]. The reason for such failure is detailed > in the LP, but a summary is: > - cryptsetup might make use of mlock() syscall and so, restrict its > memory usage by the memlock limit (ulimit -l), if it succeeds the > mlock() call; > - in Xenial, the memlock limit was extremely low, so mlock() failed > and the cryptsetup test hereby failing (luks2-metadata validation) > succeeded, since cryptsetup continues with no locked memory; > -in Bionic, such limit was bumped to 16M [1], and cryptsetup succeeds > in the memory locking procedure, but...the limit is low enough in > order the luks2-metadata-validation-4M exceeds that and fails - worth > to notice that chroot/containers inherit those limits from host; > - in Focal such a limit was quite increased (to 64M), so the tests do > not fail anymore. > > In my understanding, we have 2 ways of resolving that: > > (a) We could bump the memlock limit in PPA builders to 64M (to match > real-life cases of Focal, specially useful for Focal packages being > built); The simplest and IMO best way to do this would be to SRU the systemd change that bumped it to 64M [1] to bionic; we'd then pick that up in the natural course of events by way of new cloud image builds. Has that been considered? It feels as though what's happened here is simply that the Launchpad build farm upgrade has surfaced a bug in bionic, so the most appropriate thing to do would be to fix it in bionic. Failing that, can somebody advise on whether there's an appropriate way to configure this in an image without having to maintain a fork of systemd? The workflow here is that we consume Ubuntu cloud images and make a few small changes to them, mostly things like installing launchpad-buildd, before publishing them to Glance for use when starting new builder VMs. [1] https://github.com/systemd/systemd/commit/91cfdd8d29 -- Colin Watson (he/him) [cjwat...@canonical.com] ___ Mailing list: https://launchpad.net/~launchpad-users Post to : launchpad-users@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-users More help : https://help.launchpad.net/ListHelp
