Bug#1050256: [pkg-apparmor] Bug#1050256: autopkgtest fails on debci
On Tue, 2023-09-19 at 07:17 +0200, Salvatore Bonaccorso wrote: > On Sun, Sep 17, 2023 at 12:01:37PM +0530, intrigeri wrote: > > In the last month or so, a number of people from various Debian teams > > and other distributions have been tracking down a regression that > > affects systems upgraded to Bookworm: services that use certain > > systemd facilities such as PrivateNetwork=yes fail to start in LXC/LXD > > containers. Among other things, this breaks the autopkgtests of many > > packages, such as systemd, on ci.debian.net (#1050256). This was > > tracked down to a kernel regression, for which a fix landed in Linux > > 6.2: > > > > 1cf26c3d2c4c apparmor: fix apparmor mediating locking non-fs unix sockets > > > > Work is ongoing to backport the fix to linux-stable/linux-6.1.y. > > I'm Cc'ing John and Mathias who have been working on this. > > > > FYI, ideally this would be fixed in the upcoming Bookworm > > point-release (12.2, early October). > > Thanks for the details. Has this already been sent it to the stable > maintainers? I do not see it yet on the stable list. I believe that John has been working on the fix for the 6.1 branch, although I don't know what the status is. I don't have the necessary familiarity with apparmor internals to attempt to backport the fix myself, but I'll be very happy to test once it's available. Mathias signature.asc Description: This is a digitally signed message part
Bug#1050256: [pkg-apparmor] Bug#1050256: autopkgtest fails on debci
Control: tags -1 + confirmed moreinfo Hi, On Sun, Sep 17, 2023 at 12:01:37PM +0530, intrigeri wrote: > Control: reassign -1 src:linux > Control: retitle -1 AppArmor breaks locking non-fs Unix sockets > Control: affects -1 src:apparmor src:lxc src:systemd src:pdns src:policykit-1 > Control: found -1 6.1.38-1 > Control: found -1 6.1.38-2 > Control: notfound -1 6.3.1-1~exp1 > > Hi Debian Kernel Team, > > In the last month or so, a number of people from various Debian teams > and other distributions have been tracking down a regression that > affects systems upgraded to Bookworm: services that use certain > systemd facilities such as PrivateNetwork=yes fail to start in LXC/LXD > containers. Among other things, this breaks the autopkgtests of many > packages, such as systemd, on ci.debian.net (#1050256). This was > tracked down to a kernel regression, for which a fix landed in Linux > 6.2: > > 1cf26c3d2c4c apparmor: fix apparmor mediating locking non-fs unix sockets > > Work is ongoing to backport the fix to linux-stable/linux-6.1.y. > I'm Cc'ing John and Mathias who have been working on this. > > FYI, ideally this would be fixed in the upcoming Bookworm > point-release (12.2, early October). Thanks for the details. Has this already been sent it to the stable maintainers? I do not see it yet on the stable list. Regards, Salvatore
Bug#1050256: [pkg-apparmor] Bug#1050256: autopkgtest fails on debci
Control: reassign -1 src:linux Control: retitle -1 AppArmor breaks locking non-fs Unix sockets Control: affects -1 src:apparmor src:lxc src:systemd src:pdns src:policykit-1 Control: found -1 6.1.38-1 Control: found -1 6.1.38-2 Control: notfound -1 6.3.1-1~exp1 Hi Debian Kernel Team, In the last month or so, a number of people from various Debian teams and other distributions have been tracking down a regression that affects systems upgraded to Bookworm: services that use certain systemd facilities such as PrivateNetwork=yes fail to start in LXC/LXD containers. Among other things, this breaks the autopkgtests of many packages, such as systemd, on ci.debian.net (#1050256). This was tracked down to a kernel regression, for which a fix landed in Linux 6.2: 1cf26c3d2c4c apparmor: fix apparmor mediating locking non-fs unix sockets Work is ongoing to backport the fix to linux-stable/linux-6.1.y. I'm Cc'ing John and Mathias who have been working on this. FYI, ideally this would be fixed in the upcoming Bookworm point-release (12.2, early October). Current workarounds: - ci.debian.net was upgraded to the bookworm-backports kernel - various packages maintainers have added workarounds such as disabling PrivateNetwork=yes for autopkgtests Cheers, -- intrigeri
Bug#1050256: [pkg-apparmor] Bug#1050256: autopkgtest fails on debci
Control: severity -1 important Am 09.09.23 um 14:20 schrieb intrigeri: Hi again, Thank you all for working both on workarounds for Debian CI and on a proper upstream Linux kernel fix. Impressive cross-team work! :) +1 At this stage it seems clear that the bug and the corresponding ideal fix are in the AppArmor part of src:linux, and the bug affects at least src:apparmor and src:lxc. I'd like to reflect this in the metadata of #1050256 by reassigning the bug to Linux, and adding "affects" indications. I'll do so in the next few days unless someone objects soon. It also affects at least src:systemd, src:pdns, src:policykit-1 All those packages have added workarounds for this issue. I'll revert the workaround in systemd and notify the maintainers of pdns and policykit-1. Doing so will also be an opportunity for me to sum up the problem for the maintainers of src:linux, and let them know about our desired timeline: ideally this would be fixed in the upcoming Bookworm point-release. This being said, if said timeline can't be met in src:linux, it'll be up to the maintainers of LXC in Debian to decide what they want to do in the upcoming Bookworm point-release. If I misunderstood something important, please let me know. Sounds good to me. For now, given that all the debci hosts are running the backports kernel, I'm downgrading the severity again. When you do the reassignment, you should probably merge this bug report with #1038315 and #1042880, now that we know what the root cause is. Regards, Michael OpenPGP_signature.asc Description: OpenPGP digital signature
Bug#1050256: [pkg-apparmor] Bug#1050256: autopkgtest fails on debci
Hi again, Thank you all for working both on workarounds for Debian CI and on a proper upstream Linux kernel fix. Impressive cross-team work! :) At this stage it seems clear that the bug and the corresponding ideal fix are in the AppArmor part of src:linux, and the bug affects at least src:apparmor and src:lxc. I'd like to reflect this in the metadata of #1050256 by reassigning the bug to Linux, and adding "affects" indications. I'll do so in the next few days unless someone objects soon. Doing so will also be an opportunity for me to sum up the problem for the maintainers of src:linux, and let them know about our desired timeline: ideally this would be fixed in the upcoming Bookworm point-release. This being said, if said timeline can't be met in src:linux, it'll be up to the maintainers of LXC in Debian to decide what they want to do in the upcoming Bookworm point-release. If I misunderstood something important, please let me know. Cheers, -- intrigeri
Bug#1050256: [pkg-apparmor] Bug#1050256: autopkgtest fails on debci
Hello, Am Samstag, 2. September 2023, 01:13:11 CEST schrieb Mathias Gibbens: > A minimal reproducer is to install bookworm and create a container > with a systemd service using a hardening option like > PrivateNetwork=yes. With the latest bookworm kernel (6.1.38-4), the > service will fail. But, grab a kernel from testing (6.4.11-1) and then > things work -- with no other changes required. I tried the "oldest" > kernel on snapshot.d.o post 6.1 series (6.3.1+1~exp1 [1]) and the > service works properly with that version as well. So, something > changed in the kernel (either upstream or in Debian's packaging) > between 6.1 and 6.3 that "unbreaks" services within lxc containers. I asked in #apparmor, and John answered [11:04:33] can someone have a look at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050256 ? Short version: Debian gets unix denials when running lxc with kernel 6.1.38 from bookwork, but things work with kernel 6.3.1 [19:19:41] cboltz: ok, I will try and look at it today [07:00:34] cboltz: I didn't see anything that would cause unix failures in a first pass. I will take another pass at it tomorrow [10:01:30] cboltz: commit 1cf26c3d2c4c apparmor: fix apparmor mediating locking non-fs unix sockets So you could test if the bookwork kernel with 1cf26c3d2c4c applied on top fixes the issue. To answer a question from a later mail: Am Sonntag, 3. September 2023, 02:56:05 CEST schrieb Michael Biebl: > I also tested downgrading apparmor to 2.13.6-10 (i.e. the version from > oldstable) on a bookworm system. > > This was also sufficient to unbreak lxc. > > So it "looks" like apparmor 3.x makes assumptions about the kernel > that are not fulfilled by the kernel 6.1.x in bookworm. The difference is in the abi levels - without an abi/ include specified, unix rules don't get enforced (= allow everything), while with abi/3.0 and AppArmor >= 3.x userspace, unix rules get enforced. abi/3.0 got introduced in AppArmor 3.0, and my guess is that the abi/3.0 include was also added to the lxc profile. Actually the explanation might be slightly different (same result, but without abi/3.0 in the lxc profile): It looks like the Debian AppArmor maintainers pinned the abi to /etc/apparmor.d/abi/kernel-5.4-outoftree-network which, like abi/3.0, includes enforcing unix rules. (Note: I'm only looking at https://salsa.debian.org/apparmor-team/apparmor.git/ since I don't have a Debian machine running.) For completeness: 2.13.x doesn't support abi at all (besides ignoring abi/* includes if it finds them in a profile) so even if you have a profile with abi/3.0, unix rules won't be enforced. There's an exception: Ubuntu kernels carry some patches to enable unix and some other rules even with older AppArmor versions. Regards, Christian Boltz -- in my experience it's safe to assume developers never test [Stephan Kulow in opensuse-factory] signature.asc Description: This is a digitally signed message part.
Bug#1050256: [pkg-apparmor] Bug#1050256: autopkgtest fails on debci
Am 01.09.23 um 13:23 schrieb Michael Biebl: The only way to fix the container was to use the aforementioned `lxc.apparmor.profile = unconfined`. I think we should do that as the breakage is rather widespread and I already see individual packages trying to work around that to at least keep debci afloat. See e.g.: https://salsa.debian.org/systemd-team/systemd/-/merge_requests/211 https://salsa.debian.org/debian/pdns/-/commit/637e54ef73386541086da430553b82db78266bac or disabling the systemd hardening options completely_ https://salsa.debian.org/utopia-team/polkit/-/blob/master/debian/patches/debian/Don-t-use-PrivateNetwork-yes-for-the-systemd-unit.patch This is not a good outcome of this and the problem will become more apparent with debci running on bookworm now. I went ahead and submitted https://salsa.debian.org/lxc-team/lxc/-/merge_requests/18 since I don't see another solution atm. Looping in the release team as well for their input. Regards, Michael OpenPGP_signature.asc Description: OpenPGP digital signature
Bug#1050256: [pkg-apparmor] Bug#1050256: autopkgtest fails on debci
Am 31.08.23 um 19:54 schrieb Christian Boltz: Hello, Am Donnerstag, 31. August 2023, 08:41:59 CEST schrieb Michael Biebl: What we found so far is, that the AppArmor policy of lxc breaks any systemd service using PrivateNetwork=yes or PrivateIPC=yes when being run under lxc (running under bookworm using the bookworm kernel). I wonder what the best course of action is here. Should we disable the AA policy of lxc via a stable upload of the lxc package until the root cause is found? Unfortunately I know too little about AppArmor and lxc's AppArmor policy and my attempts to ask around for help weren't successful so far. Two quick hints, but let me warn you that I'm not familiar with lxc and also didn't check the content of the lxc-autopkgtest-lxc-iomhit_* profile. https://github.com/lxc/lxc/issues/4333 indicates that this issue was fixed in (much) a newer kernel - but that's probably not news to you since you wrote that comment ;-) That said - the DENIED log entry translates to unix send type=dgram, You could try if adding this rule to the lxc-autopkgtest-lxc-iomhit_* profile helps - but if the issue is really on the kernel side, my hope is limited). For testing, you could also try with a more broad unix send, or even unix, rule - but please don't add these broader rules to the production profile. I have no idea, where to add that and what specific syntax I should use. The profile above seems to be autogenerated and I only found a binary file with that name in /var/cache/apparmor. The only way to fix the container was to use the aforementioned `lxc.apparmor.profile = unconfined`. I think we should do that as the breakage is rather widespread and I already see individual packages trying to work around that to at least keep debci afloat. See e.g.: https://salsa.debian.org/systemd-team/systemd/-/merge_requests/211 https://salsa.debian.org/debian/pdns/-/commit/637e54ef73386541086da430553b82db78266bac or disabling the systemd hardening options completely_ https://salsa.debian.org/utopia-team/polkit/-/blob/master/debian/patches/debian/Don-t-use-PrivateNetwork-yes-for-the-systemd-unit.patch This is not a good outcome of this and the problem will become more apparent with debci running on bookworm now. Regards, Michael OpenPGP_signature.asc Description: OpenPGP digital signature
Bug#1050256: [pkg-apparmor] Bug#1050256: autopkgtest fails on debci
Hello, Am Donnerstag, 31. August 2023, 08:41:59 CEST schrieb Michael Biebl: > What we found so far is, that the AppArmor policy of lxc breaks any > systemd service using PrivateNetwork=yes or PrivateIPC=yes when being > run under lxc (running under bookworm using the bookworm kernel). > I wonder what the best course of action is here. > Should we disable the AA policy of lxc via a stable upload of the lxc > package until the root cause is found? > > Unfortunately I know too little about AppArmor and lxc's AppArmor > policy and my attempts to ask around for help weren't successful so > far. Two quick hints, but let me warn you that I'm not familiar with lxc and also didn't check the content of the lxc-autopkgtest-lxc-iomhit_* profile. https://github.com/lxc/lxc/issues/4333 indicates that this issue was fixed in (much) a newer kernel - but that's probably not news to you since you wrote that comment ;-) That said - the DENIED log entry translates to unix send type=dgram, You could try if adding this rule to the lxc-autopkgtest-lxc-iomhit_* profile helps - but if the issue is really on the kernel side, my hope is limited). For testing, you could also try with a more broad unix send, or even unix, rule - but please don't add these broader rules to the production profile. Regards, Christian Boltz -- you need a certificate, nobody knows how to do that securely (including the CAs ;-) [Bernd Paysan, https://bugs.kde.org/show_bug.cgi?id=131083] signature.asc Description: This is a digitally signed message part.
