[Kernel-packages] [Bug 2038567] Re: Mantic 6.5.0-7 kernel causes regression in LXD container usage
Thanks John, it has been confirmed that 1ea37b26d720 UBUNTU: SAUCE: apparmor4.0.0 [73/76]: userns - allow restricting unprivileged change_profile is causing the issue. It has a sysctl to disable its behavior, but the sysctl can't be defaulted to off in the kernel. So to disable the sysctl, either 1. lxd needs to do it dynamically like it is doing for some other sysctls 2. we need the disable it at the system level 3. we revert the patch For the time frame we are looking at, I recommend reverting the patch. Doing so will not materially affect the userns mediation feature. This patch is about closing off a confinement escape. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2038567 Title: Mantic 6.5.0-7 kernel causes regression in LXD container usage Status in Release Notes for Ubuntu: New Status in apparmor package in Ubuntu: New Status in linux package in Ubuntu: Incomplete Status in lxd package in Ubuntu: New Status in snapd package in Ubuntu: New Bug description: Following upgrade to 6.5.0-7 kernel in mantic cloud images we are seeing a regression in our cloud image tests. The test runs the following: ``` lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait ``` The `lxc exec mantic -- cloud-init status --wait` times out after 240s and will fail our test as a result. I have been able to replicate in a local VM ``` wget http://cloud-images.ubuntu.com/mantic/20231005/mantic-server-cloudimg-amd64.img wget --output-document=launch-qcow2-image-qemu.sh https://gist.githubusercontent.com/philroche/14c241c086a5730481e24178b654268f/raw/7af95cd4dfc8e1d0600e6118803d2c866765714e/gistfile1.txt chmod +x launch-qcow2-image-qemu.sh ./launch-qcow2-image-qemu.sh --password passw0rd --image ./mantic-server-cloudimg-amd64.img cat < "./reproducer.sh" #!/bin/bash -eux lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait EOF chmod +x ./reproducer.sh sshpass -p passw0rd scp -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -P ./reproducer.sh ubuntu@127.0.0.1:~/ sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get update sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get upgrade --assume-yes sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 ./reproducer.sh ``` The issue is not present with the 6.5.0-5 kernel and the issue is present regardless of the container launched. I tried the jammy container to test this. From my test VM ``` ubuntu@cloudimg:~$ uname --all Linux cloudimg 6.5.0-7-generic #7-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 29 09:14:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ubuntu@cloudimg:~$ uname --kernel-release 6.5.0-7-generic ``` This is a regression in our test that will block 23.10 cloud image release next week. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/2038567/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2038567] Re: Mantic 6.5.0-7 kernel causes regression in LXD container usage
on my machine (specs at the end) running Jammy as the host, and launching a Jammy container: 1. lxc launch ubuntu:jammy test-jammy-on-jammy from journal Oct 06 07:36:47 j5awry-sys76 kernel: audit: type=1400 audit(1696595807.223:51559): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="lxd-test-jammy-on-jammy_" name="/run/systemd/unit-root/proc/" pid=723735 comm="(d-logind)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec" Oct 06 07:36:47 j5awry-sys76 kernel: audit: type=1400 audit(1696595807.227:51560): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="lxd-test-jammy-on-jammy_" name="/run/systemd/unit-root/tmp/" pid=723724 comm="(crub_all)" flags="rw, nosuid, remount, bind" Oct 06 07:36:47 j5awry-sys76 kernel: audit: type=1400 audit(1696595807.239:51561): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="lxd-test-jammy-on-jammy_" name="/run/systemd/unit-root/proc/" pid=723750 comm="(ostnamed)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec" so i get the same `mount - failed flags match` that i see above, but not the `file-inherit` denies. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2038567 Title: Mantic 6.5.0-7 kernel causes regression in LXD container usage Status in Release Notes for Ubuntu: New Status in apparmor package in Ubuntu: New Status in linux package in Ubuntu: Incomplete Status in lxd package in Ubuntu: New Status in snapd package in Ubuntu: New Bug description: Following upgrade to 6.5.0-7 kernel in mantic cloud images we are seeing a regression in our cloud image tests. The test runs the following: ``` lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait ``` The `lxc exec mantic -- cloud-init status --wait` times out after 240s and will fail our test as a result. I have been able to replicate in a local VM ``` wget http://cloud-images.ubuntu.com/mantic/20231005/mantic-server-cloudimg-amd64.img wget --output-document=launch-qcow2-image-qemu.sh https://gist.githubusercontent.com/philroche/14c241c086a5730481e24178b654268f/raw/7af95cd4dfc8e1d0600e6118803d2c866765714e/gistfile1.txt chmod +x launch-qcow2-image-qemu.sh ./launch-qcow2-image-qemu.sh --password passw0rd --image ./mantic-server-cloudimg-amd64.img cat < "./reproducer.sh" #!/bin/bash -eux lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait EOF chmod +x ./reproducer.sh sshpass -p passw0rd scp -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -P ./reproducer.sh ubuntu@127.0.0.1:~/ sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get update sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get upgrade --assume-yes sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 ./reproducer.sh ``` The issue is not present with the 6.5.0-5 kernel and the issue is present regardless of the container launched. I tried the jammy container to test this. From my test VM ``` ubuntu@cloudimg:~$ uname --all Linux cloudimg 6.5.0-7-generic #7-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 29 09:14:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ubuntu@cloudimg:~$ uname --kernel-release 6.5.0-7-generic ``` This is a regression in our test that will block 23.10 cloud image release next week. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/2038567/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2038567] Re: Mantic 6.5.0-7 kernel causes regression in LXD container usage
Repeating a bit with a Jammy container (hence new comment) ### PRE CONDITION this is using the custom Mantic VM _and_ has apparmor_restrict_unprivileged_unconfined disabled sudo bash -c "echo 0 > /proc/sys/kernel/apparmor_restrict_unprivileged_unconfined" 1. start a jammy container lxc launch ubuntu:jammy Creating the instance Instance name is: alive-bee Starting alive-bee 2. see some apparmor denies in journal Oct 06 12:32:57 mantic-cust-vm kernel: audit: type=1400 audit(1696595577.647:954): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="lxd-alive-bee_" name="/run/systemd/unit-root/proc/" pid=5421 comm="(d-logind)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec" Oct 06 12:33:01 mantic-cust-vm kernel: kauditd_printk_skb: 20 callbacks suppressed Oct 06 12:33:01 mantic-cust-vm kernel: audit: type=1400 audit(1696595581.539:975): apparmor="DENIED" operation="file_inherit" class="file" namespace="root//lxd-alive-bee_" profile="snap.lxd.hook.install" name="/apparmor/.null" pid=5538 comm="snap-exec" requested_mask="wr" denied_mask="wr" fsuid=100 ouid=0 Oct 06 12:33:03 mantic-cust-vm kernel: audit: type=1400 audit(1696595583.771:976): apparmor="DENIED" operation="file_inherit" class="net" namespace="root//lxd-alive-bee_" profile="/snap/snapd/20092/usr/lib/snapd/snap-confine" pid=5784 comm="snap-confine" family="netlink" sock_type="raw" protocol=15 requested_mask="send receive" denied_mask="send receive" Oct 06 12:33:03 mantic-cust-vm kernel: audit: type=1400 audit(1696595583.779:977): apparmor="DENIED" operation="file_inherit" class="file" namespace="root//lxd-alive-bee_" profile="snap.lxd.hook.configure" name="/apparmor/.null" pid=5784 comm="snap-exec" requested_mask="wr" denied_mask="wr" fsuid=100 ouid=0 Oct 06 12:33:03 mantic-cust-vm kernel: audit: type=1400 audit(1696595583.791:978): apparmor="DENIED" operation="file_inherit" class="file" namespace="root//lxd-alive-bee_" profile="/snap/snapd/20092/usr/lib/snapd/snap-confine" name="/apparmor/.null" pid=5784 comm="aa-exec" requested_mask="wr" denied_mask="wr" fsuid=100 ouid=0 Oct 06 12:33:04 mantic-cust-vm kernel: audit: type=1400 audit(1696595584.007:979): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="lxd-alive-bee_" name="/run/systemd/unit-root/proc/" pid=5933 comm="(imedated)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec" 3. snap changes is Done root@alive-bee:~# snap changes ID Status Spawn Ready Summary 1Done9 days ago, at 02:11 UTC today at 12:33 UTC Initialize system state 2Donetoday at 12:32 UTCtoday at 12:33 UTC Initialize device 4. cloud-init is done root@alive-bee:~# cloud-init status status: done So using the latest released jammy container is also now launching "successfully." Unsure how the other apparmor things denies affect container performance. Running a quick spot check of my machine (Jammy) launching a Jammy container -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2038567 Title: Mantic 6.5.0-7 kernel causes regression in LXD container usage Status in Release Notes for Ubuntu: New Status in apparmor package in Ubuntu: New Status in linux package in Ubuntu: Incomplete Status in lxd package in Ubuntu: New Status in snapd package in Ubuntu: New Bug description: Following upgrade to 6.5.0-7 kernel in mantic cloud images we are seeing a regression in our cloud image tests. The test runs the following: ``` lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait ``` The `lxc exec mantic -- cloud-init status --wait` times out after 240s and will fail our test as a result. I have been able to replicate in a local VM ``` wget http://cloud-images.ubuntu.com/mantic/20231005/mantic-server-cloudimg-amd64.img wget --output-document=launch-qcow2-image-qemu.sh https://gist.githubusercontent.com/philroche/14c241c086a5730481e24178b654268f/raw/7af95cd4dfc8e1d0600e6118803d2c866765714e/gistfile1.txt chmod +x launch-qcow2-image-qemu.sh ./launch-qcow2-image-qemu.sh --password passw0rd --image ./mantic-server-cloudimg-amd64.img cat < "./reproducer.sh" #!/bin/bash -eux lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait EOF chmod +x ./reproducer.sh sshpass -p passw0rd scp -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -P ./reproducer.sh ubuntu@127.0.0.1:~/ sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get update sshpass -p
[Kernel-packages] [Bug 2038567] Re: Mantic 6.5.0-7 kernel causes regression in LXD container usage
Did the following: 1. launched a new VM from the custom build lxc launch mantic-20231005 --vm --device root,size=20GiB mantic-cust-vm 2. pushed squashfs and lxc metadata from same custom build lxc file push build.output/livecd.ubuntu-cpc.squashfs mantic-cust-vm/root/ lxc file push vm/mantic-server-cloudimg-amd64-lxd.tar.xz mantic-cust-vm/root/ 3. initialized lxd lxd init --auto 3. imported into lxc lxc image import ./mantic-server-cloudimg-amd64-lxd.tar.xz ./livecd.ubuntu-cpc.squashfs --alias mantic-cust-con 4. launched the container lxc launch mantic-cust-con m-c-c 5. observed the DENIED ptrace Oct 06 12:27:01 mantic-cust-vm kernel: audit: type=1400 audit(1696595221.386:113): apparmor="DENIED" operation="ptrace" class="ptrace" profile="lxd-m-c-c_" pid=2420 comm="systemd" requested_mask="read" denied_mask="read" peer="lxd-m-c- c_//" 6. deleted the container lxc stop m-c-c lxc delete m-c-c 6. disabled apparmor_restrict_unprivileged_unconfined sudo bash -c "echo 0 > /proc/sys/kernel/apparmor_restrict_unprivileged_unconfined" 7. launched container lxc launch mantic-cust-con m-c-c 8. see different apparmor denied messages: Oct 06 12:29:58 mantic-cust-vm kernel: audit: type=1400 audit(1696595398.722:905): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="lxd-m-c-c_" name="/run/systemd/unit-root/proc/" pid=4055 comm="(d-logind)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec" Oct 06 12:29:58 mantic-cust-vm kernel: audit: type=1400 audit(1696595398.766:906): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="lxd-m-c-c_" name="/run/systemd/unit-root/proc/" pid=4048 comm="(polkitd)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec" Oct 06 12:29:58 mantic-cust-vm kernel: audit: type=1400 audit(1696595398.818:907): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="lxd-m-c-c_" name="/run/systemd/unit-root/proc/" pid=4071 comm="(ostnamed)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec" Oct 06 12:29:58 mantic-cust-vm kernel: audit: type=1400 audit(1696595398.874:908): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" label="lxd-m-c-c_//&:lxd-m-c-c_:unconfined" name="rsyslogd" pid=4062 comm="apparmor_parser" Oct 06 12:29:59 mantic-cust-vm kernel: audit: type=1400 audit(1696595399.106:909): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" label="lxd-m-c-c_//&:lxd-m-c-c_:unconfined" name="/usr/lib/snapd/snap-confine" pid=4130 comm="apparmor_parser" Oct 06 12:29:59 mantic-cust-vm kernel: audit: type=1400 audit(1696595399.106:910): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" label="lxd-m-c-c_//&:lxd-m-c-c_:unconfined" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=4130 comm="apparmor_parser" Oct 06 12:29:59 mantic-cust-vm kernel: audit: type=1400 audit(1696595399.482:911): apparmor="DENIED" operation="file_inherit" class="net" namespace="root//lxd-m-c-c_" profile="/usr/lib/snapd/snap-confine" pid=4146 comm="snap-confine" family="netlink" sock_type="raw" protocol=15 requested_mask="send receive" denied_mask="send receive" Oct 06 12:29:59 mantic-cust-vm kernel: audit: type=1400 audit(1696595399.498:912): apparmor="DENIED" operation="file_inherit" class="file" namespace="root//lxd-m-c-c_" profile="snap-update-ns.lxd" name="/apparmor/.null" pid=4157 comm="6" requested_mask="wr" denied_mask="wr" fsuid=100 ouid=0 Oct 06 12:29:59 mantic-cust-vm kernel: audit: type=1400 audit(1696595399.558:913): apparmor="DENIED" operation="file_inherit" class="file" namespace="root//lxd-m-c-c_" profile="snap.lxd.hook.install" name="/apparmor/.null" pid=4146 comm="snap-exec" requested_mask="wr" denied_mask="wr" fsuid=100 ouid=0 9. However, these are not affecting the container in the same fashion. Snap finishes initializing, and cloud-init finishes root@mantic-cust-vm:~# lxc shell m-c-c root@m-c-c:~# snap changes ID Status Spawn Ready Summary 1Doneyesterday at 21:03 UTC today at 12:30 UTC Initialize system state 2Donetoday at 12:29 UTC today at 12:30 UTC Initialize device root@m-c-c:~# cloud-init status status: done -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2038567 Title: Mantic 6.5.0-7 kernel causes regression in LXD container usage Status in Release Notes for Ubuntu: New Status in apparmor package in Ubuntu: New Status in linux package in Ubuntu: Incomplete Status in lxd package in Ubuntu: New Status in snapd package in Ubuntu: New Bug description: Following upgrade to 6.5.0-7 kernel in mantic cloud images we are seeing a regression in our cloud image tests. The test runs
[Kernel-packages] [Bug 2038567] Re: Mantic 6.5.0-7 kernel causes regression in LXD container usage
To test if 1ea37b26d720 UBUNTU: SAUCE: apparmor4.0.0 [73/76]: userns - allow restricting unprivileged change_profile is the cause of the ptrace denials. You can disable it using sudo bash -c "echo 0 > /proc/sys/kernel/apparmor_restrict_unprivileged_unconfined" -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2038567 Title: Mantic 6.5.0-7 kernel causes regression in LXD container usage Status in Release Notes for Ubuntu: New Status in apparmor package in Ubuntu: New Status in linux package in Ubuntu: Incomplete Status in lxd package in Ubuntu: New Status in snapd package in Ubuntu: New Bug description: Following upgrade to 6.5.0-7 kernel in mantic cloud images we are seeing a regression in our cloud image tests. The test runs the following: ``` lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait ``` The `lxc exec mantic -- cloud-init status --wait` times out after 240s and will fail our test as a result. I have been able to replicate in a local VM ``` wget http://cloud-images.ubuntu.com/mantic/20231005/mantic-server-cloudimg-amd64.img wget --output-document=launch-qcow2-image-qemu.sh https://gist.githubusercontent.com/philroche/14c241c086a5730481e24178b654268f/raw/7af95cd4dfc8e1d0600e6118803d2c866765714e/gistfile1.txt chmod +x launch-qcow2-image-qemu.sh ./launch-qcow2-image-qemu.sh --password passw0rd --image ./mantic-server-cloudimg-amd64.img cat < "./reproducer.sh" #!/bin/bash -eux lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait EOF chmod +x ./reproducer.sh sshpass -p passw0rd scp -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -P ./reproducer.sh ubuntu@127.0.0.1:~/ sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get update sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get upgrade --assume-yes sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 ./reproducer.sh ``` The issue is not present with the 6.5.0-5 kernel and the issue is present regardless of the container launched. I tried the jammy container to test this. From my test VM ``` ubuntu@cloudimg:~$ uname --all Linux cloudimg 6.5.0-7-generic #7-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 29 09:14:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ubuntu@cloudimg:~$ uname --kernel-release 6.5.0-7-generic ``` This is a regression in our test that will block 23.10 cloud image release next week. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/2038567/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2038567] Re: Mantic 6.5.0-7 kernel causes regression in LXD container usage
Oct 05 21:25:27 novel-ram kernel: audit: type=1400 audit(1696541127.240:6185): apparmor="DENIED" operation="ptrace" class="ptrace" profile="lxd-current-iguana_" pid=12702 comm="systemctl" requested_mask="read" denied_mask="read" peer="lxd-current-iguana_//" indicates 1ea37b26d720 UBUNTU: SAUCE: apparmor4.0.0 [73/76]: userns - allow restricting unprivileged change_profile -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2038567 Title: Mantic 6.5.0-7 kernel causes regression in LXD container usage Status in Release Notes for Ubuntu: New Status in apparmor package in Ubuntu: New Status in linux package in Ubuntu: Incomplete Status in lxd package in Ubuntu: New Status in snapd package in Ubuntu: New Bug description: Following upgrade to 6.5.0-7 kernel in mantic cloud images we are seeing a regression in our cloud image tests. The test runs the following: ``` lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait ``` The `lxc exec mantic -- cloud-init status --wait` times out after 240s and will fail our test as a result. I have been able to replicate in a local VM ``` wget http://cloud-images.ubuntu.com/mantic/20231005/mantic-server-cloudimg-amd64.img wget --output-document=launch-qcow2-image-qemu.sh https://gist.githubusercontent.com/philroche/14c241c086a5730481e24178b654268f/raw/7af95cd4dfc8e1d0600e6118803d2c866765714e/gistfile1.txt chmod +x launch-qcow2-image-qemu.sh ./launch-qcow2-image-qemu.sh --password passw0rd --image ./mantic-server-cloudimg-amd64.img cat < "./reproducer.sh" #!/bin/bash -eux lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait EOF chmod +x ./reproducer.sh sshpass -p passw0rd scp -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -P ./reproducer.sh ubuntu@127.0.0.1:~/ sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get update sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get upgrade --assume-yes sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 ./reproducer.sh ``` The issue is not present with the 6.5.0-5 kernel and the issue is present regardless of the container launched. I tried the jammy container to test this. From my test VM ``` ubuntu@cloudimg:~$ uname --all Linux cloudimg 6.5.0-7-generic #7-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 29 09:14:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ubuntu@cloudimg:~$ uname --kernel-release 6.5.0-7-generic ``` This is a regression in our test that will block 23.10 cloud image release next week. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/2038567/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2038567] Re: Mantic 6.5.0-7 kernel causes regression in LXD container usage
livecd-rootfs 23.10.55 for mantic is currently migrating, and has apparmor changes as well (mounting different features in the build chroot). To help rule out some issues, I built a a qcow2 image and a squashfs for mantic using livecd-rootfs 23.10.55 Running the mantic host, and launching a released jammy container # On the mantic host VM journalctl -f -b -k Oct 05 21:25:26 novel-ram kernel: kauditd_printk_skb: 220 callbacks suppressed Oct 05 21:25:26 novel-ram kernel: audit: type=1400 audit(1696541126.968:6178): apparmor="DENIED" operation="ptrace" class="ptrace" profile="lxd-current-iguana_" pid=11660 comm="systemd" requested_mask="read" denied_mask="read" peer="lxd-current-iguana_//" Oct 05 21:25:27 novel-ram kernel: audit: type=1400 audit(1696541127.036:6179): apparmor="DENIED" operation="signal" class="signal" profile="lxd-current-iguana_" pid=12656 comm="snapd" requested_mask="send" denied_mask="send" signal=urg peer="lxd-current-iguana_//" Oct 05 21:25:27 novel-ram kernel: audit: type=1400 audit(1696541127.044:6180): apparmor="DENIED" operation="ptrace" class="ptrace" profile="lxd-current-iguana_" pid=11722 comm="systemd-journal" requested_mask="read" denied_mask="read" peer="lxd-current-iguana_//" Oct 05 21:25:27 novel-ram kernel: audit: type=1400 audit(1696541127.044:6181): apparmor="DENIED" operation="ptrace" class="ptrace" profile="lxd-current-iguana_" pid=11722 comm="systemd-journal" requested_mask="read" denied_mask="read" peer="lxd-current-iguana_//" Oct 05 21:25:27 novel-ram kernel: audit: type=1400 audit(1696541127.168:6182): apparmor="DENIED" operation="ptrace" class="ptrace" profile="lxd-current-iguana_" pid=12699 comm="systemctl" requested_mask="read" denied_mask="read" peer="lxd-current-iguana_//" Oct 05 21:25:27 novel-ram kernel: audit: type=1400 audit(1696541127.228:6183): apparmor="DENIED" operation="signal" class="signal" profile="lxd-current-iguana_" pid=11660 comm="systemd" requested_mask="send" denied_mask="send" signal=exists peer="lxd-current-iguana_//" Oct 05 21:25:27 novel-ram kernel: audit: type=1400 audit(1696541127.236:6184): apparmor="DENIED" operation="ptrace" class="ptrace" profile="lxd-current-iguana_" pid=12701 comm="systemctl" requested_mask="read" denied_mask="read" peer="lxd-current-iguana_//" Oct 05 21:25:27 novel-ram kernel: audit: type=1400 audit(1696541127.240:6185): apparmor="DENIED" operation="ptrace" class="ptrace" profile="lxd-current-iguana_" pid=12702 comm="systemctl" requested_mask="read" denied_mask="read" peer="lxd-current-iguana_//" Oct 05 21:25:27 novel-ram kernel: audit: type=1400 audit(1696541127.244:6186): apparmor="DENIED" operation="ptrace" class="ptrace" profile="lxd-current-iguana_" pid=12703 comm="systemctl" requested_mask="read" denied_mask="read" peer="lxd-current-iguana_//" Oct 05 21:25:27 novel-ram kernel: audit: type=1400 audit(1696541127.252:6187): apparmor="DENIED" operation="ptrace" class="ptrace" profile="lxd-current-iguana_" pid=12704 comm="systemctl" requested_mask="read" denied_mask="read" peer="lxd-current-iguana_//" within the mantic container: $ snap changes ID Status Spawn Ready Summary 1Error today at 21:03 UTC today at 21:14 UTC Initialize system state 2Donetoday at 21:14 UTC today at 21:14 UTC Initialize device 3Error today at 21:14 UTC today at 21:14 UTC Initialize system state 4Error today at 21:19 UTC today at 21:19 UTC Initialize system state 5Error today at 21:24 UTC today at 21:30 UTC Initialize system state $ snap tasks 5 Status Spawn Ready Summary Donetoday at 21:24 UTC today at 21:30 UTC Ensure prerequisites for "snapd" are available Undone today at 21:24 UTC today at 21:30 UTC Prepare snap "/var/lib/snapd/seed/snaps/snapd_20092.snap" (20092) Error today at 21:24 UTC today at 21:24 UTC Mount snap "snapd" (20092) Holdtoday at 21:24 UTC today at 21:24 UTC Copy snap "snapd" data Holdtoday at 21:24 UTC today at 21:24 UTC Setup snap "snapd" (20092) security profiles ... Mount snap "snapd" (20092) 2023-10-05T21:24:57Z ERROR systemctl command [reload-or-restart snap-snapd-20092.mount] failed with exit status 4: Failed to reload-or-restart snap-snapd-20092.mount: Transaction for snap-snapd-20092.mount/start is destructive (halt.target has 'start' job queued, but 'stop' is included in transaction). See system logs and 'systemctl status snap-snapd-20092.mount' for details. # on the mantic host journalctl -f -b -k Oct 05 21:30:55 novel-ram kernel: kauditd_printk_skb: 184 callbacks suppressed Oct 05 21:30:55 novel-ram kernel: audit: type=1400 audit(1696541455.545:7246): apparmor="DENIED" operation="signal" class="signal" profile="lxd-sharing-tick_" pid=14545 comm="snap" requested_mask="send" denied_mask="send" signal=urg peer="lxd-sharing-tick_//" Oct 05 21:30:56 novel-ram kernel: audit: type=1400 audit(1696541456.641:7247):
[Kernel-packages] [Bug 2038567] Re: Mantic 6.5.0-7 kernel causes regression in LXD container usage
apparmor side there are 2 immediate suspects. 1. kernel 0191e8433f76 UBUNTU: SAUCE: apparmor4.0.0: apparmor: Fix regression in mount mediation 2. userspace mount work to fix the mount CVE https://bugs.launchpad.net/apparmor/+bug/1597017 https://gitlab.com/apparmor/apparmor/-/merge_requests/1054 https://gitlab.com/apparmor/apparmor/-/merge_requests/333 I doubt #2 as this has been in the apparmor userspace in mantic since before FF. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2038567 Title: Mantic 6.5.0-7 kernel causes regression in LXD container usage Status in Release Notes for Ubuntu: New Status in apparmor package in Ubuntu: New Status in linux package in Ubuntu: Incomplete Status in lxd package in Ubuntu: New Status in snapd package in Ubuntu: New Bug description: Following upgrade to 6.5.0-7 kernel in mantic cloud images we are seeing a regression in our cloud image tests. The test runs the following: ``` lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait ``` The `lxc exec mantic -- cloud-init status --wait` times out after 240s and will fail our test as a result. I have been able to replicate in a local VM ``` wget http://cloud-images.ubuntu.com/mantic/20231005/mantic-server-cloudimg-amd64.img wget --output-document=launch-qcow2-image-qemu.sh https://gist.githubusercontent.com/philroche/14c241c086a5730481e24178b654268f/raw/7af95cd4dfc8e1d0600e6118803d2c866765714e/gistfile1.txt chmod +x launch-qcow2-image-qemu.sh ./launch-qcow2-image-qemu.sh --password passw0rd --image ./mantic-server-cloudimg-amd64.img cat < "./reproducer.sh" #!/bin/bash -eux lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait EOF chmod +x ./reproducer.sh sshpass -p passw0rd scp -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -P ./reproducer.sh ubuntu@127.0.0.1:~/ sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get update sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get upgrade --assume-yes sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 ./reproducer.sh ``` The issue is not present with the 6.5.0-5 kernel and the issue is present regardless of the container launched. I tried the jammy container to test this. From my test VM ``` ubuntu@cloudimg:~$ uname --all Linux cloudimg 6.5.0-7-generic #7-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 29 09:14:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ubuntu@cloudimg:~$ uname --kernel-release 6.5.0-7-generic ``` This is a regression in our test that will block 23.10 cloud image release next week. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/2038567/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2038567] Re: Mantic 6.5.0-7 kernel causes regression in LXD container usage
Current suspects are out of date apparmor features in livecd-rootfs pending https://launchpad.net/ubuntu/+source/livecd-rootfs/23.10.55 kernel, apparmor, snapd, lxd, snapd again having fits about all of them because of: .. Make snap "snapd" (20092) available to the system 2023-10-05T19:04:57Z INFO Requested daemon restart (snapd snap). .. Copy snap "lxd" data 2023-10-05T19:04:56Z ERROR unlinkat /var/snap/lxd/common/var/lib/lxcfs/proc/cpuinfo: function not implemented .. Run install hook of "lxd" snap if present 2023-10-05T19:04:55Z ERROR run hook "install": cannot read mount namespace identifier of pid 1: Permission denied and also because of: Oct 05 19:21:39 mantic-con-priv systemd[1]: snapd.service: Got notification message from PID 2560, but reception only permitted for main PID 2338 Oct 05 19:21:39 mantic-con-priv snapd[2338]: taskrunner.go:299: [change 7 "Setup snap \"snapd\" (20092) security profiles" task] failed: cannot reload udev rules: exit status 1 Oct 05 19:21:39 mantic-con-priv snapd[2338]: udev output: Oct 05 19:21:39 mantic-con-priv snapd[2338]: Failed to send reload request: No such file or directory Oct 05 19:21:39 mantic-con-priv systemd[1]: snap-snapd-20092.mount: Deactivated successfully. Oct 05 19:21:39 mantic-con-priv systemd[1]: snap-snapd-20092.mount: Unit process 2559 (snapfuse) remains running after unit stopped. Oct 05 19:21:39 mantic-con-priv systemd[1]: Reloading requested from client PID 2565 (unit snapd.service)... Oct 05 19:21:39 mantic-con-priv systemd[1]: Reloading... Oct 05 19:21:39 mantic-con-priv (sd-gens)[2568]: Read-only bind remount failed, ignoring: Permission denied and because of: Oct 05 19:20:58 cloudimg kernel: audit: type=1400 audit(1696533658.780:276): apparmor="DENIED" operation="mount" class="mount" info="failed type match" error=-13 profile="lxd-dominant- goldfish_" name="/snap/" pid=1940 comm="(sd- gens)" flags="ro, remount, bind" but could be util-linux too ** Also affects: apparmor (Ubuntu) Importance: Undecided Status: New ** Also affects: lxd (Ubuntu) Importance: Undecided Status: New ** Also affects: snapd (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2038567 Title: Mantic 6.5.0-7 kernel causes regression in LXD container usage Status in Release Notes for Ubuntu: New Status in apparmor package in Ubuntu: New Status in linux package in Ubuntu: Incomplete Status in lxd package in Ubuntu: New Status in snapd package in Ubuntu: New Bug description: Following upgrade to 6.5.0-7 kernel in mantic cloud images we are seeing a regression in our cloud image tests. The test runs the following: ``` lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait ``` The `lxc exec mantic -- cloud-init status --wait` times out after 240s and will fail our test as a result. I have been able to replicate in a local VM ``` wget http://cloud-images.ubuntu.com/mantic/20231005/mantic-server-cloudimg-amd64.img wget --output-document=launch-qcow2-image-qemu.sh https://gist.githubusercontent.com/philroche/14c241c086a5730481e24178b654268f/raw/7af95cd4dfc8e1d0600e6118803d2c866765714e/gistfile1.txt chmod +x launch-qcow2-image-qemu.sh ./launch-qcow2-image-qemu.sh --password passw0rd --image ./mantic-server-cloudimg-amd64.img cat < "./reproducer.sh" #!/bin/bash -eux lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait EOF chmod +x ./reproducer.sh sshpass -p passw0rd scp -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -P ./reproducer.sh ubuntu@127.0.0.1:~/ sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get update sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get upgrade --assume-yes sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 ./reproducer.sh ``` The issue is not present with the 6.5.0-5 kernel and the issue is present regardless of the container launched. I tried the jammy container to test this. From my test VM ``` ubuntu@cloudimg:~$ uname --all Linux cloudimg 6.5.0-7-generic #7-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 29 09:14:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ubuntu@cloudimg:~$ uname --kernel-release
[Kernel-packages] [Bug 2038567] Re: Mantic 6.5.0-7 kernel causes regression in LXD container usage
** Changed in: linux (Ubuntu) Milestone: None => ubuntu-23.10 ** Changed in: linux (Ubuntu) Importance: Undecided => Critical ** Also affects: ubuntu-release-notes Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2038567 Title: Mantic 6.5.0-7 kernel causes regression in LXD container usage Status in Release Notes for Ubuntu: New Status in linux package in Ubuntu: Incomplete Bug description: Following upgrade to 6.5.0-7 kernel in mantic cloud images we are seeing a regression in our cloud image tests. The test runs the following: ``` lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait ``` The `lxc exec mantic -- cloud-init status --wait` times out after 240s and will fail our test as a result. I have been able to replicate in a local VM ``` wget http://cloud-images.ubuntu.com/mantic/20231005/mantic-server-cloudimg-amd64.img wget --output-document=launch-qcow2-image-qemu.sh https://gist.githubusercontent.com/philroche/14c241c086a5730481e24178b654268f/raw/7af95cd4dfc8e1d0600e6118803d2c866765714e/gistfile1.txt chmod +x launch-qcow2-image-qemu.sh ./launch-qcow2-image-qemu.sh --password passw0rd --image ./mantic-server-cloudimg-amd64.img cat < "./reproducer.sh" #!/bin/bash -eux lxd init --auto --storage-backend dir lxc launch ubuntu-daily:mantic mantic lxc info mantic lxc exec mantic -- cloud-init status --wait EOF chmod +x ./reproducer.sh sshpass -p passw0rd scp -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -P ./reproducer.sh ubuntu@127.0.0.1:~/ sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get update sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 sudo apt-get upgrade --assume-yes sshpass -p passw0rd ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -p ubuntu@127.0.0.1 ./reproducer.sh ``` The issue is not present with the 6.5.0-5 kernel and the issue is present regardless of the container launched. I tried the jammy container to test this. From my test VM ``` ubuntu@cloudimg:~$ uname --all Linux cloudimg 6.5.0-7-generic #7-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 29 09:14:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ubuntu@cloudimg:~$ uname --kernel-release 6.5.0-7-generic ``` This is a regression in our test that will block 23.10 cloud image release next week. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/2038567/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
