@stgraber, I added the SRU template, let me know if something's off. Thanks!
** Description changed: + [Impact] + + Ubuntu carries a patch on top of systemd [a] to silence + namespace set up failures. This is meant as a workaround + for a bug in the LXD version shipped in Ubuntu 18.04. + + Masking namespace set up failures creates a false sense of + security for the user/admin. + + As mentioned in comment #1, systemd upstream explains that silencing + this kind of error is dangerous and should be avoided. + + Backporting the LXD fix [b] to Ubuntu 18.04 would allow namespaces + to work inside containers. This is the goal of this SRU. + + Ultimately, once LXD in Ubuntu 18.04 includes the fix [b], it would + be possible to drop the Ubuntu-specific patch for systemd [a]. This + is however *not an immediate concern for this SRU*. + + [Test Plan] + + 1) Create a 18.04 VM: + $ lxc launch images:ubuntu/18.04 lp1959047 --vm + $ sleep 30 # give it time to boot + + 2) Install and initialize LXD in it: + $ lxc exec lp1959047 -- apt-get update + $ lxc exec lp1959047 -- apt-get install -y lxd + $ lxc exec lp1959047 -- lxd init --auto + + 3) Create a Jammy container and enable systemd debugging: + $ lxc exec lp1959047 -- lxc init images:ubuntu/22.04 c1 + $ lxc exec lp1959047 -- lxc config set c1 raw.lxc 'lxc.init.cmd = /sbin/init systemd.log_level=debug' + $ lxc exec lp1959047 -- lxc start c1 + + 4) Check if namespace set up failures are logged: + $ lxc exec lp1959047 -- lxc exec c1 -- journalctl -b0 --grep 'Failed to set up namespace' + Mar 24 23:29:19 c1 systemd[99]: systemd-udevd.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied + Mar 24 23:29:19 c1 systemd[132]: systemd-networkd.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied + Mar 24 23:29:19 c1 systemd[131]: systemd-logind.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied + Mar 24 23:29:20 c1 systemd[136]: systemd-resolved.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied + Mar 24 23:29:20 c1 systemd[128]: e2scrub_reap.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied + Mar 24 23:29:23 c1 systemd[243]: systemd-hostnamed.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied + + If LXD in Ubuntu 18.04 has the patch, the "Failed to set up namespace" + messages wouldn't be there. + + [Where problems could occur] + + The LXD fix changes the Apparmor profile used for containers. This essentially + loosen the mount restrictions applied to containers. + + Weakening the Apparmor profile could make it easier for a process in the container + to do damage that would have otherwise been blocked. On the other hand, this + allows making use of namespaces/sandboxing inside the container. + + Upstream LXD has the fix since 2019 which make it less likely to run into + problems with the backport. + + The backported fix was also tested manually to ensure LXD still behaved normally + and that it avoided the namespace set up failures in Jammy containers. + + [a]: https://git.launchpad.net/ubuntu/+source/systemd/tree/debian/patches/debian/UBUNTU-Revert-namespace-be-more-careful-when-handling-namespacin.patch?h=ubuntu/jammy + [b]: https://github.com/lxc/lxd/commit/a6b780703350faff8328f3d565f6bac7b6dcf59f + + [Initial bug description] + The version of systemd (249.5-2ubuntu4) currently packaged for the Ubuntu development version (22.04 Jammy Jellyfish) totally ignores the RootDirectory= option in systemd service files. With RootDirectory, systemd should start the service after calling chroot() on the supplied directory. To test/reproduce, create a test service file with the following contents: - # /etc/systemd/system/lsb-release.service [Unit] Description=LSB Release Information [Service] Type=simple RootDirectory=/var/chroot/trusty ExecStartPre=/bin/pwd ExecStart=/usr/bin/lsb_release -a - - You should have a chroot environment in the specified RootDirectory, even though you can still deduce if systemd attempted to chroot or not from the resulting error message. + You should have a chroot environment in the specified RootDirectory, + even though you can still deduce if systemd attempted to chroot or not + from the resulting error message. In my example, I installed an end-of-life Ubuntu 14.04 Trusty Tahr in the chroot environment. On systems NOT affected by the problem, I get the following result when I start this test service. This is what I'd expect. - Jan 25 20:40:40 dolly systemd[1]: Starting LSB Release Information... Jan 25 20:40:40 dolly pwd[361]: / Jan 25 20:40:40 dolly systemd[1]: Started LSB Release Information. Jan 25 20:40:40 dolly lsb_release[362]: No LSB modules are available. Jan 25 20:40:40 dolly lsb_release[362]: Distributor ID: Ubuntu Jan 25 20:40:40 dolly lsb_release[362]: Description: Ubuntu 14.04 LTS Jan 25 20:40:40 dolly lsb_release[362]: Release: 14.04 Jan 25 20:40:40 dolly lsb_release[362]: Codename: trusty Jan 25 20:40:40 dolly systemd[1]: lsb-release.service: Succeeded. - On the problematic system, however, I get the following result. - Jan 25 21:21:08 savelog systemd[1]: Starting LSB Release Information... Jan 25 21:21:08 savelog systemd[1]: Started LSB Release Information. Jan 25 21:21:08 savelog pwd[81114]: / Jan 25 21:21:08 savelog lsb_release[81115]: No LSB modules are available. Jan 25 21:21:08 savelog lsb_release[81115]: Distributor ID: Ubuntu Jan 25 21:21:08 savelog lsb_release[81115]: Description: Ubuntu Jammy Jellyfish (development branch) Jan 25 21:21:08 savelog lsb_release[81115]: Release: 22.04 Jan 25 21:21:08 savelog lsb_release[81115]: Codename: jammy Jan 25 21:21:08 savelog systemd[1]: lsb-release.service: Deactivated successfully. - - It totally run the service on the host's root filesystem, it didn't care even the slightest that a RootDirectory is specified. + It totally run the service on the host's root filesystem, it didn't care + even the slightest that a RootDirectory is specified. Tested on the following releases / systemd versions: Ubuntu 18.04.6 Bionic Beaver – ISSUE NOT PRESENT systemd 237 +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid Ubuntu 20.04.3 Focal Fossa – ISSUE NOT PRESENT systemd 245 (245.4-4ubuntu3.15) +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid Ubuntu 21.10 Impish Indri – ISSUE NOT PRESENT systemd 248 (248.3-1ubuntu8.2) +PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS -OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP -LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified Ubuntu 22.04 Jammy Jellyfish (development branch) – ISSUE PRESENT systemd 249 (249.5-2ubuntu4) +PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS -OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP -LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified - - Note that the problem is produced under an LXC container; since systemd detects virtualization, it might change how it behaves. + Note that the problem is produced under an LXC container; since systemd + detects virtualization, it might change how it behaves. It's either a bug or an intentional change I don't understand yet (i.e. the RootDirectory option has deprecated and is about to be replaced with something else, or there are additional conditions to be met before RootDirectory is considered), but I think in the latter case I should at least get a warning that there is a change in configuration. I imagine suddenly everyone's existing service units utilizing RootDirectory silently stop working without any information regarding why. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1959047 Title: systemd ignores RootDirectory option in .service units Status in lxd package in Ubuntu: Invalid Status in systemd package in Ubuntu: Confirmed Status in lxd source package in Bionic: New Status in systemd source package in Bionic: New Status in systemd source package in Focal: New Status in systemd source package in Impish: New Bug description: [Impact] Ubuntu carries a patch on top of systemd [a] to silence namespace set up failures. This is meant as a workaround for a bug in the LXD version shipped in Ubuntu 18.04. Masking namespace set up failures creates a false sense of security for the user/admin. As mentioned in comment #1, systemd upstream explains that silencing this kind of error is dangerous and should be avoided. Backporting the LXD fix [b] to Ubuntu 18.04 would allow namespaces to work inside containers. This is the goal of this SRU. Ultimately, once LXD in Ubuntu 18.04 includes the fix [b], it would be possible to drop the Ubuntu-specific patch for systemd [a]. This is however *not an immediate concern for this SRU*. [Test Plan] 1) Create a 18.04 VM: $ lxc launch images:ubuntu/18.04 lp1959047 --vm $ sleep 30 # give it time to boot 2) Install and initialize LXD in it: $ lxc exec lp1959047 -- apt-get update $ lxc exec lp1959047 -- apt-get install -y lxd $ lxc exec lp1959047 -- lxd init --auto 3) Create a Jammy container and enable systemd debugging: $ lxc exec lp1959047 -- lxc init images:ubuntu/22.04 c1 $ lxc exec lp1959047 -- lxc config set c1 raw.lxc 'lxc.init.cmd = /sbin/init systemd.log_level=debug' $ lxc exec lp1959047 -- lxc start c1 4) Check if namespace set up failures are logged: $ lxc exec lp1959047 -- lxc exec c1 -- journalctl -b0 --grep 'Failed to set up namespace' Mar 24 23:29:19 c1 systemd[99]: systemd-udevd.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied Mar 24 23:29:19 c1 systemd[132]: systemd-networkd.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied Mar 24 23:29:19 c1 systemd[131]: systemd-logind.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied Mar 24 23:29:20 c1 systemd[136]: systemd-resolved.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied Mar 24 23:29:20 c1 systemd[128]: e2scrub_reap.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied Mar 24 23:29:23 c1 systemd[243]: systemd-hostnamed.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied If LXD in Ubuntu 18.04 has the patch, the "Failed to set up namespace" messages wouldn't be there. [Where problems could occur] The LXD fix changes the Apparmor profile used for containers. This essentially loosen the mount restrictions applied to containers. Weakening the Apparmor profile could make it easier for a process in the container to do damage that would have otherwise been blocked. On the other hand, this allows making use of namespaces/sandboxing inside the container. Upstream LXD has the fix since 2019 which make it less likely to run into problems with the backport. The backported fix was also tested manually to ensure LXD still behaved normally and that it avoided the namespace set up failures in Jammy containers. [a]: https://git.launchpad.net/ubuntu/+source/systemd/tree/debian/patches/debian/UBUNTU-Revert-namespace-be-more-careful-when-handling-namespacin.patch?h=ubuntu/jammy [b]: https://github.com/lxc/lxd/commit/a6b780703350faff8328f3d565f6bac7b6dcf59f [Initial bug description] The version of systemd (249.5-2ubuntu4) currently packaged for the Ubuntu development version (22.04 Jammy Jellyfish) totally ignores the RootDirectory= option in systemd service files. With RootDirectory, systemd should start the service after calling chroot() on the supplied directory. To test/reproduce, create a test service file with the following contents: # /etc/systemd/system/lsb-release.service [Unit] Description=LSB Release Information [Service] Type=simple RootDirectory=/var/chroot/trusty ExecStartPre=/bin/pwd ExecStart=/usr/bin/lsb_release -a You should have a chroot environment in the specified RootDirectory, even though you can still deduce if systemd attempted to chroot or not from the resulting error message. In my example, I installed an end-of-life Ubuntu 14.04 Trusty Tahr in the chroot environment. On systems NOT affected by the problem, I get the following result when I start this test service. This is what I'd expect. Jan 25 20:40:40 dolly systemd[1]: Starting LSB Release Information... Jan 25 20:40:40 dolly pwd[361]: / Jan 25 20:40:40 dolly systemd[1]: Started LSB Release Information. Jan 25 20:40:40 dolly lsb_release[362]: No LSB modules are available. Jan 25 20:40:40 dolly lsb_release[362]: Distributor ID: Ubuntu Jan 25 20:40:40 dolly lsb_release[362]: Description: Ubuntu 14.04 LTS Jan 25 20:40:40 dolly lsb_release[362]: Release: 14.04 Jan 25 20:40:40 dolly lsb_release[362]: Codename: trusty Jan 25 20:40:40 dolly systemd[1]: lsb-release.service: Succeeded. On the problematic system, however, I get the following result. Jan 25 21:21:08 savelog systemd[1]: Starting LSB Release Information... Jan 25 21:21:08 savelog systemd[1]: Started LSB Release Information. Jan 25 21:21:08 savelog pwd[81114]: / Jan 25 21:21:08 savelog lsb_release[81115]: No LSB modules are available. Jan 25 21:21:08 savelog lsb_release[81115]: Distributor ID: Ubuntu Jan 25 21:21:08 savelog lsb_release[81115]: Description: Ubuntu Jammy Jellyfish (development branch) Jan 25 21:21:08 savelog lsb_release[81115]: Release: 22.04 Jan 25 21:21:08 savelog lsb_release[81115]: Codename: jammy Jan 25 21:21:08 savelog systemd[1]: lsb-release.service: Deactivated successfully. It totally run the service on the host's root filesystem, it didn't care even the slightest that a RootDirectory is specified. Tested on the following releases / systemd versions: Ubuntu 18.04.6 Bionic Beaver – ISSUE NOT PRESENT systemd 237 +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid Ubuntu 20.04.3 Focal Fossa – ISSUE NOT PRESENT systemd 245 (245.4-4ubuntu3.15) +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid Ubuntu 21.10 Impish Indri – ISSUE NOT PRESENT systemd 248 (248.3-1ubuntu8.2) +PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS -OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP -LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified Ubuntu 22.04 Jammy Jellyfish (development branch) – ISSUE PRESENT systemd 249 (249.5-2ubuntu4) +PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS -OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP -LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified Note that the problem is produced under an LXC container; since systemd detects virtualization, it might change how it behaves. It's either a bug or an intentional change I don't understand yet (i.e. the RootDirectory option has deprecated and is about to be replaced with something else, or there are additional conditions to be met before RootDirectory is considered), but I think in the latter case I should at least get a warning that there is a change in configuration. I imagine suddenly everyone's existing service units utilizing RootDirectory silently stop working without any information regarding why. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxd/+bug/1959047/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
