Re: [systemd-devel] systemd-container: Trying to use a bookworm chroot with a buster host fails / Failed to create /init.scope control group
Am 03.12.22 um 23:38 schrieb Bernhard Übelacker: I thought if strace can observe the process in question, would gdb also be able. And found starting nspawn with gdbserver, 'set follow-fork-mode child' and gdb from inside the container via plain chroot seems working well. So it looks like the failing "syscall_0x1b7" from strace is "faccessat2" [2]. And it seems "faccessat2" got added just in kernel 5.8 [3], therefore it might fail with the kernel 4.19. So I fear this needs a newer kernel, and/or this is more a glibc issue then? Hello, just a few short additions. I was looking further into this issue, and found disabling apparmor by booting the host with "apparmor=0" did not improve the situation. Then I found following entry in the systemd debian package changelog [1][2]: * seccomp: allow turning off of seccomp filtering via env var. Since glibc 2.33 faccessat() is implemented via faccessat2(), which is breaking running containers that use such a version of glibc under systemd-nspawn in Buster. Turning off seccomp filtering via the SYSTEMD_SECCOMP env var makes it possible to run such new containers. (Closes: #984573) This fits perfectly the situation and the container starts successfully with this workaround: SYSTEMD_SECCOMP=0 systemd-nspawn --directory=/var/lib/machines/test-bookworm --boot Kind regards, Bernhard [1] https://metadata.ftp-master.debian.org/changelogs//main/s/systemd/systemd_241-7~deb10u8_changelog [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=984573
Re: [systemd-devel] systemd-container: Trying to use a bookworm chroot with a buster host fails / Failed to create /init.scope control group
On Sat, 03 Dec 2022 at 23:38:55 +0100, Bernhard Übelacker wrote: > > No SELinux or Apparmor active > > As far as I see in my test VM with minimal Debian Buster there is no SELinux. > "aa-status" returns "apparmor module is loaded.", but I did not intentionally > configure anything to it. Debian uses AppArmor by default (same as Ubuntu), unless you have intentionally configured it not to. Debian does not use SELinux by default (again, same as Ubuntu). > Yes, it is an test to use on an older Debian Buster with kernel 4.19.260-1 > a quite recent Debian Bookworm/testing system. For those on this mailing list who might not be familiar with Debian codenames, 'buster' on the host system is Debian 10, an "old stable" version released in 2019 and maintained by the Debian LTS team. The recommended stable Debian system right now is Debian 11 (released in 2021) but some change-averse organizations stay on older releases for whatever reason. 'bookworm' in the container is a development branch of the distribution: think of it as an alpha or beta of Debian 12. The Debian 12 stable release is expected to happen in mid 2023. > So it looks like the failing "syscall_0x1b7" from strace is "faccessat2" [2]. > > And it seems "faccessat2" got added just in kernel 5.8 [3], > therefore it might fail with the kernel 4.19. > So I fear this needs a newer kernel, and/or this is more a glibc issue then? The way this is usually meant to work when new user-space runs on an old kernel is that faccessat2 fails with error ENOSYS, and glibc falls back to emulating it via an older syscall like faccessat. It's OK for this fallback to occur, as long as the older syscall gives the right result. If the fallback gives the wrong result, then that would be a glibc issue (either an imperfect emulation, or a missing/unimplementable feature on older kernels). smcv
Re: [systemd-devel] systemd-container: Trying to use a bookworm chroot with a buster host fails / Failed to create /init.scope control group
(Resent after subscription, as non-subscribers get rejected.) Hello, I opened the initial Debian bug report, but did took the time to ask at systemd-devel and found this thread was already asked, so I am trying to provide further information. > Do you have any MACs in effect? No SELinux or Apparmor active As far as I see in my test VM with minimal Debian Buster there is no SELinux. "aa-status" returns "apparmor module is loaded.", but I did not intentionally configure anything to it. > Does the host use cgroupsv2 or cgroupsv2 or hybrid? The host system uses systemd v241, compiled with default-hierarchy=hybrid > Was the container configured to use either? The container uses systemd v251 with default-hierarchy=unified At the host: # systemd --version systemd 241 (241) +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS \ +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid In the container: # systemd --version systemd 252 (252.2-1) +PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID \ +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY \ -P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP \ +SYSVINIT default-hierarchy=unified > What is mounted to /sys/fs/cgroup and below? At the host: # mount | grep /sys/fs/cgroup tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) > This is new payload on old host? Yes, it is an test to use on an older Debian Buster with kernel 4.19.260-1 a quite recent Debian Bookworm/testing system. > if you force container into cgroupsv1 mode as the host (by adding > systemd.unified_cgroup_hierarchy=no to the nspawn cmdline, does that > work? I am not sure if I am using it right, but as far as I see "systemd.unified_cgroup_hierarchy=no" does not help. I added "debug" too, see below in [1]. > Also, please provide the relevant output from "strace -f -s 500 -y -o > /tmp/log.strace" (put on some pastebin) Following pastebin contains the last quarter of the log.strace file recorded by the command in [1]: https://paste.debian.net/1262752/ I thought if strace can observe the process in question, would gdb also be able. And found starting nspawn with gdbserver, 'set follow-fork-mode child' and gdb from inside the container via plain chroot seems working well. So it looks like the failing "syscall_0x1b7" from strace is "faccessat2" [2]. And it seems "faccessat2" got added just in kernel 5.8 [3], therefore it might fail with the kernel 4.19. So I fear this needs a newer kernel, and/or this is more a glibc issue then? Kind regards, Bernhard [1]# strace -f -s 500 -y -o /tmp/log.strace systemd-nspawn --directory=/var/lib/machines/test-bookworm --boot systemd.unified_cgroup_hierarchy=no debug Spawning container test-bookworm on /var/lib/machines/test-bookworm. Press ^] three times within 1s to kill container. systemd 252.2-1 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified) Detected virtualization systemd-nspawn. Detected architecture x86-64. Detected initialized system, this is not the first boot. Kernel version 4.19.0-22-amd64, our baseline is 4.15 Welcome to Debian GNU/Linux bookworm/sid! Hostname set to . sd-netlink: Failed to enable NETLINK_GET_STRICT_CHK option, ignoring: Protocol not available Failed to add address 127.0.0.1 to loopback interface: Operation not permitted
Re: [systemd-devel] systemd-container: Trying to use a bookworm chroot with a buster host fails / Failed to create /init.scope control group
On Mo, 17.10.22 01:38, Michael Biebl (mbi...@gmail.com) wrote: > What are you Missing? Come on, you have the original email: https://lists.freedesktop.org/archives/systemd-devel/2022-October/048453.html "What is mounted to /sys/fs/cgroup and below?" "if you force container into cgroupsv1 mode as the host (by adding systemd.unified_cgroup_hierarchy=no to the nspawn cmdline, does that work?" Also, please provide the relevant output from "strace -f -s 500 -y -o /tmp/log.strace" (put on some pastebin) Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] systemd-container: Trying to use a bookworm chroot with a buster host fails / Failed to create /init.scope control group
Host system: Debian bookworm, systemd v241 (default-hierarchy=hybrid) container (systemd compiled with default-hierarchy=unified) v247: works v248: works v249: works v250: fails, with the aforementioned error So something apparently regressed between v249 and v250. Am Mo., 17. Okt. 2022 um 01:38 Uhr schrieb Michael Biebl : > > What are you Missing? > > Lennart Poettering schrieb am So., 16. Okt. 2022, > 23:45: >> >> On So, 16.10.22 21:02, Michael Biebl (mbi...@gmail.com) wrote: >> >> > Am So., 16. Okt. 2022 um 16:23 Uhr schrieb Lennart Poettering >> > : >> > > >> > > On Fr, 14.10.22 22:57, Michael Biebl (mbi...@gmail.com) wrote: >> > > >> > > > Hi, >> > > > >> > > > since the issue came up on the Debian bug tracker at >> > > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1019147 , I figured >> > > > I ask here: >> > > >> > > Do you have any MACs in effect? >> > >> > No SELinux or Apparmor active >> > >> > > Does the host use cgroupsv2 or cgroupsv2 or hybrid? What is mounted to >> > > /sys/fs/cgroup and below? >> > >> > The host system uses systemd v241, compiled with default-hierarchy=hybrid >> > >> > >> > > Was the container configured to use either? >> > >> > The container uses systemd v251 with default-hierarchy=unified >> > >> > Trying to boot this container v251 container via systemd-nspawn leads to >> > >> > Welcome to Debian GNU/Linux bookworm/sid! >> > >> > Hostname set to . >> > Failed to create /init.scope control group: Operation not permitted >> > Failed to allocate manager object: Operation not permitted >> > [!!] Failed to allocate manager object. >> > Exiting PID 1... >> > Container test-bookworm failed with error code 255. >> >> Please answer the questions I asked, otherwise not actionable... >> >> Lennart >> >> -- >> Lennart Poettering, Berlin
Re: [systemd-devel] systemd-container: Trying to use a bookworm chroot with a buster host fails / Failed to create /init.scope control group
What are you Missing? Lennart Poettering schrieb am So., 16. Okt. 2022, 23:45: > On So, 16.10.22 21:02, Michael Biebl (mbi...@gmail.com) wrote: > > > Am So., 16. Okt. 2022 um 16:23 Uhr schrieb Lennart Poettering > > : > > > > > > On Fr, 14.10.22 22:57, Michael Biebl (mbi...@gmail.com) wrote: > > > > > > > Hi, > > > > > > > > since the issue came up on the Debian bug tracker at > > > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1019147 , I > figured > > > > I ask here: > > > > > > Do you have any MACs in effect? > > > > No SELinux or Apparmor active > > > > > Does the host use cgroupsv2 or cgroupsv2 or hybrid? What is mounted to > > > /sys/fs/cgroup and below? > > > > The host system uses systemd v241, compiled with default-hierarchy=hybrid > > > > > > > Was the container configured to use either? > > > > The container uses systemd v251 with default-hierarchy=unified > > > > Trying to boot this container v251 container via systemd-nspawn leads to > > > > Welcome to Debian GNU/Linux bookworm/sid! > > > > Hostname set to . > > Failed to create /init.scope control group: Operation not permitted > > Failed to allocate manager object: Operation not permitted > > [!!] Failed to allocate manager object. > > Exiting PID 1... > > Container test-bookworm failed with error code 255. > > Please answer the questions I asked, otherwise not actionable... > > Lennart > > -- > Lennart Poettering, Berlin >
Re: [systemd-devel] systemd-container: Trying to use a bookworm chroot with a buster host fails / Failed to create /init.scope control group
On So, 16.10.22 21:02, Michael Biebl (mbi...@gmail.com) wrote: > Am So., 16. Okt. 2022 um 16:23 Uhr schrieb Lennart Poettering > : > > > > On Fr, 14.10.22 22:57, Michael Biebl (mbi...@gmail.com) wrote: > > > > > Hi, > > > > > > since the issue came up on the Debian bug tracker at > > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1019147 , I figured > > > I ask here: > > > > Do you have any MACs in effect? > > No SELinux or Apparmor active > > > Does the host use cgroupsv2 or cgroupsv2 or hybrid? What is mounted to > > /sys/fs/cgroup and below? > > The host system uses systemd v241, compiled with default-hierarchy=hybrid > > > > Was the container configured to use either? > > The container uses systemd v251 with default-hierarchy=unified > > Trying to boot this container v251 container via systemd-nspawn leads to > > Welcome to Debian GNU/Linux bookworm/sid! > > Hostname set to . > Failed to create /init.scope control group: Operation not permitted > Failed to allocate manager object: Operation not permitted > [!!] Failed to allocate manager object. > Exiting PID 1... > Container test-bookworm failed with error code 255. Please answer the questions I asked, otherwise not actionable... Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] systemd-container: Trying to use a bookworm chroot with a buster host fails / Failed to create /init.scope control group
Am So., 16. Okt. 2022 um 16:23 Uhr schrieb Lennart Poettering : > > On Fr, 14.10.22 22:57, Michael Biebl (mbi...@gmail.com) wrote: > > > Hi, > > > > since the issue came up on the Debian bug tracker at > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1019147 , I figured > > I ask here: > > Do you have any MACs in effect? No SELinux or Apparmor active > Does the host use cgroupsv2 or cgroupsv2 or hybrid? What is mounted to > /sys/fs/cgroup and below? The host system uses systemd v241, compiled with default-hierarchy=hybrid > Was the container configured to use either? The container uses systemd v251 with default-hierarchy=unified Trying to boot this container v251 container via systemd-nspawn leads to Welcome to Debian GNU/Linux bookworm/sid! Hostname set to . Failed to create /init.scope control group: Operation not permitted Failed to allocate manager object: Operation not permitted [!!] Failed to allocate manager object. Exiting PID 1... Container test-bookworm failed with error code 255.
Re: [systemd-devel] systemd-container: Trying to use a bookworm chroot with a buster host fails / Failed to create /init.scope control group
On Fr, 14.10.22 22:57, Michael Biebl (mbi...@gmail.com) wrote: > Hi, > > since the issue came up on the Debian bug tracker at > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1019147 , I figured > I ask here: Do you have any MACs in effect? Does the host use cgroupsv2 or cgroupsv2 or hybrid? What is mounted to /sys/fs/cgroup and below? Was the container configured to use either? This is new payload on old host? if you force container into cgroupsv1 mode as the host (by adding systemd.unified_cgroup_hierarchy=no to the nspawn cmdline, does that work? Generally, systemd should discover everything on its own and just work when run in an older container manager/cgroup environment. But it's not something we would regularly test. Lennart -- Lennart Poettering, Berlin
[systemd-devel] systemd-container: Trying to use a bookworm chroot with a buster host fails / Failed to create /init.scope control group
Hi, since the issue came up on the Debian bug tracker at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1019147 , I figured I ask here: Am 04.09.22 um 18:40 schrieb Bernhard Übelacker: > > Package: systemd-container > Severity: wishlist > X-Debbugs-Cc: bernha...@mailbox.org > > > Dear Maintainer, > I tried to run on top of a buster system > with systemd-container 241-7~deb10u8 to start a container > with a current bookworm chroot with systemd-container 251.4-3. > This buster system was running linux-image 4.19.0-21-amd64. > > This failed with following error: > > root@debian:~# systemd-nspawn > --directory=/var/lib/machines/test-bookworm --boot --network-veth > Spawning container test-bookworm on /var/lib/machines/test-bookworm. > Press ^] three times within 1s to kill container. > systemd 251.4-3 running in system mode (+PAM +AUDIT +SELINUX > +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID > +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK > +PCRE2 -PWQUALITY -P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD > -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified) > Detected virtualization systemd-nspawn. > Detected architecture x86-64. > > Welcome to Debian GNU/Linux bookworm/sid! > > Hostname set to . > Failed to create /init.scope control group: Operation not permitted > Failed to allocate manager object: Operation not permitted > [!!] Failed to allocate manager object. > Exiting PID 1... > Container test-bookworm failed with error code 255. > > > So this report is mostly to ask if this expected or desired to work? Good question. Maybe raise that on the systemd-devel mailing list? Keep in mind, that in bullseye we switched to cgroupv2, i.e. we build systemd with -Ddefault-hierarchy=unified I'm honestly not sure which combination of versions (and cgroup layouts) are supported.
