[Touch-packages] [Bug 2016908] Re: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs
** Description changed: I'm assuming the image being used for these deploys is 20230417 or 20230417.1 based on the fact that I saw a 6.2 kernel being used which I don't believe was part of the 20230319 serial. I don't have access to the maas server, so I can't directly check any log files. MAAS Version: 3.3.2 Here's where the serial log indicates it can't download the squashfs. The full log is attached as scobee-lunar-no-squashfs.log (there are some other console message intermixed): no search or nameservers found in /run/net-BOOTIF.conf /run/net-*.conf /run/net6 -*.conf :: root=squash:http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.04/lunar/candi date/squa[ 206.804704] Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity =yes shfs :: mount_squash downloading http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.0 4/lunar/candidate/squashfs to /root.tmp.img Connecting to 10.229.32.21:5248 (10.229.32.21:5248) wget: can't connect to remote host (10.229.32.21): Network is unreachable :: mount -t squashfs -o loop '/root.tmp.img' '/root.tmp' mount: mounting /root.tmp.img on /root.tmp failed: No such file or directory done. + Still gathering logs and info and will update as I go. - Still gathering logs and info and will update as I go. + + + Kernel Bug / Apparmor + reproducer + + $ wget https://images.maas.io/ephemeral-v3/candidate/lunar/amd64/20230419/ga-23.04/generic/boot-kernel + $ wget https://images.maas.io/ephemeral-v3/candidate/lunar/amd64/20230419/ga-23.04/generic/boot-initrd + $ qemu-system-x86_64 -nographic -m 2G -kernel ./boot-kernel -initrd ./boot-initrd -append 'console=ttyS0 break=modules apparmor=0' + + + #start the VM + + Starting systemd-udevd version 252.5-2ubuntu3 + Spawning shell within the initramfs + + + BusyBox v1.35.0 (Ubuntu 1:1.35.0-4ubuntu1) built-in shell (ash) + Enter 'help' for a list of built-in commands. + + (initramfs) udevadm info --export-db + Failed to set death signal: Invalid argument + + Observe that udevadm fails to setup death signal, with in systemd code + is this + + https://github.com/systemd/systemd/blob/08c2f9c626e0f0052d505b1b7e52f335c0fbfa1d/src/basic/process- + util.c#L1252 + + if (flags & (FORK_DEATHSIG|FORK_DEATHSIG_SIGINT)) + if (prctl(PR_SET_PDEATHSIG, (flags & FORK_DEATHSIG_SIGINT) ? SIGINT : SIGTERM) < 0) { + log_full_errno(prio, errno, "Failed to set death signal: %m"); + _exit(EXIT_FAILURE); + } + + + + MAAS bug + Why is maas setting `apparmor=0` ? Ubuntu shouldn't be used without apparmor. Even for deployment and commisioning. ** Changed in: linux (Ubuntu) Status: Incomplete => Triaged ** Changed in: maas-images Status: Incomplete => Invalid ** Changed in: systemd (Ubuntu) Status: New => Invalid ** Also affects: maas Importance: Undecided Status: New ** Summary changed: - Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs + udev fails to make prctl() syscall with apparmor=0 (as used by maas by default) ** Description changed: I'm assuming the image being used for these deploys is 20230417 or 20230417.1 based on the fact that I saw a 6.2 kernel being used which I don't believe was part of the 20230319 serial. I don't have access to the maas server, so I can't directly check any log files. MAAS Version: 3.3.2 Here's where the serial log indicates it can't download the squashfs. The full log is attached as scobee-lunar-no-squashfs.log (there are some other console message intermixed): no search or nameservers found in /run/net-BOOTIF.conf /run/net-*.conf /run/net6 -*.conf :: root=squash:http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.04/lunar/candi date/squa[ 206.804704] Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity =yes shfs :: mount_squash downloading http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.0 4/lunar/candidate/squashfs to /root.tmp.img Connecting to 10.229.32.21:5248 (10.229.32.21:5248) wget: can't connect to remote host (10.229.32.21): Network is unreachable :: mount -t squashfs -o loop '/root.tmp.img' '/root.tmp' mount: mounting /root.tmp.img on /root.tmp failed: No such file or directory done. Still gathering logs and info and will update as I go. - Kernel Bug / Apparmor reproducer $ wget https://images.maas.io/ephemeral-v3/candidate/lunar/amd64/20230419/ga-23.04/generic/boot-kernel $ wget https://images.maas.io/ephemeral-v3/candidate/lunar/amd64/20230419/ga-23.04/generic/boot-initrd $ qemu-system-x86_64 -nographic -m 2G -kernel ./boot-kernel -initrd ./boot-initrd -append 'console=ttyS0 break=modules apparmor=0' - #start the VM Starting systemd-udevd version 252.5-2ubuntu3 Spawning shell within the initramfs - BusyBox v1.35.0 (Ubuntu 1:1.35.0-4ubuntu1) buil
[Touch-packages] [Bug 2016908] Re: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs
horay i managed to reproduce it locally. ** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2016908 Title: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs Status in maas-images: Incomplete Status in linux package in Ubuntu: New Status in systemd package in Ubuntu: New Bug description: I'm assuming the image being used for these deploys is 20230417 or 20230417.1 based on the fact that I saw a 6.2 kernel being used which I don't believe was part of the 20230319 serial. I don't have access to the maas server, so I can't directly check any log files. MAAS Version: 3.3.2 Here's where the serial log indicates it can't download the squashfs. The full log is attached as scobee-lunar-no-squashfs.log (there are some other console message intermixed): no search or nameservers found in /run/net-BOOTIF.conf /run/net-*.conf /run/net6 -*.conf :: root=squash:http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.04/lunar/candi date/squa[ 206.804704] Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity =yes shfs :: mount_squash downloading http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.0 4/lunar/candidate/squashfs to /root.tmp.img Connecting to 10.229.32.21:5248 (10.229.32.21:5248) wget: can't connect to remote host (10.229.32.21): Network is unreachable :: mount -t squashfs -o loop '/root.tmp.img' '/root.tmp' mount: mounting /root.tmp.img on /root.tmp failed: No such file or directory done. Still gathering logs and info and will update as I go. To manage notifications about this bug go to: https://bugs.launchpad.net/maas-images/+bug/2016908/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2016908] Re: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs
We discussed fixes to the maas bootloader builds and as a result, we have a new maas bootloader which should have the latest shim and grub: http://images.maas.io/ephemeral-v3/candidate/bootloaders/uefi/amd64/20230420.0/ I manually copied these to my test maas, it did not resolve the issue (or have any other visible change). Paolo built a kernel to try and debug this from the prctl interface. With this (and udev with -debug), I see these two additional console log entries: [ 18.029479] __do_sys_prctl::2375 [ 18.042840] __do_sys_prctl::2380 There are lots of these, but only the same two lines. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2016908 Title: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs Status in maas-images: Incomplete Status in systemd package in Ubuntu: New Bug description: I'm assuming the image being used for these deploys is 20230417 or 20230417.1 based on the fact that I saw a 6.2 kernel being used which I don't believe was part of the 20230319 serial. I don't have access to the maas server, so I can't directly check any log files. MAAS Version: 3.3.2 Here's where the serial log indicates it can't download the squashfs. The full log is attached as scobee-lunar-no-squashfs.log (there are some other console message intermixed): no search or nameservers found in /run/net-BOOTIF.conf /run/net-*.conf /run/net6 -*.conf :: root=squash:http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.04/lunar/candi date/squa[ 206.804704] Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity =yes shfs :: mount_squash downloading http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.0 4/lunar/candidate/squashfs to /root.tmp.img Connecting to 10.229.32.21:5248 (10.229.32.21:5248) wget: can't connect to remote host (10.229.32.21): Network is unreachable :: mount -t squashfs -o loop '/root.tmp.img' '/root.tmp' mount: mounting /root.tmp.img on /root.tmp failed: No such file or directory done. Still gathering logs and info and will update as I go. To manage notifications about this bug go to: https://bugs.launchpad.net/maas-images/+bug/2016908/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2016908] Re: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs
I am annoyed that i cannot reproduce this locally outside of MAAS. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2016908 Title: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs Status in maas-images: Incomplete Status in systemd package in Ubuntu: New Bug description: I'm assuming the image being used for these deploys is 20230417 or 20230417.1 based on the fact that I saw a 6.2 kernel being used which I don't believe was part of the 20230319 serial. I don't have access to the maas server, so I can't directly check any log files. MAAS Version: 3.3.2 Here's where the serial log indicates it can't download the squashfs. The full log is attached as scobee-lunar-no-squashfs.log (there are some other console message intermixed): no search or nameservers found in /run/net-BOOTIF.conf /run/net-*.conf /run/net6 -*.conf :: root=squash:http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.04/lunar/candi date/squa[ 206.804704] Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity =yes shfs :: mount_squash downloading http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.0 4/lunar/candidate/squashfs to /root.tmp.img Connecting to 10.229.32.21:5248 (10.229.32.21:5248) wget: can't connect to remote host (10.229.32.21): Network is unreachable :: mount -t squashfs -o loop '/root.tmp.img' '/root.tmp' mount: mounting /root.tmp.img on /root.tmp failed: No such file or directory done. Still gathering logs and info and will update as I go. To manage notifications about this bug go to: https://bugs.launchpad.net/maas-images/+bug/2016908/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2016908] Re: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs
Kernel side: kernel/sys.c::SYSCALL_DEFINE5(prctl, int, option,...): { struct task_struct *me = current; unsigned char comm[sizeof(me->comm)]; long error; error = security_task_prctl(option, arg2, arg3, arg4, arg5); if (error != -ENOSYS) return error; error = 0; switch (option) { case PR_SET_PDEATHSIG: if (!valid_signal(arg2)) { error = -EINVAL; break; } me->pdeath_signal = arg2; break; ... return error; } and include/linux/signal.h::valid_signal(): /* Test if 'sig' is valid signal. Use this instead of testing _NSIG directly */ static inline int valid_signal(unsigned long sig) { return sig <= _NSIG ? 1 : 0; } and arch/x86/include/asm/signal.h:#define _NSIG 64 I wonder about security_task_prctl() though. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2016908 Title: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs Status in maas-images: Incomplete Status in systemd package in Ubuntu: New Bug description: I'm assuming the image being used for these deploys is 20230417 or 20230417.1 based on the fact that I saw a 6.2 kernel being used which I don't believe was part of the 20230319 serial. I don't have access to the maas server, so I can't directly check any log files. MAAS Version: 3.3.2 Here's where the serial log indicates it can't download the squashfs. The full log is attached as scobee-lunar-no-squashfs.log (there are some other console message intermixed): no search or nameservers found in /run/net-BOOTIF.conf /run/net-*.conf /run/net6 -*.conf :: root=squash:http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.04/lunar/candi date/squa[ 206.804704] Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity =yes shfs :: mount_squash downloading http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.0 4/lunar/candidate/squashfs to /root.tmp.img Connecting to 10.229.32.21:5248 (10.229.32.21:5248) wget: can't connect to remote host (10.229.32.21): Network is unreachable :: mount -t squashfs -o loop '/root.tmp.img' '/root.tmp' mount: mounting /root.tmp.img on /root.tmp failed: No such file or directory done. Still gathering logs and info and will update as I go. To manage notifications about this bug go to: https://bugs.launchpad.net/maas-images/+bug/2016908/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2016908] Re: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs
** Also affects: systemd (Ubuntu) Importance: Undecided Status: New ** Attachment added: "lunar-udev-debug.log" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2016908/+attachment/5665324/+files/lunar-udev-debug.log -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2016908 Title: Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs Status in maas-images: Incomplete Status in systemd package in Ubuntu: New Bug description: I'm assuming the image being used for these deploys is 20230417 or 20230417.1 based on the fact that I saw a 6.2 kernel being used which I don't believe was part of the 20230319 serial. I don't have access to the maas server, so I can't directly check any log files. MAAS Version: 3.3.2 Here's where the serial log indicates it can't download the squashfs. The full log is attached as scobee-lunar-no-squashfs.log (there are some other console message intermixed): no search or nameservers found in /run/net-BOOTIF.conf /run/net-*.conf /run/net6 -*.conf :: root=squash:http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.04/lunar/candi date/squa[ 206.804704] Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity =yes shfs :: mount_squash downloading http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.0 4/lunar/candidate/squashfs to /root.tmp.img Connecting to 10.229.32.21:5248 (10.229.32.21:5248) wget: can't connect to remote host (10.229.32.21): Network is unreachable :: mount -t squashfs -o loop '/root.tmp.img' '/root.tmp' mount: mounting /root.tmp.img on /root.tmp failed: No such file or directory done. Still gathering logs and info and will update as I go. To manage notifications about this bug go to: https://bugs.launchpad.net/maas-images/+bug/2016908/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp