Bug#1040622: systemd-sysv: reboot doesn't honor the grub-reboot settings; reboot -f does
Control: tags -1 -moreinfo Control: forcemerge 1039248 -1 On Sat, 08 Jul 2023 15:37:25 +0100 Luca Boccassi wrote: > Control: tags -1 moreinfo > > On Sat, 08 Jul 2023 00:16:28 -0400 "Theodore Y. Ts'o" > wrote: > > Package: systemd-sysv > > Version: 252.6-1 > > Severity: normal > > > > Dear Maintainer, > > > > * What led up to the situation? > > > > I was updating the gce-xfstests[1] test appliance to Debian Bookworm > from > > Debian Bullseye. > > > > [1] https://thunk.org/gce-xfstests > > > > * What exactly did you do (or not do) that was effective (or > > ineffective)? > > > > Unfortunately kexec has not been reliable ever since sometime after > the > > 5.4 kernel, at least on Google Compute Engine VM's. (About 30-40% of > > the time, the VM hangs after the kexec; about 10% of the time, the > > machine is up, but it is very slow and limping, and /proc/interrupts > > shows that some interrupt channel is going wild. This is no doubt > the > > kernel bug interacting with some virtual hardware in the GCE VM, but > > I've never been able to debug it.) > Check whether you have kexec-tools installed. It has some crufty old > and broken sysv-init script that it enables by default and messes > with > the reboot and silently makes it a kexec. I had the same issue and > disabling and masking kexec.service (which is autogenerated from the > crusty init script) fixed the problem for me. > > Nothing to do with systemd, which cannot 'bypass grub', once the > system > is rebooted, it's rebooted, control is given back to the kernel to do > what it's configured to do. Merging with #1039248 as I'm quite sure this is a problem with kexec- tool's old init script, as I had the same issue. Once the package provides a native and working unit file it should go away. -- Kind regards, Luca Boccassi signature.asc Description: This is a digitally signed message part
Bug#1040622: systemd-sysv: reboot doesn't honor the grub-reboot settings; reboot -f does
Control: tags -1 moreinfo On Sat, 08 Jul 2023 00:16:28 -0400 "Theodore Y. Ts'o" wrote: > Package: systemd-sysv > Version: 252.6-1 > Severity: normal > > Dear Maintainer, > > * What led up to the situation? > > I was updating the gce-xfstests[1] test appliance to Debian Bookworm from > Debian Bullseye. > > [1] https://thunk.org/gce-xfstests > > * What exactly did you do (or not do) that was effective (or > ineffective)? > > Unfortunately kexec has not been reliable ever since sometime after the > 5.4 kernel, at least on Google Compute Engine VM's. (About 30-40% of > the time, the VM hangs after the kexec; about 10% of the time, the > machine is up, but it is very slow and limping, and /proc/interrupts > shows that some interrupt channel is going wild. This is no doubt the > kernel bug interacting with some virtual hardware in the GCE VM, but > I've never been able to debug it.) > > Because of issues with kexec, the primary way that I reboot into the > kernel that I want to test is to install the kernel as a dpkg package, > and then examine /boot/grub/grub.cfg to find out where it was inserted > into the grub's menu listing, and then run a command like "grub- reboot > 1>4", where the number is found by examing the grub.cfg file. An > example of this works can be found here[2]. > > [2] https://github.com/tytso/xfstests-bld/blob/9bae3253d57456987d995cf85379e9165e054381/test-appliance/files/usr/local/lib/gce-load-kernel#L169 > > This works *just* *fine* when using Debian Bullseye (which is using > systemd 247.3-7+deb11u2). > > * What was the outcome of this action? > > Unfortunately, this no longer works in Debian Bookworm (with systemd > 252.6-1). In Debian Bookworm, the grub-reboot(8) setting is ignored > after triggering a reboot via /sbin/reboot. > > Connecting to the serial console, it appears that "reboot" is going > through some code path that does NOT involve triggering a BIOS message > and going through grub, where (assuming the GRUB_TIMEOUT is set to some > non-zero value like 15) the grub menu would be displayed, and after 15 > seconds, it would boot the kernel specified by grub-reboot, and then > clear the next_entry entry in /boot/grub/grubenv. > > Instead, systemd appears to boot the default kernel, ignoring > /boot/grub/grubenv, so I don't enter the kernel which I had just > installed, and had selected via the grub-reboot(8) command. Looking at > /boot/grub/grubenv, it still has the "next_entry=1>4" set by > grub-reboot(8), so it appears that /boot/grub/grubenv is being > completely ignored by reboot. Check whether you have kexec-tools installed. It has some crufty old and broken sysv-init script that it enables by default and messes with the reboot and silently makes it a kexec. I had the same issue and disabling and masking kexec.service (which is autogenerated from the crusty init script) fixed the problem for me. Nothing to do with systemd, which cannot 'bypass grub', once the system is rebooted, it's rebooted, control is given back to the kernel to do what it's configured to do. -- Kind regards, Luca Boccassi signature.asc Description: This is a digitally signed message part
Bug#1040622: systemd-sysv: reboot doesn't honor the grub-reboot settings; reboot -f does
Package: systemd-sysv Version: 252.6-1 Severity: normal Dear Maintainer, * What led up to the situation? I was updating the gce-xfstests[1] test appliance to Debian Bookworm from Debian Bullseye. [1] https://thunk.org/gce-xfstests * What exactly did you do (or not do) that was effective (or ineffective)? Unfortunately kexec has not been reliable ever since sometime after the 5.4 kernel, at least on Google Compute Engine VM's. (About 30-40% of the time, the VM hangs after the kexec; about 10% of the time, the machine is up, but it is very slow and limping, and /proc/interrupts shows that some interrupt channel is going wild. This is no doubt the kernel bug interacting with some virtual hardware in the GCE VM, but I've never been able to debug it.) Because of issues with kexec, the primary way that I reboot into the kernel that I want to test is to install the kernel as a dpkg package, and then examine /boot/grub/grub.cfg to find out where it was inserted into the grub's menu listing, and then run a command like "grub-reboot 1>4", where the number is found by examing the grub.cfg file. An example of this works can be found here[2]. [2] https://github.com/tytso/xfstests-bld/blob/9bae3253d57456987d995cf85379e9165e054381/test-appliance/files/usr/local/lib/gce-load-kernel#L169 This works *just* *fine* when using Debian Bullseye (which is using systemd 247.3-7+deb11u2). * What was the outcome of this action? Unfortunately, this no longer works in Debian Bookworm (with systemd 252.6-1). In Debian Bookworm, the grub-reboot(8) setting is ignored after triggering a reboot via /sbin/reboot. Connecting to the serial console, it appears that "reboot" is going through some code path that does NOT involve triggering a BIOS message and going through grub, where (assuming the GRUB_TIMEOUT is set to some non-zero value like 15) the grub menu would be displayed, and after 15 seconds, it would boot the kernel specified by grub-reboot, and then clear the next_entry entry in /boot/grub/grubenv. Instead, systemd appears to boot the default kernel, ignoring /boot/grub/grubenv, so I don't enter the kernel which I had just installed, and had selected via the grub-reboot(8) command. Looking at /boot/grub/grubenv, it still has the "next_entry=1>4" set by grub-reboot(8), so it appears that /boot/grub/grubenv is being completely ignored by reboot. *However*, it is properly handled if I use reboot -f. The "reboot -f" command will briefly show the BIOS version information, and then the Grub Menu, and then will boot the kernel selected by grub-reboot(8), and afterwords the next_entry field is cleared from /boot/grub/grubenv. So this is not a "grub" problem, but in the reboot path selected by systemd when doing a clean shutdown via the "reboot" command. As near as I can tell, grub is being bypassed, or at least grub is getting called in some way whic causes it to ignore the /boog/grub/grubenv parameter. In Debian Bookworm, "reboot" works like "reboot -f", in that we go through the BIOS initialization step, printing the BIOS version, and then grub is invoked in such a way that /boot/grub/grubenv is honored. * What outcome did you expect instead? The "reboot" command should go through the normal, full reboot sequence, such that grub-reboot(8) works correctly. As a workaround, I've replaced reboot sleep 60 with sync /boot sync sleep 1 reboot -f sleep 60 However, it would be desirable to let the system go through a full, clean shutdown, with file systems properly unmounted or remounted read-only in the case of the root file system. -- System Information: Debian Release: 12.0 APT prefers stable APT policy: (900, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 6.1.0-9-amd64 (SMP w/20 CPU threads; PREEMPT) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages systemd-sysv depends on: ii systemd 252.6-1 Versions of packages systemd-sysv recommends: ii libnss-systemd 252.6-1 ii libpam-systemd 252.6-1 systemd-sysv suggests no packages. -- no debconf information