Control: tags -1 + moreinfo HI Andy,
On Wed, Aug 05, 2026 at 12:42:03PM +0100, Andy Simpkins wrote: > Package: src:linux > Version: 6.12.96-1 > Severity: critical > Justification: breaks the whole system > X-Debbugs-Cc: [email protected], [email protected] > User: [email protected] > Usertags: amd64 kernel > > Dear Maintainer, > > *** Reporter, please consider answering these questions, where appropriate *** > > * What led up to the situation? > My server failed to restert after a dist-upgrade on 2026-08-01 > the only update was the kernel that went from 6.12.88+deb13-amd64 to > 6.12.100+deb13-amd64 > packages had previously been updated... > > > Debugging so far: > 2026-08-01 > > (1) My server failed to restert after a dist-upgrade on 2026-08-01 > the only update was the kernel that went from 6.12.88+deb13-amd64 to > 6.12.100+deb13-amd64 > > Watching the system boot on the local screen showed: > | Loading Linux 6.12.100+deb13-amd64 ... > | Loading initial ramdisk ... > > and nothing else. > > (2) Breaking into the grub menu and removing the qiate option yeilds a > little more information (but not much): > | Loading Linux 6.12.100+deb13-amd64 ... > | Loading initial ramdisk ... > | > | > | EFI stub: Loaded initrd from > LINUX_EFI_INITRD_MEDIA_GUID d > | evice path > | EFI stub: Measured initrd data into PCR 9 > > and nothing else. > > (3) Initial debugging: > - Confirmed that I could still boot the machine with the old kernel > 6.12.88+deb13-amd64 > - Checked that /boot had enough space > - Went to google. Actually not mush referanced EFI stub: Measured > initrd data into PCR 9 apart from > the usual "turn off secure boot" (on this server that isn't turned > on - bad me) > > (4) Start looking for where the fault first occured > - between 6.12.88+deb13 and 6.12.100+deb13 Debian stable has only had > .90 .94 .95 and .96 kernels > - testing them .95 worked and .96 yeilded the same lock up on boot > as .100 > > i So the problem first appers in the .96 kernel and is not in the .95 > kernel > This is a good place to report a bug, but can I (with a lot of help) > track down this still further > > > 2026-08-02 > > i up until now I have been walking to my garage where the server is > located and standing in front of a rack > with a monitor and keyboard plugged into the machine. However this > machine supports IPMI so I spent a little time getting that up and running so > that I can continue from the relative comfort of my desk (with lights, a > chair and not needing to hold the keyboard with one hand) > > > (5) Sledge had mentioned this problem in irc/#debain-kernal where iam_tj > suggested that we try appending > 'debug earlycon=efifb' to the kernal command line. This yielded 15 > seconds worth of messages before the system locked up the last few messages > being (vmlinuz-6.12.96+deb13-amd64): > | [ 14.663477] RCU Tasks: Setting shift to 5 and lim to > 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20. > | [ 14.750474] RCU Tasks Rude: Setting shift to 5 and > lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20. > | [ 14.838024] RCU Tasks Trace: Setting shift to 5 and > lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20. > | [ 14.929752] NR_IRQS: 524544, nr_irqs: 584, > preallocated irqs: 16 > | [ 15.016814] rcu: srcu_init: Setting srcu_struct > sizes based on contention. > | [ 15.104011] Console: colour dummy device 80x25 > | [ 15.191236] printk: legacy console [tty0] enabled > | [ 15.278249] printk: legacy bootconsole [efifb0] > disabled > > booting the working kernel with the same kernal options yeilds > the SAME messages with slightly differing times, but then continues to login > prompt: > | [ 14.697466] RCU Tasks: Setting shift to 5 and lim to > 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20. > | [ 14.784936] RCU Tasks Rude: Setting shift to 5 and > lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20. > | [ 14.872067] RCU Tasks Trace: Setting shift to 5 and > lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20. > | [ 14.964000] NR_IRQS: 524544, nr_irqs: 584, > preallocated irqs: 16 > | [ 15.051482] rcu: srcu_init: Setting srcu_struct > sizes based on contention. > | [ 15.138589] Console: colour dummy device 80x25 > | [ 15.226079] printk: legacy console [tty0] enabled > | [ 15.313751] printk: legacy bootconsole [efifb0] > disabled > | [ 15.400831] ACPI: Core revision 20240827 > | [ 15.401415] clocksource: hpet: mask: 0xffffffff > max_cycles: 0xffffffff, max_idle_ns: 79635855245 ns > | [ 15.401464] APIC: Switch to symmetric I/O mode setup > | ... and so on > > 2026-08-05 > > (6) iam_tj also suggested adding keep_bootcon - with 'debug earlycon=efifb > keep_bootcon' on vmlinuz-6.12.96+deb13-amd64 resulted in the same screen as > step (2) > > re-reunning steps 1-6 again with a witness (I failed to get .96 to > display any debug first time round this morning) > > .95 boots ok > > .96 with 'debug earlycon=efifb keep_bootcon' the kernal command line: > yes we get a LOT further - and we see a crash / traceback > | [ 34.285342] BUG: kernel NULL pointer dereference, > address: 0000000000000000 As you seem to have a clear indication now when system is not working, would you be willing to do a bisect of the kernel to identify the breaking changes? If you need some instructions they would be: git clone --single-branch -b linux-6.12.y https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git cd linux-stable git checkout v6.12.95 cp /boot/config-$(uname -r) .config yes '' | make localmodconfig make savedefconfig mv defconfig arch/x86/configs/my_defconfig # test 6.12.95 to ensure this is "good" make my_defconfig make -j $(nproc) bindeb-pkg ... install the resulting .deb package and confirm it successfully boots # test 6.12.96 to ensure this is "bad" git checkout v6.12.96 make my_defconfig make -j $(nproc) bindeb-pkg ... install the resulting .deb package and confirm it fails to boot and triggers the NULL pointer dereference. With that confirmed, the bisection can start: git bisect start git bisect good v6.12.95 git bisect bad v6.12.96 In each bisection step git checks out a state between the oldest known-bad and the newest known-good commit. In each step test using: make my_defconfig make -j $(nproc) bindeb-pkg ... install, try to boot and verify if problem exists. and if the problem is hit run: git bisect bad and if the problem doesn't trigger run: git bisect good . Please pay attention to always select the just built kernel for booting, it won't always be the default kernel picked up by grub. Iterate until git announces to have identified the first bad commit. Then provide the output of git bisect log In the course of the bisection you might have to uninstall previous kernels again to not exhaust the disk space in /boot. Also in the end uninstall all self-built kernels again. That could help narrowing down towards the issue. Additionally could you temporarily test as well 7.1.6-1 from unstable on that system? Does 7.1.6 has the same problem? Regards, Salvatore
