Package: src:linux
Version: 6.1.192-1
Severity: critical
Tags: regression
Dear Maintainer,
After upgrading two bookworm VMs from linux-image-6.1.0-54-amd64
(6.1.190-1) to linux-image-6.1.0-55-amd64 (6.1.192-1), both hit the same
kernel oops a few minutes after boot:
BUG: kernel NULL pointer dereference (CR2: 0000000000000000)
RIP: 0010:ext4_mb_load_buddy_gfp+0xbc/0x380 [ext4]
Machine A (mail server, VMware ESXi guest, vmw_pvscsi, ext4 on /):
the oops hit PID 1 at 369 s uptime ->
"Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009"
Machine B (config management server, KVM guest, virtio, ext4 on /, 1 GB
RAM):
oops at 413 s uptime; the system did not panic but hung: tasks touching
the filesystem got stuck, and systemd-journald and systemd-timesyncd hit
their watchdog timeouts and could not be killed ("Processes still around
after final SIGKILL"). Load average went to ~14 per CPU with near-zero
iowait.
Both machines were rebooted into 6.1.0-54-amd64 (6.1.190-1) and are stable
there. A forced fsck (fsck.mode=force) on machine A found only journal
recovery and orphaned inodes from the crash, no bitmap or free-block errors,
so on-disk corruption does not seem to be the cause.
Three other bookworm VMs (3-4 GB RAM, mostly free) ran 6.1.0-55 for about
40 minutes without an oops before we rolled them back as a precaution.
Both affected machines were under memory pressure: B is a 1 GB VM that was
nearly out of available memory, and A is a busy mail server (Maildir) that
also uses swap.
Oops tail, machine A (console screenshot, transcribed):
RIP: 0010:ext4_mb_load_buddy_gfp+0xbc/0x380 [ext4]
Code: 89 44 24 08 f7 3c 24 48 63 d8 89 54 24 0c ba 01 00 00 00 48 89 de
e8 d3 0e ed f8 49 89 c7 48 3d 00 f0 ff ff 0f 87 28 01 00 00 <48> 8b
00 a8 04 0f 84 12 01 00 00 49 8b 07 a8 01 0f 85 07 01 00 00
RSP: 0018:ffffd1fbc0013618 EFLAGS: 00010207
RAX: 0000000000000000 RBX: 00000000000000c0 RCX: 0000000000000003
CR2: 0000000000000000
...
Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009
(the visible, unreliable "?" frames from the user task included
do_faccessat, filename_lookup, path_lookupat, terminate_walk,
__legitimize_path, try_to_unlazy, entry_SYSCALL_64_after_hwframe)
Oops tail, machine B (console screenshot, transcribed):
RIP: 0010:ext4_mb_load_buddy_gfp+0xbc/0x380 [ext4]
Code: 89 44 24 08 f7 3c 24 48 63 d8 89 54 24 0c ba 01 00 00 00 48 89 de
e8 d3 fe 38 d4 49 89 c7 48 3d 00 f0 ff ff 0f 87 28 01 00 00 <48> 8b
00 a8 04 0f 84 12 01 00 00 49 8b 07 a8 01 0f 85 07 01 00 00
RAX: 0000000000000000 RBX: 0000000000000102 RCX: 0000000000000003
CR2: 0000000000000000
A possible explanation, not verified against the source: the code bytes
around the faulting instruction look like
call <...> ; returns a page/folio pointer
mov %rax,%r15
cmp $0xfffffffffffff000,%rax
ja <error path> ; IS_ERR() check
mov (%rax),%rax ; <-- fault, RAX == 0
test $0x4,%al ; PG_uptodate
...
test $0x1,%al ; PG_locked
That is, the return value is checked with IS_ERR() but is NULL. In
mainline (6.3 and later), __filemap_get_folio() returns ERR_PTR(-ENOENT) on
a page-cache miss; in 6.1 it returns NULL. If a backported ext4 mballoc
change in 6.1.191/6.1.192 assumes the newer convention, a buddy/bitmap page
that is not in the page cache (more likely under memory pressure) would give
exactly this NULL dereference. That would also fit with only the
memory-constrained machines being hit.
We do not have the top of the oops (the "BUG:" line and the full Call
Trace) from either machine, because the journal was not flushed. We can
provide more information or test a candidate kernel on a non-critical VM.
Kernel 6.1.0-54-amd64 is held (apt-mark hold linux-image-amd64) on all
affected hosts for now.