Source: linux
Version: 6.1.192-1
Severity: grave

Hi,

after unattended-upgrades installed linux-image-6.1.0-55-cloud-amd64 (6.1.192-1)
and rebooted, my EC2 instance repeatedly hangs after boot. Ping still works,
but ssh and all services stop responding. With 6.1.0-54-cloud-amd64 (6.1.190-1)
the same system [ran for 19 days without problems / is stable since the 
downgrade].

Environment
- Debian 12 (bookworm), linux-image-6.1.0-55-cloud-amd64 6.1.192-1
- AWS EC2 t3a.nano and t3a.small (AMD EPYC 7571, KVM), same behaviour on both
- Root filesystem: ext4 on NVMe EBS volume (gp3), 40 GB, swapfile on root
- Services: MariaDB 10.11, postfix, courier-imap/pop3, apache2, rspamd, redis,
  slapd; kernel not tainted before the oops

Symptom
The system works for a few seconds to ~26 minutes after boot, then the
writeback worker oopses. After that all writers block (ssh, journald, ...).
On t3a.nano (512 MB RAM) the hang occurred within seconds of boot, on t3a.small
(2 GB RAM) after about 26 minutes. This suggests it is related to the buddy
cache pages not being in the page cache, but I have not verified this.

Oops (from the EC2 console log)

[ 1551.008730] BUG: kernel NULL pointer dereference, address: 0000000000000000
[ 1551.011768] #PF: supervisor read access in kernel mode
[ 1551.014014] #PF: error_code(0x0000) - not-present page
[ 1551.018730] Oops: 0000 [#1] PREEMPT SMP NOPTI
[ 1551.020592] CPU: 1 PID: 60 Comm: kworker/u4:3 Not tainted 
6.1.0-55-cloud-amd64 #1  Debian 6.1.192-1
[ 1551.024407] Hardware name: Amazon EC2 t3a.small/, BIOS 1.0 10/16/2017
[ 1551.027124] Workqueue: writeback wb_workfn (flush-259:0)
[ 1551.029361] RIP: 0010:ext4_mb_load_buddy_gfp+0xbc/0x380
[ 1551.031543] Code: 89 44 24 08 f7 3c 24 48 63 d8 89 54 24 0c ba 01 00 00 00 
48 89 de e8 33 d8 e3 ff 49 89 c7 48 3d 00 f0 ff ff 0f 87 28 01 00 00 <48> 8b 00 
a8 04 0f 84 12 01 00 00 49 8b 07 a8 01 0f 85 07 01 00 00
[ 1551.039575] RSP: 0018:ffffcee38080f670 EFLAGS: 00010207
[ 1551.041787] RAX: 0000000000000000 RBX: 0000000000000062 RCX: 0000000000000003
[ 1551.044798] RDX: 0000000000000022 RSI: ffff89fa01da4490 RDI: ffff89fa02390000
[ 1551.072332] Call Trace:
[ 1551.075502]  <TASK>
[ 1551.078533]  ext4_mb_regular_allocator+0x29e/0xce0
[ 1551.082719]  ext4_mb_new_blocks+0x820/0x1240
[ 1551.094357]  ext4_ext_map_blocks+0x3c7/0x1990
[ 1551.117780]  ext4_map_blocks+0x1b6/0x5f0
[ 1551.121457]  ext4_writepages+0x88a/0x1190
[ 1551.141958]  do_writepages+0xc0/0x1c0
[ 1551.146555]  __writeback_single_inode+0x3d/0x360
[ 1551.157427]  writeback_sb_inodes+0x1f5/0x4d0
[ 1551.172494]  wb_writeback+0x204/0x2f0
[ 1551.176287]  wb_workfn+0x2aa/0x4f0
[ 1551.188065]  process_one_work+0x1c7/0x380
[ 1551.192405]  worker_thread+0x4d/0x380
[ 1551.196165]  kthread+0xda/0x100
[ 1551.207703]  ret_from_fork+0x22/0x30
[ 1551.214325] Modules linked in: binfmt_misc intel_rapl_msr intel_rapl_common 
iosf_mbi crct10dif_pclmul crc32_pclmul ghash_clmulni_intel nls_ascii nls_cp437 
sha512_ssse3 squashfs sha512_generic vfat fat sha256_ssse3 sha1_ssse3 
aesni_intel crypto_simd cryptd button evdev serio_raw nf_tables libcrc32c 
nfnetlink loop fuse efi_pstore dm_mod configfs ip_tables x_tables autofs4 ena 
crc32c_intel
[ 1551.312523] note: kworker/u4:3[60] exited with irqs disabled
[ 1551.321835] WARNING: CPU: 1 PID: 60 at kernel/exit.c:823 do_exit+0x904/0xb10

Possibly related
A similar oops in ext4_mb_load_buddy_gfp was reported for the 6.1.167-rc1
stable review (bisected to "ext4: convert bd_bitmap_page to bd_bitmap_folio",
commit 29897d75d6491ffe23cdc9d96caba9282a20dfc3):
https://lore.kernel.org/stable/ (thread "[PATCH 6.1 000/481] 6.1.167-rc1 
review")
I do not know whether it is the same issue.

Workaround
Booting 6.1.0-54-cloud-amd64 (6.1.190-1) and holding linux-image-cloud-amd64.

I can provide more information or test patches if needed.

Regards,
Thorsten

Reply via email to