Public bug reported:
SUMMARY
=======
Since upgrading from Ubuntu 24.04 (Linux 6.8) to Ubuntu 26.04 (Linux 7.0),
this machine panics roughly once a week. The panic is always a NULL pointer
dereference in mempool_free_bulk(), reached from crypt_endio() in dm_crypt
via bio_put()/bio_free(), running in softirq context from a SCSI completion.
It reproduces during teardown of a dm-crypt device: a late bio completion
arrives after the dm-crypt target's bio_set has already been destroyed, so
bio_free() returns the bio to a mempool whose ->elements is NULL.
The system was stable on 6.8 for approximately two years with the identical
storage configuration and the identical backup script. Nothing in userspace
changed at the time the crashes began.
ENVIRONMENT
===========
Distribution: Ubuntu 26.04 LTS
Kernel: 7.0.0-31-generic #31-Ubuntu PREEMPT(lazy) x86_64
Hardware: Supermicro X13SAE-F, BIOS 5.3 06/04/2026
Memory: 64 GB ECC (ie31200_edac; 0 CE, 0 UE reported)
DKMS modules: none
Taint: G W -- the W is an unrelated boot-time WARN in i915
(print_ddi_port, intel_bios.c VBT parsing, headless system).
No out-of-tree modules are loaded.
STORAGE STACK
=============
Four backup volumes, each:
iSCSI (iscsi_tcp, Synology target over TCP/IP)
-> sd (SCSI disk)
-> dm-crypt
-> btrfs (zstd:3, blake2b csums, async discard enabled)
Mounted together nightly by a backup script, then unmounted and torn down
(umount -> cryptsetup close -> iscsiadm logout) when the backup finishes.
Separate from this, the system also uses hardware RAID -> bcache -> btrfs
for primary storage and md RAID1 for SSD storage. Neither appears in any
crash trace.
TRIGGER
=======
Every crash occurs during teardown of the backup volumes, within seconds of
unmount. The preceding log is consistent across occurrences:
[256109.45] BTRFS info (device dm-15): last unmount of filesystem ...
[256109.53] sd 9:0:0:1: [sdb] Synchronizing SCSI cache
[256109.75] BTRFS info (device dm-16): last unmount of filesystem ...
[256109.96] sd 10:0:0:1: [sdc] Synchronizing SCSI cache
[256110.06] BTRFS info (device dm-17): last unmount of filesystem ...
[256110.17] sd 11:0:0:1: [sdd] Synchronizing SCSI cache
[256111.50] BTRFS info (device dm-18): last unmount of filesystem ...
[256115.15] BTRFS warning (device dm-18): folio private not zero on folio
906100736
[~100 more identical warnings for consecutive folios]
[256115.18] BUG: kernel NULL pointer dereference, address: 0000000000000000
The "folio private not zero" flood immediately before the oops indicates
btrfs is releasing folios that still carry I/O state, i.e. the unmount is
completing while work is still outstanding beneath it.
OOPS
====
BUG: kernel NULL pointer dereference, address: 0000000000000000
#PF: supervisor write access in kernel mode
#PF: error_code(0x0002) - not-present page
PGD 0 P4D 0
Oops: Oops: 0002 [#1] SMP NOPTI
CPU: 9 UID: 0 PID: 0 Comm: swapper/9 Tainted: G W
7.0.0-31-generic #31-Ubuntu PREEMPT(lazy)
Tainted: [W]=WARN
Hardware name: Supermicro Super Server/X13SAE-F, BIOS 5.3 06/04/2026
RIP: 0010:mempool_free_bulk+0x140/0x180
Code: 8b 7d f8 48 8b 75 f0 8b 4f 08 85 c9 75 40 8b 57 04 85 d2 78 46 48 8b
0e 48 8b 57 10 48 89 c6 48 89 7d f8 c7 47 08 01 00 00 00 <48> 89 0a e8 48 99
fa 00 48 8b 7d f8 41 b8 01 00 00 00 e9 6a ff ff
RSP: 0018:ffffcef780780d20 EFLAGS: 00010046
RAX: 0000000000000246 RBX: ffff8b34577609e0 RCX: ffff8b3d1f9aa700
RDX: 0000000000000000 RSI: 0000000000000246 RDI: ffff8b34577609e0
RBP: ffffcef780780d38 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8b3d1f9aa700
R13: 0000000000000000 R14: ffff8b33e3ab86c0 R15: 0000000000004000
CR2: 0000000000000000 CR3: 0000000746c44006 CR4: 0000000000f72ef0
Call Trace:
<IRQ>
? netif_receive_skb_list_internal+0x1ec/0x310
mempool_free+0x30/0x60
bio_free+0x4a/0x60
bio_put+0x89/0x170
? napi_complete_done+0x81/0x200
crypt_endio+0x6c/0xd5 [dm_crypt]
bio_endio+0x15e/0x210
blk_update_request+0x134/0x4b0
? skb_defer_free_flush+0xa4/0xd0
scsi_end_request+0x2c/0x1c0
scsi_io_completion+0x4f/0x380
scsi_finish_command+0xc3/0x110
scsi_complete+0x81/0x1e0
blk_done_softirq+0x4d/0x70
handle_softirqs+0xe1/0x360
? handle_irq_event+0x53/0x80
__irq_exit_rcu+0x100/0x120
irq_exit_rcu+0xe/0x20
common_interrupt+0xb5/0xe0
</IRQ>
<TASK>
asm_common_interrupt+0x27/0x40
RIP: 0010:cpuidle_enter_state+0xca/0x700
cpuidle_enter+0x30/0x50
call_cpuidle+0x21/0x50
cpuidle_idle_call+0x16b/0x1f0
do_idle+0x94/0xf0
cpu_startup_entry+0x29/0x30
start_secondary+0x125/0x180
common_startup_64+0x13e/0x141
</TASK>
CR2: 0000000000000000
---[ end trace 0000000000000000 ]---
(Full module list and complete log available in the attached pstore
dump.)
ANALYSIS
========
Decoding the faulting instruction sequence from the Code: bytes, with
RDI = the mempool:
48 8b 7d f8 mov -0x8(%rbp),%rdi
48 8b 75 f0 mov -0x10(%rbp),%rsi
8b 4f 08 mov 0x8(%rdi),%ecx ; pool->curr_nr
85 c9 / 75 40 test %ecx,%ecx / jne ; taken only if curr_nr == 0
8b 57 04 mov 0x4(%rdi),%edx ; pool->min_nr
85 d2 / 78 46 test %edx,%edx / js
48 8b 0e mov (%rsi),%rcx ; element
48 8b 57 10 mov 0x10(%rdi),%rdx ; pool->elements -> NULL
c7 47 08 01 00 00 00 movl $0x1,0x8(%rdi) ; pool->curr_nr = 1
48 89 0a mov %rcx,(%rdx) ; elements[0] = element <--
FAULT
RDX is 0 at the fault, and CR2 is 0. So at the time of the free:
pool->curr_nr == 0
pool->elements == NULL
That is the state left behind by mempool_exit(), which frees the element
array and sets ->elements to NULL. The bio_set backing this dm-crypt target
has already been through bioset_exit(), yet a bio belonging to it is still
being completed.
The completion path is entirely in softirq context off a SCSI command
completion, on an otherwise idle CPU (PID 0, swapper/9), so the oops is
fatal rather than recoverable.
This is the same bug class as several previously fixed races, all with the
identical pool->elements == NULL signature:
d208b89401e0 dm: fix mempool NULL pointer race when completing IO
9f6dc633761006f974701d4c88da71ab68670749
dm: interlock pending dm_io and dm_wait_for_bios_completion
341097ee53573 md: fix a crash in mempool_free
Those specific instances were fixed years ago and are present in 7.0. The
path here is different: bio_free() -> mempool_free() -> mempool_free_bulk()
reached from dm-crypt's own bio_set via crypt_endio(), during target
destruction rather than table swap.
ADDITIONAL DATA POINT
=====================
On a night when the machine did not panic, the same teardown window produced
a hang instead. Multiple btrfs workers blocked for over 122 seconds, all
waiting in the same place:
INFO: task btrfs-transacti:1138929 blocked for more than 122 seconds.
...
read_extent_buffer_pages+0x40/0x80 [btrfs]
btrfs_read_extent_buffer+0x78/0xc0 [btrfs]
read_block_for_search+0x21f/0x3f0 [btrfs]
btrfs_search_slot+0x37e/0x1060 [btrfs]
btrfs_lookup_csum+0x6a/0x170 [btrfs]
btrfs_csum_file_blocks+0x1b3/0x7b0 [btrfs]
btrfs_finish_one_ordered+0x595/0xc00 [btrfs]
btrfs_finish_ordered_io+0x45/0x60 [btrfs]
finish_ordered_fn+0x15/0x20 [btrfs]
btrfs_work_helper+0xc7/0x210 [btrfs]
Same window, same stack, non-fatal outcome. This suggests I/O outstanding at
teardown time is the common factor, with the race either stalling or
crashing depending on timing.
WORKAROUND APPLIED
==================
Serializing the teardown (fully draining and closing each dm-crypt device
before touching the next, and deferring iSCSI logout until all dm-crypt
devices are closed), plus mounting the btrfs volumes with nodiscard instead
of async discard. Effectiveness not yet confirmed; the crash interval is
roughly one week.
NOT YET TESTED
==============
Whether this reproduces on mainline. This is a production machine and I
cannot easily run a test kernel on it, but I can collect further data via
pstore/ERST, which is working reliably on this hardware
ProblemType: Bug
DistroRelease: Ubuntu 26.04
Package: linux-image-7.0.0-31-generic 7.0.0-31.31
ProcVersionSignature: Ubuntu 7.0.0-31.31-generic 7.0.14
Uname: Linux 7.0.0-31-generic x86_64
AlsaVersion: Advanced Linux Sound Architecture Driver Version k7.0.0-31-generic.
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.34.1-0ubuntu0.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path',
'/dev/snd/controlC0', '/dev/snd/hwC0D2', '/dev/snd/hwC0D0',
'/dev/snd/pcmC0D9p', '/dev/snd/pcmC0D8p', '/dev/snd/pcmC0D7p',
'/dev/snd/pcmC0D3p', '/dev/snd/pcmC0D2c', '/dev/snd/pcmC0D1p',
'/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/seq', '/dev/snd/timer']
failed with exit code 1:
CRDA: N/A
Card0.Amixer.info: Error: [Errno 2] No such file or directory: 'amixer'
Card0.Amixer.values: Error: [Errno 2] No such file or directory: 'amixer'
CasperMD5CheckResult: pass
Date: Wed Sep 16 16:01:04 2026
InstallationDate: Installed on 2025-04-22 (512 days ago)
InstallationMedia: Ubuntu-Server 24.04.2 LTS "Noble Numbat" - Release amd64
(20250216.2)
MachineType: Supermicro Super Server
ProcEnviron:
LANG=en_US.UTF-8
PATH=(custom, no user)
SHELL=/bin/bash
TERM=xterm-256color
XDG_RUNTIME_DIR=<set>
ProcFB: 0 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-7.0.0-31-generic
root=/dev/mapper/ubuntu--vg-ubuntu--lv ro
crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: Upgraded to resolute on 2026-06-07 (101 days ago)
dmi.bios.date: 06/04/2026
dmi.bios.release: 5.32
dmi.bios.vendor: American Megatrends International, LLC.
dmi.bios.version: 5.3
dmi.board.asset.tag: Default string
dmi.board.name: X13SAE-F
dmi.board.vendor: Supermicro
dmi.board.version: 1.02B
dmi.chassis.asset.tag: Chassis Asset Tag
dmi.chassis.type: 17
dmi.chassis.vendor: Supermicro
dmi.chassis.version: 0123456789
dmi.modalias:
dmi:bvnAmericanMegatrendsInternational,LLC.:bvr5.3:bd06/04/2026:br5.32:svnSupermicro:pnSuperServer:pvr0123456789:rvnSupermicro:rnX13SAE-F:rvr1.02B:cvnSupermicro:ct17:cvr0123456789:skuDefaultstring:pfaFamily:
dmi.product.family: Family
dmi.product.name: Super Server
dmi.product.sku: Default string
dmi.product.version: 0123456789
dmi.sys.vendor: Supermicro
** Affects: linux (Ubuntu)
Importance: Undecided
Status: New
** Tags: amd64 apport-bug kernel-bug resolute
** Attachment added: "dmesg.txt"
https://bugs.launchpad.net/bugs/2167492/+attachment/6000626/+files/dmesg.txt
** Attachment removed: "CurrentDmesg.txt"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2167492/+attachment/6000630/+files/CurrentDmesg.txt
** Attachment removed: "WifiSyslog.txt"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2167492/+attachment/6000644/+files/WifiSyslog.txt
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2167492
Title:
NULL pointer dereference in mempool_free_bulk via crypt_endio during
dm-crypt device teardown
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2167492/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
