Hello,
Here is the root cause analysis of this issue:
The issue is caused by the FADump kernel running out of memory (OOM)
during early boot. Since the OOM occurs very early, the corresponding
OOM trace is not printed or observed. The kernel then hits a NULL pointer
dereference because the component whose memory allocation failed does not
handle the allocation failure correctly and allows the kernel to continue
booting. When the kernel later accesses the memory assuming the allocation
succeeded, it results in a NULL pointer dereference.
The component whose allocation failed due to OOM is sparse:
[ 0.000000] sparse_init_nid: node[0] memory map backing failed. Some
memory will not be available. <--- allocation failed
<snip...>
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x0000000effffffff]
[ 0.000000] BUG: Kernel NULL pointer dereference on read at
0x00000010 <----- kernel hitting error
This gap - where the kernel continues booting even after a memory allocation
for a critical data structure fails in the sparse component - has already
been identified and fixed upstream:
commit b3ef855262928c5e80e881895694891164c13fc8
Author: Muchun Song <[email protected]>
Date: Fri Jun 12 11:58:50 2026 +0800
mm/sparse: panic on memmap and usemap allocation failure
If you reproduce the same scenario with this commit included, you'll instead
hit the following panic:
[ 0.000000] Initmem setup node 0 [mem
0x0000000000000000-0x00000063ffffffff]
[ 0.000000] Kernel panic - not syncing: Failed to allocate memmap for
section 15619
[ 0.000000] CPU: 0 UID: 0 PID: 0 Comm: swapper Not tainted 7.3.0-rc1+
#1 PREEMPT(undef)
[ 0.000000] Hardware name: IBM,9105-22A Power11 (architected)
0x820200 0xf000007 of:IBM,FW1110.00 (RB1110_048) hv:phyp pSeries
[ 0.000000] Call Trace:
[ 0.000000] [c000000002d4fd50] [c0000000012b56f8]
dump_stack_lvl+0xd8/0xf0 (unreliable)
[ 0.000000] [c000000002d4fd80] [c000000000212d58] vpanic+0x50c/0x550
[ 0.000000] [c000000002d4fe20] [c000000000212dd8]
do_panic_on_target_cpu+0x0/0x28
[ 0.000000] [c000000002d4fe40] [c00000000206d83c]
sparse_init_nid+0x274/0x278
[ 0.000000] [c000000002d4fed0] [c00000000206da08] sparse_init+0x1c8/0x290
[ 0.000000] [c000000002d4ff20] [c000000002061c5c]
mm_core_init_early+0x30/0x44
[ 0.000000] [c000000002d4ff40] [c000000002005f90] start_kernel+0x90/0x608
[ 0.000000] [c000000002d4ffe0] [c00000000000e788]
start_here_common+0x1c/0x20
[ 0.000000] Rebooting in 10 seconds..
The above commit does not resolve the reported issue; it only makes the
failure easier to understand.
Increasing the crashkernel reservation should resolve the issue for this
configuration. I recommend increasing it by 2G or more.
The configured crashkernel value follows the recommendation, but the
recommended value is not sufficient for this particular configuration
because it uses a 4K page size. With 4K pages, the sparse and similar
per-page/per-section allocations require more memory than with 64K pages.
This is why the issue is not observed with the same configuration using a
64K page size.
I’m exploring ways to reduce FADump’s memory footprint by limiting
per-page/per-section allocations to memory actually used by FADump.
Until then, please increase the crashkernel size to avoid this issue.
- Sourabh Jain
On 23/04/26 17:53, Venkat Rao Bagalkote wrote:
Hello,
IBM CI has observed a kernel crash when testing FADUMP on ppc64le.
i.e., when a crash is explicitly triggered while FADUMP is active.
System details:
--------------
Architecture : ppc64le
Platform : IBM Power11 (pSeries)
Machine : 9080-HEX
Firmware : FW1110.01 (NH1110_069)
MMU : Radix
Kernel : 7.0.0+
Boot mode : FADUMP enabled
Upstream kernel version: 7.0.0+
Upstream kernel commit-id: 2e68039281932e6dc37718a1ea7cbb8e2cda42e6
kexec-tools 2.0.32.git
makedumpfile: version 1.7.9 (released on 20 Apr 2026)
Attached is the .config file.
Crash Logs:
[ 0.000000] BUG: Kernel NULL pointer dereference on read at 0x00000010
[ 0.000000] Faulting instruction address: 0xc000000000aba3d0
[ 0.000000] Oops: Kernel access of bad area, sig: 7 [#1]
[ 0.000000] LE PAGE_SIZE=4K MMU=Radix SMP NR_CPUS=8192 NUMA pSeries
[ 0.000000] Modules linked in:
[ 0.000000] CPU: 0 UID: 0 PID: 0 Comm: swapper Not tainted 7.0.0+
#1 PREEMPT(undef)
[ 0.000000] Hardware name: IBM,9080-HEX Power11 (architected)
0x820200 0xf000007 of:IBM,FW1110.01 (NH1110_069) hv:phyp pSeries
[ 0.000000] NIP: c000000000aba3d0 LR: c000000000708c38 CTR:
0000000000000000
[ 0.000000] REGS: c000000002d6fb10 TRAP: 0300 Not tainted (7.0.0+)
[ 0.000000] MSR: 8000000000001033 <SF,ME,IR,DR,RI,LE> CR:
44000248 XER: 20040001
[ 0.000000] CFAR: c000000000aba41c DAR: 0000000000000010 DSISR:
00080000 IRQMASK: 1
[ 0.000000] GPR00: c0000000020548b8 c000000002d6fdb0
c000000001a9a100 0000000000000010
[ 0.000000] GPR04: 0000000000000000 0000000000000008
c000000002d6fcc8 0000000000000000
[ 0.000000] GPR08: ffffffffffffffff ffffffffffffffff
0000000000000008 ffffffffffffffc8
[ 0.000000] GPR12: c000000002b30cd0 c00000000307a000
0000000000000000 0000000000000000
[ 0.000000] GPR16: 0000000000000000 0000000000000000
0000000000000000 0000000000000000
[ 0.000000] GPR20: 0000000000c00000 0000000000000008
0000000000000000 0000000000306000
[ 0.000000] GPR24: 0000000000000eff 0000000000000bfa
c000000001461398 c000000001891c58
[ 0.000000] GPR28: c000000002d44230 c000000002d44268
0000000000bfa000 0000000000001000
[ 0.000000] NIP [c000000000aba3d0] __bitmap_set+0x90/0xe0
[ 0.000000] LR [c000000000708c38] subsection_mask_set+0x38/0x50
[ 0.000000] Call Trace:
[ 0.000000] [c000000002d6fdb0] [c000000002744280]
init_task+0x0/0x1d80 (unreliable)
[ 0.000000] [c000000002d6fdd0] [c0000000020548b8]
sparse_init_subsection_map+0xac/0x138
[ 0.000000] [c000000002d6fe80] [c00000000204aa8c]
free_area_init+0x258/0x4f4
[ 0.000000] [c000000002d6ff50] [c000000002004f5c]
start_kernel+0x98/0x5fc
[ 0.000000] [c000000002d6ffe0] [c00000000000e998]
start_here_common+0x1c/0x20
[ 0.000000] Code: f92afff8 4200ffe4 55893032 38630008 798c18e8
7d295850 7c636214 7d2907b4 2c290000 4d820020 7d452214 3920ffff
<e8e30000> 7d4a00d0 554a06be 7d295436
[ 0.000000] ---[ end trace 0000000000000000 ]---
[ 0.000000]
[ 0.000000] Kernel panic - not syncing: Fatal exception
If you happen to fix this, please add below tag.
Reported-by: Venkat Rao Bagalkote <[email protected]>
Regards,
Venkat.
