Public bug reported: [Impact]
The resolute generic kernel ships CONFIG_KEXEC_HANDOVER_ENABLE_DEFAULT=y, so KHO is on at every boot with no kho= parameter. At boot KHO reserves scratch areas and marks their pageblocks MIGRATE_CMA. Only movable allocations can use those blocks. Anything unmovable or pinned can't use them: GPU buffers (amdgpu GTT on APUs), VFIO/PCI-passthrough guest memory, RDMA registrations. On a 128 GiB Ryzen AI Max+ 395 (Strix Halo, unified memory), the scratch is 13.3 GiB, about 10% of RAM. A single GPU process stops at 107 GiB under a 120 GiB GTT limit, with 13 GiB of RAM still free. With kho=off the same test reaches 117 GiB and stops only at RAM. Nothing on this machine uses kexec handover / live update. The scratch is reserved anyway. LP: #2143831 enabled KHO for resolute and said "The feature is disabled at runtime by default unless the cmdline kho=on is specified". The shipped config does the opposite. 26.10 already turned the default off (7.3.0-9.9, "[Config] Disable KEXEC_HANDOVER_ENABLE_DEFAULT", LP: #2169198, filed for a mini-iso hang). This bug asks for the same one-line config change on resolute. Debian and Fedora both leave the option unset (KHO only with kho=on). Proxmox proposed the same change in July 2026 for the same reason, pinned memory for passthrough guests: https://lore.proxmox.com/all/[email protected]/ [System] - Framework Desktop, AMD Ryzen AI Max+ 395, Radeon 8060S (gfx1151), 128 GB LPDDR5X - Ubuntu Server 26.04 LTS, linux-image-7.0.0-38-generic 7.0.0-38.38 - Stock cmdline apart from crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M - ttm pages_limit=31457280 (120 GiB GTT), swap off, ROCm 10.1 [Evidence that the CMA pages are KHO scratch] There is no CMA area: CmaTotal: 0 kB, /sys/kernel/mm/cma/ is empty, the boot log shows "0K cma-reserved", and CONFIG_CMA_SIZE_MBYTES=0. CmaFree is still non-zero, 14.27 GB five minutes after boot. /proc/pagetypeinfo, CMA column: DMA32 380 + Normal 6424 = 6804 pageblocks x 2 MiB = 13,608 MiB. /sys/kernel/debug/kho/out/scratch_len: 0x2f800000 = 760 MiB (lowmem; matches the 380 DMA32 blocks) 0x213e00000 = 8,510 MiB 0x10f200000 = 4,338 MiB total = 13,608 MiB, an exact match for the CMA pageblocks Boot log, same kernel, only kho=off added: KHO on: Memory: 114581060K/133474628K available (... 18849804K reserved, 0K cma-reserved) kho=off: Memory: 128515616K/133474628K available (... 4915232K reserved, 0K cma-reserved) difference: 13,934,572K = 13,608 MiB, the scratch total again (My guess, not verified: the scratch scales with early memblock reservations, so the 4 GiB crashkernel reservation is probably a large part of the 8.5 GiB global area. Machines with large crashkernel settings would then lose the most.) [Test case] A small HIP program allocates 1 GiB with hipMalloc, touches it with hipMemset, and repeats until hipMalloc fails or MemAvailable drops below 4 GiB. KHO on (default) kho=off allocated + touched 107 GiB 117 GiB stopped by hipMalloc out of memory the 4 GiB MemAvailable floor MemAvailable at stop 13.4 GiB 3.3 GiB CmaFree at stop 13.0 GiB 0 kernel log page allocation failure (below) none KHO on, at the point of failure: page allocation failure: order:0, mode:...(GFP_HIGHUSER|__GFP_ZERO|__GFP_RETRY_MAYFAIL) ... gpu_active:112960512kB ... free_cma:3419351 With kho=off: no CMA pageblocks in any zone, CmaFree: 0, no /sys/kernel/debug/kho. To check any machine without a GPU: if CmaTotal is 0 and CmaFree is non-zero in /proc/meminfo, or the CMA column in /proc/pagetypeinfo is non-zero with no CMA area, KHO scratch is reserved. Verify a fixed kernel: boot without kho=, then CmaFree should be 0 and /sys/kernel/debug/kho/out should not exist. [Workaround] Add kho=off to the kernel cmdline (e.g. a /etc/default/grub.d/ snippet, then update-grub). [Where problems could occur] Anyone who relies on KHO / live update without passing kho=on would need to add kho=on. On a general-purpose LTS kernel that group should be small, and LP: #2143831 already told users to expect kho=on. kdump's crashkernel reservation is separate and does not change. [Possibly related] - LP: #2152152, "KHO: Failed to reserve lowmem scratch buffer on every boot" (resolute, Confirmed). The same default causes it. - LP: #2161812, kdump capture kernel never reaches userspace on resolute. Not verified, but an August 2026 upstream patch reports that kdump kernels with KHO enabled by default reserve scratch out of the crashkernel area and fail early ("kho: do not reserve scratch memory in kdump kernel"). ** Affects: linux (Ubuntu) Importance: Undecided Status: New ** Tags: amd64 resolute -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2171297 Title: resolute: please disable CONFIG_KEXEC_HANDOVER_ENABLE_DEFAULT (as done for 26.10 in LP: #2169198); KHO scratch makes 13 GiB unusable for GPU / pinned memory To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2171297/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
