Public bug reported:
linux-image-7.0.0-28-generic (7.0.0-28.28~24.04.1) took this commit in the -28
upload itself. It is listed under -28 in the source package changelog and absent
from the -26 and -14 stanzas:
drm/amdgpu: fix amdgpu_hmm_range_get_pages
upstream c08972f55594, 2026-02-18
The follow-up that repairs the consequence has never been applied:
drm/amdgpu: drop retry loop in amdgpu_hmm_range_get_pages
upstream 342981fff328, 2026-05-29
This is visible in the shipped source, not only in the changelog. In "apt-get
source linux-hwe-7.0" at 7.0.0-28.28~24.04.1, the file
drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c has mmu_interval_read_begin() hoisted
above the do loop, which is what c08972f55594 did, and still carries the timeout
variable and the retry: label that 342981fff328 removes. The string "drop retry
loop" appears in none of the three changelogs in that source package.
The first commit moved mmu_interval_read_begin() out of the per-chunk loop, so
the captured notifier_seq is never refreshed across retries. Quoting the second
commit's message:
the "goto retry" therefore degenerates into a busy spin that simply burns
CPU for the full HMM_RANGE_DEFAULT_TIMEOUT (~1s) window before finally
bailing out with -EAGAIN. This is pure latency with no chance of recovery,
and it actively hurts the KFD userptr stack
WHY IT WAS NOT BACKPORTED
342981fff328 carries neither a Fixes: tag nor Cc: [email protected], so it
was never queued for the stable trees, while the commit it repairs did reach
-28.
This is not an Ubuntu packaging slip. Any series tracking 7.0.y that took the
first commit is in the same state.
IMPACT
On ROCm, a host-to-device copy whose source is a writable MAP_PRIVATE file
mapping with resident pages stalls in one-second units. That is the path every
PyTorch process takes when loading a safetensors checkpoint, because safetensors
calls torch.UntypedStorage.from_file(shared=False) and PyTorch maps it writable.
Loading a 15.26 GiB checkpoint takes 86.7 s on -28 against 14.4 s on -14.
-28 is what linux-image-generic-hwe-24.04 resolves to today. Confirmed with
apt-cache policy: candidate 7.0.0-28.28~24.04.1, out of both noble-updates and
noble-security. So this reaches any 24.04 machine that lets updates run.
REPRODUCER
A 32 MiB copy, no PyTorch needed, about 30 seconds to run:
https://github.com/cadamcat/dual-radeon-vllm/blob/main/benchmarks/repro-
mmap-prot.hip.cpp
MEASURED
2x RX 7900 XT (gfx1100), ROCm 7.14, same machine and same userspace throughout,
only the kernel changed. 32 MiB copy, writable mapping, pages resident:
7.0.0-28-generic 16 019.3 / 16 019.6 / 16 019.9 / 16 020.1 ms
7.0.0-14-generic 18.6 / 18.9 / 20.2 ms
6.8.0-136-generic 22.1 / 23.3 / 26.0 ms
HMM_RANGE_DEFAULT_TIMEOUT is 1000 ms (include/linux/hmm.h), and every -28
measurement is an integer multiple of it. 16 019 ms is sixteen windows, and the
same reproducer on overlayfs and tmpfs gave 17 020 ms, which is seventeen.
Per-tensor stalls during a real checkpoint load measured 1005, 1042 and 1522 ms.
perf puts 98.7% of the time in
kfd_ioctl_svm -> svm_range_validate_and_map -> hmm_range_fault.
CONFIRMED BY REVERT
I rebuilt amdgpu.ko from "apt-get source linux-hwe-7.0" at 7.0.0-28.28~24.04.1
with 342981fff328 as the only change. It applies cleanly, net -8/+1. The .config
came verbatim from /boot/config-7.0.0-28-generic, built with the same gcc 13.3.0
the package was built with. I then loaded it into the running -28 kernel. Same
machine, same userspace, same reproducer, same file:
stock -28 16 019.7 / 16 019.6 / 16 019.1 / 16 019.7 / 16 019.0 ms
-28 + 342981fff328 17.0 / 17.0 / 17.0 ms
The reproducer's other three cases do not move. The read-only case stays at
3.0-3.1 ms and the not-resident writable case at 14.5-14.6 ms, before and after.
Only the case that entered the futile retry changes. The residual matches the
arithmetic: 16 019.7 ms is sixteen 1000 ms windows plus 19.7 ms of real work,
and
17.0 ms is inside the band 7.0.0-14-generic gives natively.
REQUEST
Backport 342981fff328 to the 7.0.0 series. It removes a retry path that upstream
has established can never succeed.
ENVIRONMENT CAVEAT
This machine is an Ubuntu 24.04 guest under Proxmox VE with the two GPUs passed
through via VFIO; systemd-detect-virt reports kvm. Passthrough is not the
variable. The -14 and -28 rows above are the same guest with only the kernel ABI
swapped, and the same reproducer run outside any VM on the Proxmox host's own
7.0.14 kernel completes in 24-29 ms. But I have no bare-metal machine running an
Ubuntu kernel, so this report cannot rule out that something about the
virtualised setup is required to see it.
Discussion with AMD, including their confirmation of the underlying
copy-on-write behaviour that triggers the retry:
https://github.com/ROCm/ROCm/issues/6523
ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: linux-image-7.0.0-28-generic 7.0.0-28.28~24.04.1
ProcVersionSignature: User Name 7.0.0-28.28~24.04.1-generic 7.0.12
Uname: Linux 7.0.0-28-generic x86_64
AlsaVersion: Advanced Linux Sound Architecture Driver Version k7.0.0-28-generic.
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.28.2-0ubuntu0.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path',
'/dev/snd/controlC1', '/dev/snd/hwC1D0', '/dev/snd/pcmC1D9p',
'/dev/snd/pcmC1D8p', '/dev/snd/pcmC1D7p', '/dev/snd/pcmC1D3p',
'/dev/snd/controlC2', '/dev/snd/hwC2D0', '/dev/snd/pcmC2D9p',
'/dev/snd/pcmC2D8p', '/dev/snd/pcmC2D7p', '/dev/snd/pcmC2D3p',
'/dev/snd/controlC0', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
Card0.Amixer.info: Error: [Errno 2] No such file or directory: 'amixer'
Card0.Amixer.values: Error: [Errno 2] No such file or directory: 'amixer'
Card1.Amixer.info: Error: [Errno 2] No such file or directory: 'amixer'
Card1.Amixer.values: Error: [Errno 2] No such file or directory: 'amixer'
Card2.Amixer.info: Error: [Errno 2] No such file or directory: 'amixer'
Card2.Amixer.values: Error: [Errno 2] No such file or directory: 'amixer'
CasperMD5CheckResult: unknown
CloudBuildName: server
CloudSerial: 20260705
CurrentDmesg: Error: command ['dmesg'] failed with exit code 1: dmesg: read
kernel buffer failed: Operation not permitted
Date: Tue Jul 28 05:34:41 2026
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: QEMU Standard PC (Q35 + ICH9, 2009)
ProcEnviron:
LANG=C.UTF-8
PATH=(custom, no user)
SHELL=/bin/bash
XDG_RUNTIME_DIR=<set>
ProcFB: 0 bochs-drmdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-7.0.0-28-generic
root=UUID=6239d00e-fbe6-43e9-89d8-d80643d1e789 ro console=tty1 console=ttyS0
RelatedPackageVersions:
linux-restricted-modules-7.0.0-28-generic N/A
linux-backports-modules-7.0.0-28-generic N/A
linux-firmware 20240318.git3b128b60-0ubuntu2.27
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux-hwe-7.0
UpgradeStatus: No upgrade log present (probably fresh install)
acpidump:
dmi.bios.date: 11/13/2025
dmi.bios.release: 0.0
dmi.bios.vendor: Proxmox distribution of EDK II
dmi.bios.version: 4.2025.05-2
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-q35-11.0
dmi.modalias:
dmi:bvnProxmoxdistributionofEDKII:bvr4.2025.05-2:bd11/13/2025:br0.0:svnQEMU:pnStandardPC(Q35+ICH9,2009):pvrpc-q35-11.0:cvnQEMU:ct1:cvrpc-q35-11.0:sku:pfa:
dmi.product.name: Standard PC (Q35 + ICH9, 2009)
dmi.product.version: pc-q35-11.0
dmi.sys.vendor: QEMU
** Affects: linux-hwe-7.0 (Ubuntu)
Importance: Undecided
Status: New
** Tags: amd64 apport-bug cloud-image noble
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2161985
Title:
7.0.0-28 backports c08972f55594 without its follow-up 342981fff328:
every amdgpu HMM retry burns a full 1s timeout
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-hwe-7.0/+bug/2161985/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
