== Summary ==
Restarting a KFD application reliably deadlocks the next instance during
its initial GPU memory allocations. Two threads block permanently:
(a) the new userspace process in
kfd_ioctl_alloc_memory_of_gpu -> svm_range_list_lock_and_flush_work
-> flush_work -> __flush_work (D state)
(b) a kworker running
svm_range_restore_work -> svm_range_list_lock_and_flush_work
-> flush_work -> __flush_work (D state)
Both wait for svms->deferred_list_work, which never completes. On a
7.0.12 kernel the userspace side was instead observed blocked in
dma_fence_default_wait, suggesting the deferred work item is itself
waiting on a dma fence that cannot complete (plausibly because it
requires KFD queue progress while queues are suspended for
eviction/restore). Once this state is reached, every subsequent
kfd_ioctl_alloc_memory_of_gpu queues behind the stuck flush; the
process cannot be killed (uninterruptible sleep) and the device is
unusable until reboot.
The trigger is a KFD process teardown followed by a fresh KFD process
doing VRAM allocations — i.e. simply restarting a ROCm/HIP application.
First start after boot succeeds; a restart wedges with high
probability (observed ~70-100%).
This does NOT appear to be a regression: reproduced identically on
6.17.13, 7.0.12 and 7.2.6 (all Proxmox pve builds plus one custom
config). Possibly long-standing; likely related to the unresolved
"svm restore worker stuck" issues discussed on amd-gfx in 2025.
== Environment ==
GPU: AMD Radeon Instinct MI50 32GB, gfx906, 0000:05:00.0
(Vega 20, PCIe, attached via Thunderbolt 3 eGPU enclosure,
hotplug-capable PCI topology — may be relevant since SVM
eviction/restore work is tied to hotplug/fault events)
Host: AMD Strix Halo (Ryzen AI Max+ 395), Proxmox VE 9 host
Kernels: 6.17.13-13-pve, 7.0.12-1-pve, 7.2.6-k726 (custom config)
— all affected
Cmdline: amdgpu.dc=0 modprobe.blacklist=amdxdna apparmor=0
Userspace: llama.cpp llama-server (HIP/ROCm), hsakmt/ROCm stack;
any KFD client restart reproduces it
Taint: 512 (U — unsigned module loaded early at boot; no DKMS,
no out-of-tree module in the KFD path)
== Reproduction ==
1. Boot the system with the MI50 bound to amdgpu/KFD.
2. Start a KFD application that allocates substantial VRAM
(llama-server --model <19GB GGUF> -ngl 99; ROCm offload).
Wait for it to finish initializing (serves inference).
3. Stop it (SIGTERM/SIGKILL/systemctl stop) and start it again.
4. During model VRAM allocation the new process blocks in
kfd_ioctl_alloc_memory_of_gpu and never returns.
Faster reproducer sketch (what llama init does): a fresh process does
hsa_init -> acquire gfx906 node -> repeated hsaKmtAllocMemory +
hsaKmtMapMemoryToGPU of large (1-4 GB) VRAM blocks. The wedged state
is always entered via the first few alloc ioctls after process start.
== Observed stacks (kernel 7.2.6, SysRq-w / /proc/<pid>/stack) ==
Userspace process (llama-server, D state):
__flush_work+0x2c4/0x3c0
flush_work+0x1c/0x30
svm_range_list_lock_and_flush_work+0x2f/0x100 [amdgpu]
kfd_ioctl_alloc_memory_of_gpu+0x7c/0x3a0 [amdgpu]
kfd_ioctl+0x408/0x640 [amdgpu]
__x64_sys_ioctl+0xa5/0x100
Concurrent kworker (D state, same boot):
__flush_work+0x2c4/0x3c0
flush_work+0x1c/0x30
svm_range_list_lock_and_flush_work+0x2f/0x100 [amdgpu]
svm_range_restore_work+0x75/0x3a0 [amdgpu]
process_one_work+0x1a9/0x390
worker_thread+0x1b8/0x360
On 7.0.12-1-pve the userspace process was observed in:
dma_fence_default_wait+0x87/0x2a0
== Analysis ==
svm_range_list_lock_and_flush_work() flushes svms->deferred_list_work
then takes mmap_write_lock, retrying while deferred_range_list is
non-empty. The observed state has BOTH the alloc path and
svm_range_restore_work blocked in flush_work() on the same
deferred_list_work — i.e. the deferred work item can never complete.
Likely circular wait: svm_range_restore_work runs while KFD queues
are suspended for eviction/restore; a queued deferred work item
(migrate/map) needs GPU progress / a dma fence that requires those
queues to run; restore_work waits for the deferred item to drain;
the alloc path waits behind the same flush.
Related prior work (this may be a remaining/unfixed variant):
- CVE-2022-49133 / commit "drm/amdkfd: svm range restore work
deadlock when process exit" — fixed a different restore_work
flush deadlock at process exit.
- [PATCH] "drm/amdkfd: Fix the deadlock in svm_range_restore_work"
(Emily Deng, amd-gfx, Feb 2025) — mmap_write_downgrade() fix for
restore_work vs svm_range_evict_svm_bo_worker.
- [PATCH v2 3/3] "drm/amdkfd: Don't stuck in svm restore worker"
(amd-gfx, Oct 2025) — restore work retrying forever when the VMA
is gone; the corresponding !vma unmap path does not appear to be
present in current mainline kfd_svm.c.
== What was ruled out ==
- HSA_XNACK=0 does not avoid the deadlock (alloc path always enters
svm_range_list_lock_and_flush_work regardless).
- Not context-size/alloc-size specific: reproduces with an 8GB
instead of 32GB context.
- Not specific to the custom kernel: identical deadlock on stock
7.0.12-1-pve and 6.17.13-13-pve builds.
- kfdtest (ROCT-Thunk-Interface test suite) memory/queue tests all
pass while a healthy KFD process holds the device — the deadlock
specifically needs the process-teardown -> fresh-init sequence.
- Hardware side: link stable at 40 Gb/s, card healthy — the wedge
reproduces with zero link events in dmesg.
== Logs ==
Full SysRq-w dumps, both blocked stacks, dmesg and llama-side logs
attached / available on request (saved per-trial).
== Notes for maintainers ==
- The MI50 is attached via Thunderbolt (pciehp hotplug topology).
If SVM restore work scheduling is influenced by hotplug-related
eviction/fault paths, this setup may expose a timing window that
does not exist on fixed PCIe. Happy to test patches — reproduction
takes ~5 minutes per trial on this machine.
