Follow-up with new evidence from the same machine (MI50 gfx906,
Thunderbolt eGPU, 7.2.6-k726 with CONFIG_UBSAN=y). Three distinct
reproductions now — the original D-state wedge, a TB-hot-unplug
variant, and a clean-restart capture that produced a GPU page fault +
devcoredump.
== Reproduction 1: hot-unplug teardown (kernel-side splats) ==
A Thunderbolt link flap drove PCIe hot-unplug while llama-server was
running; the coredump walked into the SVM path and produced:
svm_migrate_copy_memory_gart -> amdgpu_copy_buffer ->
amdgpu_job_submit:
drm_sched_rq_add_entity: *ERROR* Trying to push to a killed entity
refcount_t: addition on 0; use-after-free.
refcount_t: saturated; leaking memory.
i.e. migration submits a job to a drm_sched_entity already killed by
queue eviction — the job never runs, the fence never signals, deferred
work blocks forever, subsequent allocs queue behind it. This supplies
the missing "why the fence never completes" for the alloc-path wedge.
Same boot, UBSAN invalid-loads inside evict_process_queues_cpsch:
UBSAN: invalid-load kfd_device_queue_manager.c:1373
UBSAN: invalid-load kfd_device_queue_manager.c:1388
UBSAN: invalid-load kfd_device_queue_manager.c:1391
reached via svm_migrate_vram_to_ram -> migrate_vma_setup ->
__mmu_notifier_invalidate_range_start ->
svm_range_cpu_invalidate_pagetables -> kgd2kfd_quiesce_mm ->
kfd_process_evict_queues -> evict_process_queues_cpsch
(reading torn/freed queue-manager state mid-eviction).
== Reproduction 2: clean restart — userspace spin + device wedge ==
A clean stop/start of llama-cpp.service (SIGTERM, no hotplug, link
stable 40Gb/s) wedged differently this time:
- VRAM allocs SUCCEEDED (19.5GB resident) — the alloc path did not
block this trial
- the process then spun in userspace at ~90% CPU, zero syscalls, with
gpu_busy=0 forever
- gdb shows the spin inside libhsa-runtime64 (rdtsc wait loop) —
hsa_signal_wait polling a completion signal that never arrives;
the submitted GPU work never executes
So the wedge has a second symptom class: allocs pass, queues are dead,
userspace spins on a signal that can never fire.
== Reproduction 3: the smoking gun — stale SVM mapping faults ==
The next auto-restart produced the hardware-level evidence:
amdgpu 0000:05:00.0: [gfxhub0] no-retry page fault
(src_id:0 ring:158 vmid:8 pasid:58)
Process llama-server pid 1356287
in page starting at address 0x00007bc82af9c000 (UTCL2)
WALKER_ERROR: 0x6 MAPPING_ERROR: 0x1 PERMISSION_FAULTS: 0x3
qcm fence wait loop timeout expired
cp might be in an unrecoverable state due to unsuccessful queues
preemption
GPU reset begin! (BACO)
VRAM is lost due to GPU reset!
Fence fallback timer expired on ring comp_1.0.0
GPU reset(2) succeeded!
device wedged, but no recovery needed
The GPU's page walker faulted on a VA inside the process's own address
space (0x7bc82af9c000, adjacent to the main thread's stack at
0x7bc82afb1000) — MAPPING_ERROR+WALKER_ERROR means the walk hit a
bogus/stale PTE, consistent with a torn SVM mapping surviving process
teardown into the next process's restored tables. Fault -> preempt
timeout -> reset -> device wedged; every subsequent instance spins in
hsa_signal_wait until reboot.
After reboot: first start healthy in ~60s (confirming
first-start-works / restart-wedges).
== Artifacts ==
- amdgpu devcoredump captured from /sys/class/drm/card0/device/
devcoredump/data at the 15:15 reset (3472 lines, ring/reg state)
- full journalctl -k covering both restart attempts
- userspace stacks: main thread spinning in libhsa-runtime64
(rdtsc loop @ file offset ~0x6a7c5), worker threads in
kfd_wait_on_events
Happy to send the devcoredump or run further captures — reproduction
is ~15-20 min per trial on this box.
Olivier
On Fri, 25 Sept 2026 at 14:52, Olivier Kaloudoff <
[email protected]> wrote:
> == Summary ==
>
> Restarting a KFD application reliably deadlocks the next instance during
> its initial GPU memory allocations. Two threads block permanently:
>
> (a) the new userspace process in
> kfd_ioctl_alloc_memory_of_gpu -> svm_range_list_lock_and_flush_work
> -> flush_work -> __flush_work (D state)
>
> (b) a kworker running
> svm_range_restore_work -> svm_range_list_lock_and_flush_work
> -> flush_work -> __flush_work (D state)
>
> Both wait for svms->deferred_list_work, which never completes. On a
> 7.0.12 kernel the userspace side was instead observed blocked in
> dma_fence_default_wait, suggesting the deferred work item is itself
> waiting on a dma fence that cannot complete (plausibly because it
> requires KFD queue progress while queues are suspended for
> eviction/restore). Once this state is reached, every subsequent
> kfd_ioctl_alloc_memory_of_gpu queues behind the stuck flush; the
> process cannot be killed (uninterruptible sleep) and the device is
> unusable until reboot.
>
> The trigger is a KFD process teardown followed by a fresh KFD process
> doing VRAM allocations — i.e. simply restarting a ROCm/HIP application.
> First start after boot succeeds; a restart wedges with high
> probability (observed ~70-100%).
>
> This does NOT appear to be a regression: reproduced identically on
> 6.17.13, 7.0.12 and 7.2.6 (all Proxmox pve builds plus one custom
> config). Possibly long-standing; likely related to the unresolved
> "svm restore worker stuck" issues discussed on amd-gfx in 2025.
>
> == Environment ==
>
> GPU: AMD Radeon Instinct MI50 32GB, gfx906, 0000:05:00.0
> (Vega 20, PCIe, attached via Thunderbolt 3 eGPU enclosure,
> hotplug-capable PCI topology — may be relevant since SVM
> eviction/restore work is tied to hotplug/fault events)
> Host: AMD Strix Halo (Ryzen AI Max+ 395), Proxmox VE 9 host
> Kernels: 6.17.13-13-pve, 7.0.12-1-pve, 7.2.6-k726 (custom config)
> — all affected
> Cmdline: amdgpu.dc=0 modprobe.blacklist=amdxdna apparmor=0
> Userspace: llama.cpp llama-server (HIP/ROCm), hsakmt/ROCm stack;
> any KFD client restart reproduces it
> Taint: 512 (U — unsigned module loaded early at boot; no DKMS,
> no out-of-tree module in the KFD path)
>
> == Reproduction ==
>
> 1. Boot the system with the MI50 bound to amdgpu/KFD.
> 2. Start a KFD application that allocates substantial VRAM
> (llama-server --model <19GB GGUF> -ngl 99; ROCm offload).
> Wait for it to finish initializing (serves inference).
> 3. Stop it (SIGTERM/SIGKILL/systemctl stop) and start it again.
> 4. During model VRAM allocation the new process blocks in
> kfd_ioctl_alloc_memory_of_gpu and never returns.
>
> Faster reproducer sketch (what llama init does): a fresh process does
> hsa_init -> acquire gfx906 node -> repeated hsaKmtAllocMemory +
> hsaKmtMapMemoryToGPU of large (1-4 GB) VRAM blocks. The wedged state
> is always entered via the first few alloc ioctls after process start.
>
> == Observed stacks (kernel 7.2.6, SysRq-w / /proc/<pid>/stack) ==
>
> Userspace process (llama-server, D state):
>
> __flush_work+0x2c4/0x3c0
> flush_work+0x1c/0x30
> svm_range_list_lock_and_flush_work+0x2f/0x100 [amdgpu]
> kfd_ioctl_alloc_memory_of_gpu+0x7c/0x3a0 [amdgpu]
> kfd_ioctl+0x408/0x640 [amdgpu]
> __x64_sys_ioctl+0xa5/0x100
>
> Concurrent kworker (D state, same boot):
>
> __flush_work+0x2c4/0x3c0
> flush_work+0x1c/0x30
> svm_range_list_lock_and_flush_work+0x2f/0x100 [amdgpu]
> svm_range_restore_work+0x75/0x3a0 [amdgpu]
> process_one_work+0x1a9/0x390
> worker_thread+0x1b8/0x360
>
> On 7.0.12-1-pve the userspace process was observed in:
>
> dma_fence_default_wait+0x87/0x2a0
>
> == Analysis ==
>
> svm_range_list_lock_and_flush_work() flushes svms->deferred_list_work
> then takes mmap_write_lock, retrying while deferred_range_list is
> non-empty. The observed state has BOTH the alloc path and
> svm_range_restore_work blocked in flush_work() on the same
> deferred_list_work — i.e. the deferred work item can never complete.
>
> Likely circular wait: svm_range_restore_work runs while KFD queues
> are suspended for eviction/restore; a queued deferred work item
> (migrate/map) needs GPU progress / a dma fence that requires those
> queues to run; restore_work waits for the deferred item to drain;
> the alloc path waits behind the same flush.
>
> Related prior work (this may be a remaining/unfixed variant):
>
> - CVE-2022-49133 / commit "drm/amdkfd: svm range restore work
> deadlock when process exit" — fixed a different restore_work
> flush deadlock at process exit.
> - [PATCH] "drm/amdkfd: Fix the deadlock in svm_range_restore_work"
> (Emily Deng, amd-gfx, Feb 2025) — mmap_write_downgrade() fix for
> restore_work vs svm_range_evict_svm_bo_worker.
> - [PATCH v2 3/3] "drm/amdkfd: Don't stuck in svm restore worker"
> (amd-gfx, Oct 2025) — restore work retrying forever when the VMA
> is gone; the corresponding !vma unmap path does not appear to be
> present in current mainline kfd_svm.c.
>
> == What was ruled out ==
>
> - HSA_XNACK=0 does not avoid the deadlock (alloc path always enters
> svm_range_list_lock_and_flush_work regardless).
> - Not context-size/alloc-size specific: reproduces with an 8GB
> instead of 32GB context.
> - Not specific to the custom kernel: identical deadlock on stock
> 7.0.12-1-pve and 6.17.13-13-pve builds.
> - kfdtest (ROCT-Thunk-Interface test suite) memory/queue tests all
> pass while a healthy KFD process holds the device — the deadlock
> specifically needs the process-teardown -> fresh-init sequence.
> - Hardware side: link stable at 40 Gb/s, card healthy — the wedge
> reproduces with zero link events in dmesg.
>
> == Logs ==
>
> Full SysRq-w dumps, both blocked stacks, dmesg and llama-side logs
> attached / available on request (saved per-trial).
>
> == Notes for maintainers ==
>
> - The MI50 is attached via Thunderbolt (pciehp hotplug topology).
> If SVM restore work scheduling is influenced by hotplug-related
> eviction/fault paths, this setup may expose a timing window that
> does not exist on fixed PCIe. Happy to test patches — reproduction
> takes ~5 minutes per trial on this machine.
>
--
Olivier Kaloudoff
Résidence la source - bâtiment B
12 allée des Grillons
13 620 Carry le Rouet
06 20 65 71 33
