On 2026-10-13 14:53, Olivier Kaloudoff wrote:
On surprise removal of a GPU behind Thunderbolt (MI50, PCIe slot
power cut), the KFD teardown path wedges the host:
- the killed KFD process faults on its own SVM ranges during exit /
coredump (SIGBUS -> get_dump_page -> do_swap_page ->
svm_migrate_to_ram), submits SDMA work to an already-killed
drm_sched entity, and waits uninterruptibly on a fence that can
never signal;
- the dying task holds svms->lock, which wedges
svm_range_restore_work() and prevents kfd_processes_count from
draining;
- kgd2kfd_teardown_processes() therefore spins forever in
cond_resched() while holding the pciehp removal thread, producing
soft-lockup and hung-task splats.
This series:
1. tears KFD processes down in amdgpu_device_fini_hw() *before*
amdgpu_amdkfd_device_fini_sw() frees kfd.dev on disconnect, so the
drain happens while kfd.dev is still valid, and guards the second
kgd2kfd_teardown_processes() call against kfd.dev already being
gone;
2. makes the userptr and SVM range restore workers stop rescheduling
themselves on -ENODEV;
3. guards the hardware-accessing SVM migration sections with
drm_dev_enter()/drm_dev_exit() and bounds the job fence wait with
AMDGPU_FENCE_JIFFIES_TIMEOUT, so a migration cannot race with
unplug and a fence on a dead device cannot stall the critical
section or the faulting task indefinitely. A failed copy now also
rolls back migrate->dst and propagates the error instead of
completing as a successful migration.
Verified on MI50 behind Thunderbolt (2 GB SVM workload, PCIe slot
power cut): the workload takes SIGBUS, coredump completes,
kfd_processes_count drains in ~9s, pciehp removal finishes without
warnings, and the device re-enumerates cleanly on slot re-power.
Previously the same repro wedged the host every time.
Changes in v6:
- patch 3: check svm_migrate_copy_done() return in both callers; on
failure release dst pages, clear migrate->dst before finalize, and
propagate the error (Felix Kuehling)
- patch 3: dropped the dead ret < 0 branch (Christian König)
Changes in v5:
- patch 3: dropped the drm_dev_is_unplugged() and
pci_dev_is_disconnected() checks entirely; drm_dev_enter()/exit()
plus the bounded fence wait is sufficient (Felix Kuehling)
Changes in v4:
- patch 3: AMDGPU_FENCE_JIFFIES_TIMEOUT instead of an open-coded 10s;
drm_dev_enter()/exit() guarding the whole migration section
including the bounded wait (Felix Kuehling)
- patch 2: folded in the svm_range_restore_work() -ENODEV change;
fixed a latent uninitialized ret read on the evicted_bos-mismatch
and confirm_valid-WARN paths
Changes in v3:
- patch 1: NULL guard for the second teardown call (Christian König)
- patch 3: new, fail fast on gone device in SVM migration paths
Changes in v2:
- patch 1: drain before fini_sw instead of bounding the wait
(Christian König)
- patch 2: new, stop restore retries on unplug
Cc: Christian König <[email protected]>
Cc: Felix Kuehling <[email protected]>
Cc: Alex Deucher <[email protected]>
Cc: Xiaogang Chen <[email protected]>
Patches 1 and 2 are
Reviewed-by: Felix Kuehling <[email protected]>
For patch 3 I defer to Christian's expertise. That said, if we don't
need the dma_fence_wait_timeout there, then we also don't need to
complicate the error handling/unwinding in the migration functions.
Let me know if you want me to submit patches 1 and 2 independently of
patch 3.
Regards,
Felix
Olivier Kaloudoff (3):
drm/amdkfd: tear down KFD processes before early fini_sw on disconnect
drm/amdkfd: stop restore retries on device unplug
drm/amdkfd: bound SVM migration fence wait and guard with drm_dev_enter
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 5 +-
.../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 29 ++++--
drivers/gpu/drm/amd/amdkfd/kfd_device.c | 9 +-
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 97 ++++++++++++++---
drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 2 +-
5 files changed, 113 insertions(+), 29 deletions(-)
