On 2026-10-08 13:33, Olivier Kaloudoff wrote:
On surprise removal of a GPU behind Thunderbolt (MI50, PCIe slot power cut), the KFD teardown path wedges the host:- the killed KFD process faults on its own SVM ranges during exit / coredump (SIGBUS -> get_dump_page -> do_swap_page -> svm_migrate_to_ram), submits SDMA work to an already-killed drm_sched entity, and waits uninterruptibly on a fence that can never signal; - the dying task holds svms->lock, which wedges svm_range_restore_work() and prevents kfd_processes_count from draining; - kgd2kfd_teardown_processes() therefore spins forever in cond_resched() while holding the pciehp removal thread, producing soft-lockup and hung-task splats. This series: 1. tears KFD processes down in amdgpu_device_fini_hw() *before* amdgpu_amdkfd_device_fini_sw() frees kfd.dev on disconnect, so the drain happens while kfd.dev is still valid, and guards the second kgd2kfd_teardown_processes() call against kfd.dev already being gone; 2. makes the userptr and SVM range restore workers stop rescheduling themselves on -ENODEV; 3. guards the hardware-accessing SVM migration sections with drm_dev_enter()/drm_dev_exit() and bounds the job fence wait with AMDGPU_FENCE_JIFFIES_TIMEOUT, so a migration cannot race with unplug and a fence on a dead device cannot stall the critical section or the faulting task indefinitely. Verified on MI50 behind Thunderbolt (2 GB SVM workload, PCIe slot power cut): the workload takes SIGBUS, coredump completes, kfd_processes_count drains in ~7s, pciehp removal finishes without warnings, and the device re-enumerates cleanly on slot re-power. Previously the same repro wedged the host every time. Changes in v5: - patch 3: dropped the drm_dev_is_unplugged() and pci_dev_is_disconnected() checks entirely; drm_dev_enter()/exit() plus the bounded fence wait is sufficient (Felix Kuehling)
Thanks. Patches 1 and 2 look good now. I found one more problem with the handling of the new timeout error conditions in patch 3 that will need to be addressed before this can be submitted.
Regards, Felix
Changes in v4: - patch 3: AMDGPU_FENCE_JIFFIES_TIMEOUT instead of an open-coded 10s; drm_dev_enter()/exit() guarding the whole migration section including the bounded wait (Felix Kuehling) - patch 2: folded in the svm_range_restore_work() -ENODEV change; fixed a latent uninitialized ret read on the evicted_bos-mismatch and confirm_valid-WARN paths Changes in v3: - patch 1: NULL guard for the second teardown call (Christian König) - patch 3: new, fail fast on gone device in SVM migration paths Changes in v2: - patch 1: drain before fini_sw instead of bounding the wait (Christian König) - patch 2: new, stop restore retries on unplug Cc: Christian König <[email protected]> Cc: Felix Kuehling <[email protected]> Cc: Alex Deucher <[email protected]> Cc: Xiaogang Chen <[email protected]> Olivier Kaloudoff (3): drm/amdkfd: tear down KFD processes before early fini_sw on disconnect drm/amdkfd: stop restore retries on device unplug drm/amdkfd: bound SVM migration fence wait and guard with drm_dev_enter drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 5 ++- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 29 +++++++++++++---- drivers/gpu/drm/amd/amdkfd/kfd_device.c | 9 ++++- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 36 ++++++++++++++++++-- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 2 +- 5 files changed, 70 insertions(+), 11 deletions(-)
