On surprise removal, amdgpu_device_fini_hw() calls amdgpu_amdkfd_device_fini_sw() when pci_dev_is_disconnected() is true. fini_sw frees kfd.dev, but KFD processes may still be alive and hold references to it: kfd_process_destroy() dereferences kfd.dev via kfd_processes_remove(), and svm_range_list_fini() / kfd_process_free_outstanding_kfd_bos() touch per-device SVM state.
The kfd_processes_count wait only happens later in kgd2kfd_teardown_processes(), called from the top-half remove path well after fini_sw. In the observed failure the killed KFD process (SIGBUS on unplug) stuck in D state on its exit path, so kfd_processes_count never drained and the PCIe removal thread wedged permanently in cond_resched(), producing soft-lockup and hung-task splats. Tear KFD processes down before freeing kfd.dev on the disconnect path, and guard kgd2kfd_teardown_processes() against kfd.dev already being gone when it runs a second time. Cc: Xiaogang Chen <[email protected]> Signed-off-by: Olivier Kaloudoff <[email protected]> --- Changes in v5: - none Changes in v4: - none Changes in v3: - added NULL guard for the second teardown_processes call once kfd.dev is already freed (Christian König) - dropped the bounded-wait mechanism from v1; the drain now happens before fini_sw instead (Christian König) Changes in v2: - none --- diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c index 9269e78..bd9d73a 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c @@ -4315,8 +4315,11 @@ void amdgpu_device_fini_hw(struct amdgpu_device *adev) * before ip_fini_early to prevent kfd locking refcount issues by calling * amdgpu_amdkfd_suspend() */ - if (pci_dev_is_disconnected(adev->pdev)) + if (pci_dev_is_disconnected(adev->pdev)) { + /* Drain KFD processes while kfd.dev is still valid */ + amdgpu_amdkfd_teardown_processes(adev); amdgpu_amdkfd_device_fini_sw(adev); + } amdgpu_coredump_fini(adev); amdgpu_device_ip_fini_early(adev); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c index 586e640..43029b2 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c @@ -1801,7 +1801,14 @@ void kgd2kfd_teardown_processes(struct amdgpu_device *adev) mutex_unlock(&kfd_processes_mutex); - /* wait all kfd processes use adev terminate */ + /* kfd.dev may already be freed on surprise removal */ + if (!adev->kfd.dev) + return; + + /* + * Wait for all KFD processes using adev to terminate. Must be + * unbounded: device state is freed once they are gone. + */ while (!!atomic_read(&adev->kfd.dev->kfd_processes_count)) cond_resched(); } -- 2.47.3
