amdgpu_amdkfd_restore_userptr_worker() reschedules itself every AMDGPU_USERPTR_RESTORE_DELAY_MS whenever validation of invalidated userptr BOs fails, and svm_range_restore_work() likewise reschedules itself every AMDGPU_SVM_RANGE_RESTORE_DELAY_MS when range validation fails. On PCIe surprise removal the underlying update/migration paths fail fast with -ENODEV (amdgpu_vm_update_pdes() and amdgpu_vm_update_range() return -ENODEV when drm_dev_enter() fails, which propagates up through validate_invalid_user_pages(); the SVM validation paths get -ENODEV once the migration paths check for a gone device, see patch 3). Both workers ignore the error code and reschedule unconditionally, producing endless retry loops on a device that is permanently gone. The worker cancel in the teardown paths only runs at last-VM/process destroy, which itself can be delayed arbitrarily on unplug.
Observed on MI50 eGPU TB3 surprise removal: ~11k rescheduled userptr restore cycles, the worker holding reclaim-related locks until the system soft-locked up. Stop rescheduling when the failure is -ENODEV: the device is gone for good and the affected processes are being torn down anyway. Other errors keep the existing retry behavior, as they may be transient. Cc: Xiaogang Chen <[email protected]> Signed-off-by: Olivier Kaloudoff <[email protected]> --- Changes in v4: - also stop the svm_range_restore_work() retries on -ENODEV (moved from patch 3, per Felix Kuehling) - assign ret on the evicted_bos-mismatch and confirm_valid_user_pages paths: posted v2 read ret uninitialized on those two paths before the reschedule check - retitle to cover both restore workers Changes in v3: resubmission, unchanged from the v2 already posted. Changes in v2: - Don't pre-initialize ret; every path to the reschedule check is dominated by an assignment (Christian König) - Drop the pr_warn_once; hot removal is a normal abort condition for the worker (Christian König) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c index d668816..2ad6777 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c @@ -2827,6 +2827,7 @@ static void amdgpu_amdkfd_restore_userptr_worker(struct work_struct *work) struct task_struct *usertask; struct mm_struct *mm; uint32_t evicted_bos; + int ret; mutex_lock(&process_info->notifier_lock); evicted_bos = process_info->evicted_bos; @@ -2846,14 +2847,16 @@ static void amdgpu_amdkfd_restore_userptr_worker(struct work_struct *work) mutex_lock(&process_info->lock); - if (update_invalid_user_pages(process_info, mm)) + ret = update_invalid_user_pages(process_info, mm); + if (ret) goto unlock_out; /* userptr_inval_list can be empty if all evicted userptr BOs * have been freed. In that case there is nothing to validate * and we can just restart the queues. */ if (!list_empty(&process_info->userptr_inval_list)) { - if (validate_invalid_user_pages(process_info)) + ret = validate_invalid_user_pages(process_info); + if (ret) goto unlock_out; } /* Final check for concurrent evicton and atomic update. If @@ -2862,11 +2865,14 @@ static void amdgpu_amdkfd_restore_userptr_worker(struct work_struct *work) * reference counting inside KFD will handle this case. */ mutex_lock(&process_info->notifier_lock); - if (process_info->evicted_bos != evicted_bos) + if (process_info->evicted_bos != evicted_bos) { + ret = -EAGAIN; goto unlock_notifier_out; + } if (confirm_valid_user_pages_locked(process_info)) { WARN(1, "User pages unexpectedly invalid"); + ret = -EINVAL; goto unlock_notifier_out; } @@ -2884,8 +2890,10 @@ unlock_notifier_out: unlock_out: mutex_unlock(&process_info->lock); - /* If validation failed, reschedule another attempt */ - if (evicted_bos) { + /* If validation failed, reschedule another attempt, unless the + * device is gone for good + */ + if (evicted_bos && ret != -ENODEV) { queue_delayed_work(system_freezable_wq, &process_info->restore_userptr_work, msecs_to_jiffies(AMDGPU_USERPTR_RESTORE_DELAY_MS)); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index fa4054d..3dab456 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -2013,8 +2013,10 @@ out_reschedule: mmap_write_unlock(mm); mutex_unlock(&process_info->lock); - /* If validation failed, reschedule another attempt */ - if (evicted_ranges) { + /* If validation failed, reschedule another attempt, unless the + * device is gone for good + */ + if (evicted_ranges && r != -ENODEV) { pr_debug("reschedule to restore svm range\n"); queue_delayed_work(system_freezable_wq, &svms->restore_work, msecs_to_jiffies(AMDGPU_SVM_RANGE_RESTORE_DELAY_MS));
