amdgpu_amdkfd_restore_userptr_worker() reschedules itself every AMDGPU_USERPTR_RESTORE_DELAY_MS whenever validation of invalidated userptr BOs fails. On PCIe surprise removal the VM update paths already fail fast: amdgpu_vm_update_pdes() and amdgpu_vm_update_range() return -ENODEV when drm_dev_enter() fails, which propagates up through process_update_pds()/update_gpuvm_pte()/validate_invalid_user_pages(). The worker ignores the error code and reschedules unconditionally, producing an endless retry loop on a device that is permanently gone.
Observed on MI50 eGPU TB3 surprise removal: ~11k rescheduled restore cycles, the worker holding reclaim-related locks until the system soft-locked up. Stop rescheduling when the validation failure is -ENODEV: the device is gone for good and the affected processes are being torn down anyway. Other errors keep the existing retry behavior, as they may be transient. Cc: Xiaogang Chen <[email protected]> Signed-off-by: Olivier Kaloudoff <[email protected]> --- Changes in v2: - Don't pre-initialize ret; every path to the reschedule check is dominated by an assignment (Christian König) - Drop the pr_warn_once; hot removal is a normal abort condition for the worker (Christian König) .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c index d668816..dc94d32 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c @@ -2827,6 +2827,7 @@ static void amdgpu_amdkfd_restore_userptr_worker(struct work_struct *work) struct task_struct *usertask; struct mm_struct *mm; uint32_t evicted_bos; + int ret; mutex_lock(&process_info->notifier_lock); evicted_bos = process_info->evicted_bos; @@ -2846,14 +2847,16 @@ static void amdgpu_amdkfd_restore_userptr_worker(struct work_struct *work) mutex_lock(&process_info->lock); - if (update_invalid_user_pages(process_info, mm)) + ret = update_invalid_user_pages(process_info, mm); + if (ret) goto unlock_out; /* userptr_inval_list can be empty if all evicted userptr BOs * have been freed. In that case there is nothing to validate * and we can just restart the queues. */ if (!list_empty(&process_info->userptr_inval_list)) { - if (validate_invalid_user_pages(process_info)) + ret = validate_invalid_user_pages(process_info); + if (ret) goto unlock_out; } /* Final check for concurrent evicton and atomic update. If @@ -2884,8 +2887,8 @@ unlock_notifier_out: unlock_out: mutex_unlock(&process_info->lock); /* If validation failed, reschedule another attempt */ - if (evicted_bos) { + if (evicted_bos && ret != -ENODEV) { queue_delayed_work(system_freezable_wq, &process_info->restore_userptr_work, msecs_to_jiffies(AMDGPU_USERPTR_RESTORE_DELAY_MS)); -- 2.54.0 (Apple Git-157)
