Re: [PATCH] drm/amdgpu/userq: fix indefinite fence wait during GPU reset

[Christian König]

On 7/10/26 12:01, Jesse Zhang wrote:
> pre_reset only force-completes fences of MAPPED queues. A queue in any
> other state (e.g. mid-eviction) keeps its last_fence pending; after a
> GPU reset that fence never signals, so the eviction/suspend worker and
> process teardown (amdgpu_evf_mgr_flush_suspend) wait on it forever and
> wedge the machine:
> 
>   INFO: task kworker/6:28 blocked for more than 120 seconds.
>   Workqueue: events amdgpu_eviction_fence_suspend_worker [amdgpu]
>   Call Trace:
>dma_fence_wait_timeout+0x7e/0x130
>amdgpu_userq_evict+0x67/0x140 [amdgpu]
>amdgpu_eviction_fence_suspend_worker+0xd8/0x160 [amdgpu]
>process_scheduled_works+0xa6/0x420
> 
> Force-complete every queue's fence regardless of state. The unmap and
> mark-hung step stays gated on MAPPED, since unmapping a queue that is
> not mapped is invalid.
> 
> Fixes: 290f46cf5726 ("drm/amdgpu: Implement user queue reset functionality")
> Signed-off-by: Jesse Zhang 

Reviewed-by: Christian König 

> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 25 +--
>  1 file changed, 14 insertions(+), 11 deletions(-)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
> index aa5cc5642e87..6aa75da27f91 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
> @@ -1424,18 +1424,21 @@ void amdgpu_userq_pre_reset(struct amdgpu_device 
> *adev)
>  
>   /* TODO: We probably need a new lock for the queue state */
>   xa_for_each(&adev->userq_doorbell_xa, queue_id, queue) {
> - if (queue->state != AMDGPU_USERQ_STATE_MAPPED)
> - continue;
> -
> - trace_amdgpu_userq_state_start(queue);
> - userq_funcs = adev->userq_funcs[queue->queue_type];
> - userq_funcs->unmap(queue);
> - /* just mark all queues as hung at this point.
> -  * if unmap succeeds, we could map again
> -  * in amdgpu_userq_post_reset() if vram is not lost
> + if (queue->state == AMDGPU_USERQ_STATE_MAPPED) {
> + trace_amdgpu_userq_state_start(queue);
> + userq_funcs = adev->userq_funcs[queue->queue_type];
> + userq_funcs->unmap(queue);
> + /* just mark all queues as hung at this point.
> +  * if unmap succeeds, we could map again
> +  * in amdgpu_userq_post_reset() if vram is not lost
> +  */
> + trace_amdgpu_userq_state_changed(queue, 
> AMDGPU_USERQ_STATE_HUNG);
> + queue->state = AMDGPU_USERQ_STATE_HUNG;
> + }
> + /* Force-complete any pending fence regardless of queue state so
> +  * that eviction/suspend and queue teardown waiters don't block
> +  * forever on a fence that will never signal after the reset.
>*/
> - trace_amdgpu_userq_state_changed(queue, 
> AMDGPU_USERQ_STATE_HUNG);
> - queue->state = AMDGPU_USERQ_STATE_HUNG;
>   amdgpu_userq_fence_driver_force_completion(queue);
>   }
>  }

[PATCH] drm/amdgpu/userq: fix indefinite fence wait during GPU reset

[Jesse Zhang]

pre_reset only force-completes fences of MAPPED queues. A queue in any
other state (e.g. mid-eviction) keeps its last_fence pending; after a
GPU reset that fence never signals, so the eviction/suspend worker and
process teardown (amdgpu_evf_mgr_flush_suspend) wait on it forever and
wedge the machine:

  INFO: task kworker/6:28 blocked for more than 120 seconds.
  Workqueue: events amdgpu_eviction_fence_suspend_worker [amdgpu]
  Call Trace:
   dma_fence_wait_timeout+0x7e/0x130
   amdgpu_userq_evict+0x67/0x140 [amdgpu]
   amdgpu_eviction_fence_suspend_worker+0xd8/0x160 [amdgpu]
   process_scheduled_works+0xa6/0x420

Force-complete every queue's fence regardless of state. The unmap and
mark-hung step stays gated on MAPPED, since unmapping a queue that is
not mapped is invalid.

Fixes: 290f46cf5726 ("drm/amdgpu: Implement user queue reset functionality")
Signed-off-by: Jesse Zhang 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 25 +--
 1 file changed, 14 insertions(+), 11 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
index aa5cc5642e87..6aa75da27f91 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
@@ -1424,18 +1424,21 @@ void amdgpu_userq_pre_reset(struct amdgpu_device *adev)
 
/* TODO: We probably need a new lock for the queue state */
xa_for_each(&adev->userq_doorbell_xa, queue_id, queue) {
-   if (queue->state != AMDGPU_USERQ_STATE_MAPPED)
-   continue;
-
-   trace_amdgpu_userq_state_start(queue);
-   userq_funcs = adev->userq_funcs[queue->queue_type];
-   userq_funcs->unmap(queue);
-   /* just mark all queues as hung at this point.
-* if unmap succeeds, we could map again
-* in amdgpu_userq_post_reset() if vram is not lost
+   if (queue->state == AMDGPU_USERQ_STATE_MAPPED) {
+   trace_amdgpu_userq_state_start(queue);
+   userq_funcs = adev->userq_funcs[queue->queue_type];
+   userq_funcs->unmap(queue);
+   /* just mark all queues as hung at this point.
+* if unmap succeeds, we could map again
+* in amdgpu_userq_post_reset() if vram is not lost
+*/
+   trace_amdgpu_userq_state_changed(queue, 
AMDGPU_USERQ_STATE_HUNG);
+   queue->state = AMDGPU_USERQ_STATE_HUNG;
+   }
+   /* Force-complete any pending fence regardless of queue state so
+* that eviction/suspend and queue teardown waiters don't block
+* forever on a fence that will never signal after the reset.
 */
-   trace_amdgpu_userq_state_changed(queue, 
AMDGPU_USERQ_STATE_HUNG);
-   queue->state = AMDGPU_USERQ_STATE_HUNG;
amdgpu_userq_fence_driver_force_completion(queue);
}
 }
-- 
2.49.0