On Thu, Sep 3, 2026 at 6:20 AM Jesse Zhang <[email protected]> wrote: > > A hung SDMA user queue wedges MES, so the MES packet reset times out and > falls back to a full MODE1 reset - once per in-flight job, a reset storm. > > The queue is still on its HW slot at the first hang-detect, so use > detect_hung_queue to recover its slot from the doorbell and reset it over > MMIO, which does not need MES. Mark it HUNG (not UNMAPPED) so the restore > worker does not re-map and re-run the guilty job, and short-circuit the > per-fence hang-detect re-entry once the queue is already reset.
I thought newer versions of MES firmware handled this. Maybe we need a firmware version check? For now, the series is: Reviewed-by: Alex Deucher <[email protected]> > > Signed-off-by: Jesse Zhang <[email protected]> > --- > drivers/gpu/drm/amd/amdgpu/mes_userqueue.c | 48 ++++++++++++++++++---- > 1 file changed, 39 insertions(+), 9 deletions(-) > > diff --git a/drivers/gpu/drm/amd/amdgpu/mes_userqueue.c > b/drivers/gpu/drm/amd/amdgpu/mes_userqueue.c > index 46ebc002548d..2ce94cd04fd5 100644 > --- a/drivers/gpu/drm/amd/amdgpu/mes_userqueue.c > +++ b/drivers/gpu/drm/amd/amdgpu/mes_userqueue.c > @@ -213,26 +213,56 @@ int mes_userq_reset(struct amdgpu_usermode_queue *queue) > struct mes_reset_queue_input queue_input; > int r; > > - /* XXX: add a FW version check for SDMA per queue reset */ > + /* already reset by an earlier job's hang-detect; just signal and > bail */ > + if (queue->state == AMDGPU_USERQ_STATE_HUNG) { > + amdgpu_userq_fence_driver_force_completion(queue); > + return 0; > + } > + > memset(&queue_input, 0x0, sizeof(struct mes_reset_queue_input)); > queue_input.doorbell_offset = queue->doorbell_index; > queue_input.queue_type = queue->queue_type; > > + /* > + * The MES packet reset fails once the hung queue wedges MES. For > SDMA the > + * queue is still on its HW slot, so reset it over MMIO instead: > recover > + * the (instance, queue_id) slot from the doorbell. > + */ > + if (queue->queue_type == AMDGPU_HW_IP_DMA && > + adev->sdma.instance[0].funcs && > + adev->sdma.instance[0].funcs->detect_hung_queue) { > + u32 instance, hw_queue_id; > + > + if (adev->sdma.instance[0].funcs->detect_hung_queue(adev, > + queue->doorbell_index, &instance, > &hw_queue_id)) { > + queue_input.use_mmio = true; > + queue_input.me_id = instance; > + queue_input.queue_id = hw_queue_id; > + } else { > + dev_warn(adev->dev, > + "SDMA userq (doorbell %llu) not on any HW > slot; falling back to MES reset\n", > + queue->doorbell_index); > + } > + } > + > + /* HUNG, not UNMAPPED: the guilty job is still in the ring, so the > + * restore worker must not re-map and re-run it. > + */ > + queue->state = AMDGPU_USERQ_STATE_HUNG; > + > amdgpu_mes_lock(&adev->mes); > r = adev->mes.funcs->reset_hw_queue(&adev->mes, &queue_input); > amdgpu_mes_unlock(&adev->mes); > if (r) > return r; > > - /* mes_userq_unmap() does not update queue->state; mark it UNMAPPED > so the > - * destroy path does not issue a second REMOVE_QUEUE for the removed > queue. > - */ > + /* drop the queue from MES and force-complete its fences */ > r = mes_userq_unmap(queue); > - if (!r) { > - trace_amdgpu_userq_state_changed(queue, > AMDGPU_USERQ_STATE_UNMAPPED); > - queue->state = AMDGPU_USERQ_STATE_UNMAPPED; > - } > - return r; > + if (r) > + return r; > + > + amdgpu_userq_fence_driver_force_completion(queue); > + return 0; > } > > int mes_userq_reset_queue(struct amdgpu_device *adev, > -- > 2.49.0 >
