AMD General @Deucher, Alexander @Lazar, Lijo
Could you help review this patch? Thanks River -----Original Message----- From: amd-gfx <[email protected]> On Behalf Of YuanShang Mao (River) Sent: Monday, August 24, 2026 2:40 PM To: Deucher, Alexander <[email protected]> Cc: Zhang, Tiantian (Celine) <[email protected]>; [email protected] Subject: RE: [PATCH] drm/amdkfd: mark queues as reset after a full GPU reset AMD General @Deucher, Alexander Could you help review this patch? Thanks River -----Original Message----- From: YuanShang Mao (River) <[email protected]> Sent: Thursday, August 20, 2026 6:56 PM To: [email protected] Cc: YuanShang Mao (River) <[email protected]> Subject: [PATCH] drm/amdkfd: mark queues as reset after a full GPU reset From: Yuansheng Mao <[email protected]> A full GPU reset reloads the MES firmware and wipes its queue table, but KFD keeps the queue objects with is_reset left false. Pre-reset cleanup cannot fix this: it runs with the reset domain write lock held, so remove_queue_mes() fails with -EIO and the cleanup loops bail out early. When the process later exits, process_termination_cpsch() sends REMOVE_QUEUE for doorbells MES no longer knows about. MES rejects them, remove_queue_mes_on_reset_option() mistakes the rejection for a hung queue and ends up in kfd_hws_hang(), requesting another GPU reset. On SR-IOV that resets the whole card, so one injected hang costs two resets and disturbs the other VFs. Observed on Navi32 SR-IOV with a compute workload running: ring sdma1 timeout, signaled seq=239, emitted seq=240 GPU reset begin!. Source: 1 Failed to evict queue 2 GPU reset(1) succeeded! [ ~34s later, when the killed process is reaped ] failed to remove queue from MES, doorbell=0x1002 MES might be in unrecoverable state, issue a GPU reset GPU reset begin!. Source: 3 remove_queue_mes_on_reset_option() already skips queues with is_reset set, but only the per-queue reset paths ever set it. Set it in kgd2kfd_post_reset() as well. Suspend/resume is unaffected: it goes through kgd2kfd_resume(), where queues are re-added to MES. Signed-off-by: Yuansheng Mao <[email protected]> --- drivers/gpu/drm/amd/amdkfd/kfd_device.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c index 586e640f13dc..d326e41a8656 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c @@ -1033,6 +1033,27 @@ int kgd2kfd_post_reset(struct kfd_dev *kfd) if (!kfd->init_complete) return 0; + /* + * The reset wiped the MES queue table. Mark the queues as reset so + * that teardown does not try to remove them from MES again. + */ + if (kfd->shared_resources.enable_mes) { + for (i = 0; i < kfd->num_nodes; i++) { + struct device_queue_manager *dqm = kfd->nodes[i]->dqm; + struct device_process_node *cur; + struct qcm_process_device *qpd; + struct queue *q; + + dqm_lock(dqm); + list_for_each_entry(cur, &dqm->queues, list) { + qpd = cur->qpd; + list_for_each_entry(q, &qpd->queues_list, list) + q->properties.is_reset = true; + } + dqm_unlock(dqm); + } + } + for (i = 0; i < kfd->num_nodes; i++) { ret = kfd_resume(kfd->nodes[i]); if (ret) -- 2.25.1
