AMD General Hi Alex,
Checked amdgpu_userq.c and mes_userq_unmap() calls remove_hw_queue without TLB flush. I'll add amdgpu_gmc_flush_gpu_tlb_pasid() in amdgpu_userq_unmap_helper() and send v3 as a 2-patch series: Patch 1/2: KFD TLB flush fix (kfd_device_queue_manager.c) Patch 2/2: userq TLB flush fix (amdgpu_userq.c) Thanks and Regards Priya Hosur -----Original Message----- From: amd-gfx <[email protected]> On Behalf Of Alex Deucher Sent: Friday, August 14, 2026 12:33 AM To: Hosur, Priya <[email protected]>; Khatri, Sunil <[email protected]> Cc: [email protected]; Liu, Shaoyun <[email protected]>; Lazar, Lijo <[email protected]>; Deucher, Alexander <[email protected]>; Limonciello, Mario <[email protected]>; Koenig, Christian <[email protected]>; Vishwakarma, Pratik <[email protected]>; Gopalakrishnan, Veerabadhran (Veera) <[email protected]>; Kuehling, Felix <[email protected]> Subject: Re: [PATCH v2 1/1] drm/amdkfd: Add TLB flush after MES queue eviction/suspension + Sunil On Thu, Aug 13, 2026 at 1:50 AM Priya Hosur <[email protected]> wrote: > > MES (Micro Engine Scheduler) does not perform heavy-weight TLB > invalidation after unmapping queues, unlike HWS which does this > automatically. This causes a race condition where in-flight DMA > descriptors can access memory that has been unmapped, leading to page > faults and GPU queue hangs during SVM page migration. > > The issue manifests as KFDSVMRangeTest.MultiThreadMigrationTest/1 > failures on gfx1151 (Ryzen AI MAX) with XNACK mode 1 enabled - the GPU > compute queue hangs with packets submitted but never consumed. > > Add kfd_flush_tlb() with TLB_FLUSH_HEAVYWEIGHT in two MES code paths: > 1. evict_process_queues_cpsch() - after MES removes queues 2. > suspend_queues() - after MES suspends queues and mem_fence completes > > This ensures all in-flight memory accesses from unmapped queues are > flushed before memory is freed or migrated. Can you check amdgpu_userq.c as well? I suspect it needs something similar when unmapping. Thanks, Alex > > Testing on gfx1151 shows this reduces failure rate from 100% to > approximately 7-10%. The residual failures require further investigation. > > Change-Id: Ia83e6af7a66fe2bff56cb62c741728e9a18883e8 > Signed-off-by: Priya Hosur <[email protected]> > Reviewed-by: Felix Kuehling <[email protected]> > --- > .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.c | 13 ++++++++++++- > 1 file changed, 12 insertions(+), 1 deletion(-) > > diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c > b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c > index a23384571193..71c60a42c581 100644 > --- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c > +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c > @@ -1450,6 +1450,14 @@ static int evict_process_queues_cpsch(struct > device_queue_manager *dqm, > dqm_evict_mqd_bo(dqm, q); > } > > + /* > + * Heavy-weight TLB flush after MES removes queues to ensure > + * in-flight memory accesses complete before memory is freed/migrated. > + * HWS does this automatically, MES does not. > + */ > + if (dqm->dev->kfd->shared_resources.enable_mes) > + kfd_flush_tlb(pdd, TLB_FLUSH_HEAVYWEIGHT); > + > if (!dqm->dev->kfd->shared_resources.enable_mes) { > pdd->last_evict_timestamp = get_jiffies_64(); > retval = execute_queues_cpsch(dqm, @@ -3736,8 +3744,11 > @@ int suspend_queues(struct kfd_process *p, > if (!per_device_suspended) { > dqm_unlock(dqm); > mutex_unlock(&p->event_mutex); > - if (total_suspended) > + if (total_suspended) { > > amdgpu_amdkfd_debug_mem_fence(dqm->dev->adev); > + /* Heavy-weight TLB flush after MES suspends > queues */ > + kfd_flush_tlb(pdd, TLB_FLUSH_HEAVYWEIGHT); > + } > continue; > } > > -- > 2.43.0 >
