Hi Tharshan, oh that is a really good point, using amdgpu_vm_wait_idle() here is not correct either because of the same problems as with the context flush.
> dma_resv_wait_timeout(vm->root.bo->tbo.base.resv, DMA_RESV_USAGE_READ, > false,MAX_SCHEDULE_TIMEOUT); That looks like the right thing to do, yes. It only might be necessary to replace DMA_RESV_USAGE_READ with DMA_RESV_USAGE_BOOKKEEP, but I'm not sure if KFD/userqueue suspend resume is actually fully implemented and handling this. Maybe we should stick with DMA_RESV_USAGE_READ for now and I will let my team double check the KFD/userqueu thing later on. Thanks, Christian. On 11/17/26 21:08, Thothiralingam, Logatharshan wrote: > AMD General > > Hi Christian, > > So adding dma_resv_wait_timeout(vm->root.bo->tbo.base.resv, > DMA_RESV_USAGE_READ, false,MAX_SCHEDULE_TIMEOUT); before the > amdgpu_vm_wait_idle() fixes that gfx reset. So it seems like > amdgpu_vm_wait_idle() alone is not enough. > > Attached the patch again. Please let me know if it's good. > > Thanks, > Tharshan > > -----Original Message----- > From: Thothiralingam, Logatharshan > Sent: Tuesday, November 17, 2026 12:52 PM > To: Koenig, Christian <[email protected]>; Limonciello, Mario > <[email protected]> > Cc: [email protected] > Subject: RE: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend > > Hi Chistian, > > Just having the amdgpu_vm_wait_idle() doesn't solve the issue. I tested it > and I do see the reset. I have attached a patch with the > dma_resv_wait_timeout(vm->root.bo->tbo.base.resv, DMA_RESV_USAGE_READ, > false,MAX_SCHEDULE_TIMEOUT); I am building it now to test. Please let me > know your thoughts > > Thanks, > Tharshan > > -----Original Message----- > From: Thothiralingam, Logatharshan > Sent: Tuesday, November 17, 2026 8:39 AM > To: Koenig, Christian <[email protected]>; Limonciello, Mario > <[email protected]> > Cc: [email protected] > Subject: RE: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend > > Hi Chistian, > > I looked a the amdgpu_vm_wait_idle() function and noticed that starting > commit 1f02f20 > (https://github.com/eseiker/kernel-asahi/commit/1f02f2044bda1db1fd995bc35961ab075fa7b5a2) > the wait on the VM's root page table fence list is removed. So correct me > if I am wrong, this alone won't wait on the gfx jobs right? > > So should I add this before the idle call? > > dma_resv_wait_timeout(vm->root.bo->tbo.base.resv, DMA_RESV_USAGE_READ, > false,MAX_SCHEDULE_TIMEOUT); > > > Thanks, > Tharshan > > -----Original Message----- > From: Koenig, Christian <[email protected]> > Sent: Tuesday, November 17, 2026 4:37 AM > To: Limonciello, Mario <[email protected]> > Cc: [email protected]; Thothiralingam, Logatharshan > <[email protected]> > Subject: Re: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend > > On 11/16/26 21:40, Mario Limonciello wrote: >> At suspend, amdgpu_fence_driver_hw_fini() only waits for the last >> fence emitted to each ring. Jobs still queued in a scheduler entity >> have not reached run_job() yet and so have no hardware fence. >> Userspace is frozen at this point, but such jobs can become runnable >> later, for example once a dependency signals, and the scheduler then >> pushes them into the ring after the hardware wait has finished and >> before the IP blocks are torn down. On resume the ring write pointer >> is reset, those jobs never signal and the GFX ring times out and hangs. >> >> This was observed as a rare (about 2 in 400 cycles) GFX hang on S3 >> resume on Picasso. >> >> Flush the context and VM scheduler entities of every open file before >> evicting resources, mirroring what amdgpu_flush() does on file close. >> Like for any other dma_fence there is no timeout, the wait is not >> bounded. >> >> Reported-by: Logatharshan Thothiralingam >> <[email protected]> >> Closes: >> https://lore.kernel.org/amd-gfx/SA1SPRMB0073292C113C6B98D9282092F87F2@ >> SA1SPRMB0073.namprd12.prod.outlook.com/ >> Signed-off-by: Mario Limonciello <[email protected]> >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 26 >> ++++++++++++++++++++++++++ >> 1 file changed, 26 insertions(+) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >> index 19a566187dfb5..d18348116d3f1 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >> @@ -4538,6 +4538,30 @@ void amdgpu_device_complete(struct drm_device *dev) >> } >> } >> >> +/** >> + * amdgpu_device_flush_file_entities - drain scheduler entities of >> +open files >> + * @adev: amdgpu device object >> + * >> + * Wait for the scheduler entities owned by all open files (the >> +context >> + * entities used for command submission and the VM entities used for >> +page >> + * table updates) to run out of queued jobs, so that nothing can slip >> +into >> + * the rings after the hardware fences were waited on at suspend time. >> + */ >> +static void amdgpu_device_flush_file_entities(struct amdgpu_device >> +*adev) { >> + struct drm_device *ddev = adev_to_drm(adev); >> + struct drm_file *file; >> + >> + guard(mutex)(&ddev->filelist_mutex); >> + list_for_each_entry(file, &ddev->filelist, lhead) { >> + struct amdgpu_fpriv *fpriv = file->driver_priv; >> + >> + amdgpu_ctx_mgr_entity_flush(&fpriv->ctx_mgr, >> + MAX_SCHEDULE_TIMEOUT); > > That is actually superfluous and even a little bit dangerous. > > amdgpu_ctx_mgr_entity_flush() is only supposed to be called from the process > which has the file descriptor open and is closing it. > > We use that to detect if that process using the file descriptor was killed > and we should discard queued up work instead of waiting for it to finish > executing. > >> + amdgpu_vm_wait_idle(&fpriv->vm, MAX_SCHEDULE_TIMEOUT); > > That here should be enough to wait for all work to complete. > > Regards, > Christian. > >> + } >> +} >> + >> /** >> * amdgpu_device_suspend - initiate device suspend >> * >> @@ -4587,6 +4611,8 @@ int amdgpu_device_suspend(struct drm_device *dev, bool >> notify_clients) >> if (r) >> goto unwind_ip_phase1; >> >> + amdgpu_device_flush_file_entities(adev); >> + >> r = amdgpu_device_evict_resources(adev); >> if (r) >> goto unwind_userq; >> >> --- >> base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb >> change-id: 20261116-superm1-drain-submissions-17d4414ab3b9 >> >> Best regards, >> -- >> Mario Limonciello <[email protected]> >> >
