Hi Tharshan,

oh that is a really good point, using amdgpu_vm_wait_idle() here is not correct 
either because of the same problems as with the context flush.

> dma_resv_wait_timeout(vm->root.bo->tbo.base.resv, DMA_RESV_USAGE_READ, 
> false,MAX_SCHEDULE_TIMEOUT);

That looks like the right thing to do, yes.

It only might be necessary to replace DMA_RESV_USAGE_READ with 
DMA_RESV_USAGE_BOOKKEEP, but I'm not sure if KFD/userqueue suspend resume is 
actually fully implemented and handling this.

Maybe we should stick with DMA_RESV_USAGE_READ for now and I will let my team 
double check the KFD/userqueu thing later on.

Thanks,
Christian.

On 11/17/26 21:08, Thothiralingam, Logatharshan wrote:
> AMD General
> 
> Hi Christian,
> 
> So adding dma_resv_wait_timeout(vm->root.bo->tbo.base.resv, 
> DMA_RESV_USAGE_READ, false,MAX_SCHEDULE_TIMEOUT); before the 
> amdgpu_vm_wait_idle() fixes that gfx reset. So it seems like 
> amdgpu_vm_wait_idle() alone is not enough.
> 
> Attached the patch again. Please let me know if it's good.
> 
> Thanks,
> Tharshan
> 
> -----Original Message-----
> From: Thothiralingam, Logatharshan
> Sent: Tuesday, November 17, 2026 12:52 PM
> To: Koenig, Christian <[email protected]>; Limonciello, Mario 
> <[email protected]>
> Cc: [email protected]
> Subject: RE: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend
> 
> Hi Chistian,
> 
> Just having the amdgpu_vm_wait_idle() doesn't solve the issue. I tested it 
> and I do see the reset. I have attached a patch with the 
> dma_resv_wait_timeout(vm->root.bo->tbo.base.resv, DMA_RESV_USAGE_READ, 
> false,MAX_SCHEDULE_TIMEOUT); I am building it now to test.  Please let me 
> know your thoughts
> 
> Thanks,
> Tharshan
> 
> -----Original Message-----
> From: Thothiralingam, Logatharshan
> Sent: Tuesday, November 17, 2026 8:39 AM
> To: Koenig, Christian <[email protected]>; Limonciello, Mario 
> <[email protected]>
> Cc: [email protected]
> Subject: RE: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend
> 
> Hi Chistian,
> 
> I looked a the amdgpu_vm_wait_idle() function and noticed that starting 
> commit 1f02f20 
> (https://github.com/eseiker/kernel-asahi/commit/1f02f2044bda1db1fd995bc35961ab075fa7b5a2)
>   the wait on the VM's root page table fence list is removed. So correct me 
> if I am wrong, this alone won't wait on the gfx jobs right?
> 
> So should I add this before the idle call?
> 
> dma_resv_wait_timeout(vm->root.bo->tbo.base.resv, DMA_RESV_USAGE_READ, 
> false,MAX_SCHEDULE_TIMEOUT);
> 
> 
> Thanks,
> Tharshan
> 
> -----Original Message-----
> From: Koenig, Christian <[email protected]>
> Sent: Tuesday, November 17, 2026 4:37 AM
> To: Limonciello, Mario <[email protected]>
> Cc: [email protected]; Thothiralingam, Logatharshan 
> <[email protected]>
> Subject: Re: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend
> 
> On 11/16/26 21:40, Mario Limonciello wrote:
>> At suspend, amdgpu_fence_driver_hw_fini() only waits for the last
>> fence emitted to each ring. Jobs still queued in a scheduler entity
>> have not reached run_job() yet and so have no hardware fence.
>> Userspace is frozen at this point, but such jobs can become runnable
>> later, for example once a dependency signals, and the scheduler then
>> pushes them into the ring after the hardware wait has finished and
>> before the IP blocks are torn down. On resume the ring write pointer
>> is reset, those jobs never signal and the GFX ring times out and hangs.
>>
>> This was observed as a rare (about 2 in 400 cycles) GFX hang on S3
>> resume on Picasso.
>>
>> Flush the context and VM scheduler entities of every open file before
>> evicting resources, mirroring what amdgpu_flush() does on file close.
>> Like for any other dma_fence there is no timeout, the wait is not
>> bounded.
>>
>> Reported-by: Logatharshan Thothiralingam
>> <[email protected]>
>> Closes:
>> https://lore.kernel.org/amd-gfx/SA1SPRMB0073292C113C6B98D9282092F87F2@
>> SA1SPRMB0073.namprd12.prod.outlook.com/
>> Signed-off-by: Mario Limonciello <[email protected]>
>> ---
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 26
>> ++++++++++++++++++++++++++
>>  1 file changed, 26 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> index 19a566187dfb5..d18348116d3f1 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> @@ -4538,6 +4538,30 @@ void amdgpu_device_complete(struct drm_device *dev)
>>       }
>>  }
>>
>> +/**
>> + * amdgpu_device_flush_file_entities - drain scheduler entities of
>> +open files
>> + * @adev: amdgpu device object
>> + *
>> + * Wait for the scheduler entities owned by all open files (the
>> +context
>> + * entities used for command submission and the VM entities used for
>> +page
>> + * table updates) to run out of queued jobs, so that nothing can slip
>> +into
>> + * the rings after the hardware fences were waited on at suspend time.
>> + */
>> +static void amdgpu_device_flush_file_entities(struct amdgpu_device
>> +*adev) {
>> +     struct drm_device *ddev = adev_to_drm(adev);
>> +     struct drm_file *file;
>> +
>> +     guard(mutex)(&ddev->filelist_mutex);
>> +     list_for_each_entry(file, &ddev->filelist, lhead) {
>> +             struct amdgpu_fpriv *fpriv = file->driver_priv;
>> +
>> +             amdgpu_ctx_mgr_entity_flush(&fpriv->ctx_mgr,
>> +                                         MAX_SCHEDULE_TIMEOUT);
> 
> That is actually superfluous and even a little bit dangerous.
> 
> amdgpu_ctx_mgr_entity_flush() is only supposed to be called from the process 
> which has the file descriptor open and is closing it.
> 
> We use that to detect if that process using the file descriptor was killed 
> and we should discard queued up work instead of waiting for it to finish 
> executing.
> 
>> +             amdgpu_vm_wait_idle(&fpriv->vm, MAX_SCHEDULE_TIMEOUT);
> 
> That here should be enough to wait for all work to complete.
> 
> Regards,
> Christian.
> 
>> +     }
>> +}
>> +
>>  /**
>>   * amdgpu_device_suspend - initiate device suspend
>>   *
>> @@ -4587,6 +4611,8 @@ int amdgpu_device_suspend(struct drm_device *dev, bool 
>> notify_clients)
>>       if (r)
>>               goto unwind_ip_phase1;
>>
>> +     amdgpu_device_flush_file_entities(adev);
>> +
>>       r = amdgpu_device_evict_resources(adev);
>>       if (r)
>>               goto unwind_userq;
>>
>> ---
>> base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb
>> change-id: 20261116-superm1-drain-submissions-17d4414ab3b9
>>
>> Best regards,
>> --
>> Mario Limonciello <[email protected]>
>>
> 

Reply via email to