AMD General

Hi Christian,

You are right, amdgpu_vm_wait_idle() is not needed. I was thinking of that 
before, but I added it in the previous patch per your request. But now, I have 
removed it.

Please see the attached patch. I have tested it, and the hang is not seen. 
Please let me know if the patch is good to go so that I can share the solution 
with our customer.

Then, I can create another patch against upstream and work on getting it pushed.

Thanks,
Tharshan
-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, November 18, 2026 5:15 AM
To: Thothiralingam, Logatharshan <[email protected]>; 
Limonciello, Mario <[email protected]>
Cc: [email protected]
Subject: Re: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend

Hi Tharshan,

oh that is a really good point, using amdgpu_vm_wait_idle() here is not correct 
either because of the same problems as with the context flush.

> dma_resv_wait_timeout(vm->root.bo->tbo.base.resv, DMA_RESV_USAGE_READ,
> false,MAX_SCHEDULE_TIMEOUT);

That looks like the right thing to do, yes.

It only might be necessary to replace DMA_RESV_USAGE_READ with 
DMA_RESV_USAGE_BOOKKEEP, but I'm not sure if KFD/userqueue suspend resume is 
actually fully implemented and handling this.

Maybe we should stick with DMA_RESV_USAGE_READ for now and I will let my team 
double check the KFD/userqueu thing later on.

Thanks,
Christian.

On 11/17/26 21:08, Thothiralingam, Logatharshan wrote:
> AMD General
>
> Hi Christian,
>
> So adding dma_resv_wait_timeout(vm->root.bo->tbo.base.resv, 
> DMA_RESV_USAGE_READ, false,MAX_SCHEDULE_TIMEOUT); before the 
> amdgpu_vm_wait_idle() fixes that gfx reset. So it seems like 
> amdgpu_vm_wait_idle() alone is not enough.
>
> Attached the patch again. Please let me know if it's good.
>
> Thanks,
> Tharshan
>
> -----Original Message-----
> From: Thothiralingam, Logatharshan
> Sent: Tuesday, November 17, 2026 12:52 PM
> To: Koenig, Christian <[email protected]>; Limonciello, Mario
> <[email protected]>
> Cc: [email protected]
> Subject: RE: [PATCH] drm/amdgpu: drain queued scheduler jobs before
> suspend
>
> Hi Chistian,
>
> Just having the amdgpu_vm_wait_idle() doesn't solve the issue. I
> tested it and I do see the reset. I have attached a patch with the
> dma_resv_wait_timeout(vm->root.bo->tbo.base.resv, DMA_RESV_USAGE_READ,
> false,MAX_SCHEDULE_TIMEOUT); I am building it now to test.  Please let
> me know your thoughts
>
> Thanks,
> Tharshan
>
> -----Original Message-----
> From: Thothiralingam, Logatharshan
> Sent: Tuesday, November 17, 2026 8:39 AM
> To: Koenig, Christian <[email protected]>; Limonciello, Mario
> <[email protected]>
> Cc: [email protected]
> Subject: RE: [PATCH] drm/amdgpu: drain queued scheduler jobs before
> suspend
>
> Hi Chistian,
>
> I looked a the amdgpu_vm_wait_idle() function and noticed that starting 
> commit 1f02f20 
> (https://github.com/eseiker/kernel-asahi/commit/1f02f2044bda1db1fd995bc35961ab075fa7b5a2)
>   the wait on the VM's root page table fence list is removed. So correct me 
> if I am wrong, this alone won't wait on the gfx jobs right?
>
> So should I add this before the idle call?
>
> dma_resv_wait_timeout(vm->root.bo->tbo.base.resv, DMA_RESV_USAGE_READ,
> false,MAX_SCHEDULE_TIMEOUT);
>
>
> Thanks,
> Tharshan
>
> -----Original Message-----
> From: Koenig, Christian <[email protected]>
> Sent: Tuesday, November 17, 2026 4:37 AM
> To: Limonciello, Mario <[email protected]>
> Cc: [email protected]; Thothiralingam, Logatharshan
> <[email protected]>
> Subject: Re: [PATCH] drm/amdgpu: drain queued scheduler jobs before
> suspend
>
> On 11/16/26 21:40, Mario Limonciello wrote:
>> At suspend, amdgpu_fence_driver_hw_fini() only waits for the last
>> fence emitted to each ring. Jobs still queued in a scheduler entity
>> have not reached run_job() yet and so have no hardware fence.
>> Userspace is frozen at this point, but such jobs can become runnable
>> later, for example once a dependency signals, and the scheduler then
>> pushes them into the ring after the hardware wait has finished and
>> before the IP blocks are torn down. On resume the ring write pointer
>> is reset, those jobs never signal and the GFX ring times out and hangs.
>>
>> This was observed as a rare (about 2 in 400 cycles) GFX hang on S3
>> resume on Picasso.
>>
>> Flush the context and VM scheduler entities of every open file before
>> evicting resources, mirroring what amdgpu_flush() does on file close.
>> Like for any other dma_fence there is no timeout, the wait is not
>> bounded.
>>
>> Reported-by: Logatharshan Thothiralingam
>> <[email protected]>
>> Closes:
>> https://lore.kernel.org/amd-gfx/SA1SPRMB0073292C113C6B98D9282092F87F2
>> @ SA1SPRMB0073.namprd12.prod.outlook.com/
>> Signed-off-by: Mario Limonciello <[email protected]>
>> ---
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 26
>> ++++++++++++++++++++++++++
>>  1 file changed, 26 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> index 19a566187dfb5..d18348116d3f1 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> @@ -4538,6 +4538,30 @@ void amdgpu_device_complete(struct drm_device *dev)
>>       }
>>  }
>>
>> +/**
>> + * amdgpu_device_flush_file_entities - drain scheduler entities of
>> +open files
>> + * @adev: amdgpu device object
>> + *
>> + * Wait for the scheduler entities owned by all open files (the
>> +context
>> + * entities used for command submission and the VM entities used for
>> +page
>> + * table updates) to run out of queued jobs, so that nothing can
>> +slip into
>> + * the rings after the hardware fences were waited on at suspend time.
>> + */
>> +static void amdgpu_device_flush_file_entities(struct amdgpu_device
>> +*adev) {
>> +     struct drm_device *ddev = adev_to_drm(adev);
>> +     struct drm_file *file;
>> +
>> +     guard(mutex)(&ddev->filelist_mutex);
>> +     list_for_each_entry(file, &ddev->filelist, lhead) {
>> +             struct amdgpu_fpriv *fpriv = file->driver_priv;
>> +
>> +             amdgpu_ctx_mgr_entity_flush(&fpriv->ctx_mgr,
>> +                                         MAX_SCHEDULE_TIMEOUT);
>
> That is actually superfluous and even a little bit dangerous.
>
> amdgpu_ctx_mgr_entity_flush() is only supposed to be called from the process 
> which has the file descriptor open and is closing it.
>
> We use that to detect if that process using the file descriptor was killed 
> and we should discard queued up work instead of waiting for it to finish 
> executing.
>
>> +             amdgpu_vm_wait_idle(&fpriv->vm, MAX_SCHEDULE_TIMEOUT);
>
> That here should be enough to wait for all work to complete.
>
> Regards,
> Christian.
>
>> +     }
>> +}
>> +
>>  /**
>>   * amdgpu_device_suspend - initiate device suspend
>>   *
>> @@ -4587,6 +4611,8 @@ int amdgpu_device_suspend(struct drm_device *dev, bool 
>> notify_clients)
>>       if (r)
>>               goto unwind_ip_phase1;
>>
>> +     amdgpu_device_flush_file_entities(adev);
>> +
>>       r = amdgpu_device_evict_resources(adev);
>>       if (r)
>>               goto unwind_userq;
>>
>> ---
>> base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb
>> change-id: 20261116-superm1-drain-submissions-17d4414ab3b9
>>
>> Best regards,
>> --
>> Mario Limonciello <[email protected]>
>>
>

Attachment: 0001-drm-amdgpu-wait-for-queued-user-submissions-at-suspe.patch
Description: 0001-drm-amdgpu-wait-for-queued-user-submissions-at-suspe.patch

Reply via email to