On 11/16/26 22:21, Mario Limonciello wrote: > On 11/16/26 15:13, [email protected] wrote: >> Thank you for your contribution! Sashiko AI review found 2 potential >> issues to consider: >> - [High] drm/amdgpu: deadlock in amdgpu_device_flush_file_entities() on >> suspend >> - [Medium] drm/amdgpu: race in amdgpu_device_flush_file_entities() >> before run_job() >> -- >> >> commit 2d3632cb05b8d04352d9590da3fabb5b096f1b19 >> Author: Mario Limonciello <[email protected]> >> >> drm/amdgpu: drain queued scheduler jobs before suspend >> >> This commit adds amdgpu_device_flush_file_entities() to flush the context >> and VM scheduler entities of all open DRM files during >> amdgpu_device_suspend() before evicting resources. >> >>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >>> index 19a566187dfb5..d18348116d3f1 100644 >>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >>> @@ -4538,6 +4538,30 @@ void amdgpu_device_complete(struct drm_device *dev) >>> } >>> } >>> +/** >>> + * amdgpu_device_flush_file_entities - drain scheduler entities of open >>> files >>> + * @adev: amdgpu device object >>> + * >>> + * Wait for the scheduler entities owned by all open files (the context >>> + * entities used for command submission and the VM entities used for page >>> + * table updates) to run out of queued jobs, so that nothing can slip into >>> + * the rings after the hardware fences were waited on at suspend time. >>> + */ >>> +static void amdgpu_device_flush_file_entities(struct amdgpu_device *adev) >>> +{ >>> + struct drm_device *ddev = adev_to_drm(adev); >>> + struct drm_file *file; >>> + >>> + guard(mutex)(&ddev->filelist_mutex); >>> + list_for_each_entry(file, &ddev->filelist, lhead) { >>> + struct amdgpu_fpriv *fpriv = file->driver_priv; >>> + >>> + amdgpu_ctx_mgr_entity_flush(&fpriv->ctx_mgr, >>> + MAX_SCHEDULE_TIMEOUT); >>> + amdgpu_vm_wait_idle(&fpriv->vm, MAX_SCHEDULE_TIMEOUT); >>> + } >>> +} >> >> [Severity: Medium] >> Can drm_sched_entity_flush() return while the scheduler worker is still in >> the middle of popping a job and before sched->ops->run_job() has emitted its >> hardware fence? >> >> CPU0 (submit_wq) CPU1 (suspend) >> drm_sched_run_job_work() >> drm_sched_entity_pop_job() >> spsc_queue_pop(&entity->job_queue) >> drm_sched_rq_pop_entity(entity) >> amdgpu_device_suspend() >> amdgpu_device_flush_file_entities() >> ... >> drm_sched_entity_flush() >> drm_sched_entity_is_idle() == true >> amdgpu_device_evict_resources() >> amdgpu_fence_driver_hw_fini() >> amdgpu_fence_wait_empty() >> sched->ops->run_job(sched_job) >> amdgpu_job_run() >> amdgpu_ib_schedule() >> amdgpu_fence_emit() >> amdgpu_ring_commit() <-- ring written after fence wait >> complete_all(&entity->entity_idle) >> >> drm_sched_entity_is_idle() reports the entity idle as soon as >> spsc_queue_pop() empties entity->job_queue, before drm_sched_run_job_work() >> calls sched->ops->run_job() and complete_all(&entity->entity_idle). On APUs >> such as Picasso where amdgpu_device_evict_resources() returns immediately >> during S3 suspend, amdgpu_fence_driver_hw_fini() can finish waiting on the >> ring before the popped job emits its fence and commits the ring write >> pointer, leaving the ring hang on resume window open. >> >>> + >>> /** >>> * amdgpu_device_suspend - initiate device suspend >>> * >>> @@ -4587,6 +4611,8 @@ int amdgpu_device_suspend(struct drm_device *dev, >>> bool notify_clients) >>> if (r) >>> goto unwind_ip_phase1; >>> + amdgpu_device_flush_file_entities(adev); >> >> [Severity: High] >> Can calling amdgpu_device_flush_file_entities() here deadlock runtime or >> system suspend when an entity has a job waiting on an unsignaled dependency? >> >> drivers/gpu/drm/scheduler/sched_entity.c:drm_sched_entity_flush() { >> ... >> if (current->flags & PF_EXITING) { >> if (timeout) >> ret = wait_event_timeout( >> sched->job_scheduled, >> drm_sched_entity_is_idle(entity), >> timeout); >> } else { >> wait_event_killable(sched->job_scheduled, >> drm_sched_entity_is_idle(entity)); >> } >> ... >> } >> >> Because the suspend task or runtime PM worker does not have PF_EXITING set, >> drm_sched_entity_flush() ignores the timeout and sleeps indefinitely in >> wait_event_killable() while holding ddev->filelist_mutex. >> >> In amdgpu_cs_ioctl(), a runtime PM reference is only held during the ioctl >> itself, and per-job runtime PM references are only taken in >> amdgpu_fence_emit() when a job reaches run_job(). If a client submits a job >> with an unsignaled syncobj dependency, runtime autosuspend can enter >> amdgpu_pmops_runtime_suspend() -> amdgpu_device_suspend() and block in >> amdgpu_device_flush_file_entities() while the device status is >> RPM_SUSPENDING. When another thread or process then calls amdgpu_cs_ioctl() >> to submit the work that would signal that syncobj, its runtime PM resume >> waits for RPM_SUSPENDING to complete, deadlocking both tasks. >> >> Killing the process owning the blocked entity does not break the wait >> either: amdgpu_flush() times out after 2 seconds and calls >> drm_sched_entity_kill(), which drains the entity without waking >> sched->job_scheduled, and the exiting task then blocks on >> ddev->filelist_mutex in drm_close_helper(). During system suspend, >> amdgpu_userq_suspend() also runs right before >> amdgpu_device_flush_file_entities() and unmaps usermode queues while >> userspace is already frozen, so any entity job waiting on a usermode queue >> or userspace fence can never make progress. >> > > I'd say that these findings both look real and this being able to potentially > deadlock the system means it's not the right solution.
That's just AI nonsense. All dma_fences *MUST* be signaled before suspending and runtime PM shouldn't be able to suspend the HW while there is still work queued up. What could be is that runtime PM is not correctly implemented here, but that is an independent problem to what we try to fix here. > How about instead we stop schedulers with drm_sched_wqueue_stop() on each of > the rings? Restart then in any unwind and resume paths. That's kinda like > how SDMA already works from what I can tell. Suspending with pending work is completely broken and can trivially deadlock. So we should never call drm_sched_wqueue_stop() from the suspend path. What we could do is to issue a warning from the run_job callback when we are already suspended. Regards, Christian.
