Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [High] drm/amdgpu: deadlock in amdgpu_device_flush_file_entities() on
  suspend
- [Medium] drm/amdgpu: race in amdgpu_device_flush_file_entities()
  before run_job()
--

commit 2d3632cb05b8d04352d9590da3fabb5b096f1b19
Author: Mario Limonciello <[email protected]>

drm/amdgpu: drain queued scheduler jobs before suspend

This commit adds amdgpu_device_flush_file_entities() to flush the context
and VM scheduler entities of all open DRM files during
amdgpu_device_suspend() before evicting resources.

> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 19a566187dfb5..d18348116d3f1 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -4538,6 +4538,30 @@ void amdgpu_device_complete(struct drm_device *dev)
>       }
>  }
>  
> +/**
> + * amdgpu_device_flush_file_entities - drain scheduler entities of open files
> + * @adev: amdgpu device object
> + *
> + * Wait for the scheduler entities owned by all open files (the context
> + * entities used for command submission and the VM entities used for page
> + * table updates) to run out of queued jobs, so that nothing can slip into
> + * the rings after the hardware fences were waited on at suspend time.
> + */
> +static void amdgpu_device_flush_file_entities(struct amdgpu_device *adev)
> +{
> +     struct drm_device *ddev = adev_to_drm(adev);
> +     struct drm_file *file;
> +
> +     guard(mutex)(&ddev->filelist_mutex);
> +     list_for_each_entry(file, &ddev->filelist, lhead) {
> +             struct amdgpu_fpriv *fpriv = file->driver_priv;
> +
> +             amdgpu_ctx_mgr_entity_flush(&fpriv->ctx_mgr,
> +                                         MAX_SCHEDULE_TIMEOUT);
> +             amdgpu_vm_wait_idle(&fpriv->vm, MAX_SCHEDULE_TIMEOUT);
> +     }
> +}

[Severity: Medium]
Can drm_sched_entity_flush() return while the scheduler worker is still in
the middle of popping a job and before sched->ops->run_job() has emitted its
hardware fence?

CPU0 (submit_wq)                  CPU1 (suspend)
drm_sched_run_job_work()
  drm_sched_entity_pop_job()
    spsc_queue_pop(&entity->job_queue)
    drm_sched_rq_pop_entity(entity)
                                  amdgpu_device_suspend()
                                    amdgpu_device_flush_file_entities()
                                      ...
                                        drm_sched_entity_flush()
                                          drm_sched_entity_is_idle() == true
                                    amdgpu_device_evict_resources()
                                    amdgpu_fence_driver_hw_fini()
                                      amdgpu_fence_wait_empty()
  sched->ops->run_job(sched_job)
    amdgpu_job_run()
      amdgpu_ib_schedule()
        amdgpu_fence_emit()
        amdgpu_ring_commit()  <-- ring written after fence wait
  complete_all(&entity->entity_idle)

drm_sched_entity_is_idle() reports the entity idle as soon as
spsc_queue_pop() empties entity->job_queue, before drm_sched_run_job_work()
calls sched->ops->run_job() and complete_all(&entity->entity_idle).  On APUs
such as Picasso where amdgpu_device_evict_resources() returns immediately
during S3 suspend, amdgpu_fence_driver_hw_fini() can finish waiting on the
ring before the popped job emits its fence and commits the ring write
pointer, leaving the ring hang on resume window open.

> +
>  /**
>   * amdgpu_device_suspend - initiate device suspend
>   *
> @@ -4587,6 +4611,8 @@ int amdgpu_device_suspend(struct drm_device *dev, bool 
> notify_clients)
>       if (r)
>               goto unwind_ip_phase1;
>  
> +     amdgpu_device_flush_file_entities(adev);

[Severity: High]
Can calling amdgpu_device_flush_file_entities() here deadlock runtime or
system suspend when an entity has a job waiting on an unsignaled dependency?

drivers/gpu/drm/scheduler/sched_entity.c:drm_sched_entity_flush() {
        ...
        if (current->flags & PF_EXITING) {
                if (timeout)
                        ret = wait_event_timeout(
                                        sched->job_scheduled,
                                        drm_sched_entity_is_idle(entity),
                                        timeout);
        } else {
                wait_event_killable(sched->job_scheduled,
                                    drm_sched_entity_is_idle(entity));
        }
        ...
}

Because the suspend task or runtime PM worker does not have PF_EXITING set,
drm_sched_entity_flush() ignores the timeout and sleeps indefinitely in
wait_event_killable() while holding ddev->filelist_mutex.

In amdgpu_cs_ioctl(), a runtime PM reference is only held during the ioctl
itself, and per-job runtime PM references are only taken in
amdgpu_fence_emit() when a job reaches run_job().  If a client submits a job
with an unsignaled syncobj dependency, runtime autosuspend can enter
amdgpu_pmops_runtime_suspend() -> amdgpu_device_suspend() and block in
amdgpu_device_flush_file_entities() while the device status is
RPM_SUSPENDING.  When another thread or process then calls amdgpu_cs_ioctl()
to submit the work that would signal that syncobj, its runtime PM resume
waits for RPM_SUSPENDING to complete, deadlocking both tasks.

Killing the process owning the blocked entity does not break the wait
either: amdgpu_flush() times out after 2 seconds and calls
drm_sched_entity_kill(), which drains the entity without waking
sched->job_scheduled, and the exiting task then blocks on
ddev->filelist_mutex in drm_close_helper().  During system suspend,
amdgpu_userq_suspend() also runs right before
amdgpu_device_flush_file_entities() and unmaps usermode queues while
userspace is already frozen, so any entity job waiting on a usermode queue
or userspace fence can never make progress.

-- 
Sashiko AI review · 
https://sashiko.dev/#/patchset/[email protected]?part=1

Reply via email to