Thank you for your contribution! Sashiko AI review found 4 potential issues to consider: - [High] drm/xe/guc: unbanned reset state in xe_guc_exec_queue_reset_trigger_cleanup() - [High] drm/xe/guc: LR queue hang in xe_guc_exec_queue_memory_cat_error_handler() - [Medium] drm/xe/guc: xe_guc_exec_queue_wake_ufence() allows early TDR via queue kill - [Medium] drm/xe/guc: multi-queue ban and wakeups in xe_guc_exec_queue_wake_ufence() It also noticed 1 related pre-existing issue this patch does not introduce: - [High] drm/xe: concurrent work_tdr execution in guc_exec_queue_stop() on suspend [1] --
commit 9843a5ccba927d9fc2ffb48f6d5f11c6006bc0ea Author: Arvind Yadav <[email protected]> drm/xe/guc: Do not trigger TDR from the memory CAT error handler This commit separates waking user-fence waiters from triggering TDR so that the GuC memory CAT error handler marks the exec queue reset and wakes waiters without queuing TDR before the context-reset notification arrives. > diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c > b/drivers/gpu/drm/xe/xe_guc_submit.c > index b14b37b767f5e..87b22284675e4 100644 > --- a/drivers/gpu/drm/xe/xe_guc_submit.c > +++ b/drivers/gpu/drm/xe/xe_guc_submit.c > @@ -557,17 +557,39 @@ static bool vf_recovery(struct xe_guc *guc) > return xe_gt_recovery_pending(guc_to_gt(guc)); > } > > -static void xe_guc_exec_queue_trigger_cleanup(struct xe_exec_queue *q) > +static void xe_guc_exec_queue_wake_ufence(struct xe_exec_queue *q) > { > struct xe_guc *guc = exec_queue_to_guc(q); > struct xe_device *xe = guc_to_xe(guc); > > /** to wakeup xe_wait_user_fence ioctl if exec queue is reset */ > wake_up_all(&xe->ufence_wq); > +} > + > +static void xe_guc_exec_queue_trigger_cleanup(struct xe_exec_queue *q) > +{ > + xe_guc_exec_queue_wake_ufence(q); > > xe_sched_tdr_queue_imm(&q->guc->sched); > } [Severity: High] This problem wasn't introduced by this patch, but calling xe_guc_exec_queue_trigger_cleanup() from guc_exec_queue_stop() during system suspend [1] queues sched->base.work_tdr on gt->ordered_wq while the GT is suspending. Unlike GT reset, where xe_guc_submit_stop() runs on gt->ordered_wq, xe_pm_suspend() calls xe_guc_submit_stop() from the suspend thread while gt->ordered_wq is still active. When guc_exec_queue_stop() bans a queue with an incomplete job and calls xe_guc_exec_queue_trigger_cleanup(), guc_exec_queue_timedout_job() runs concurrently on gt->ordered_wq with xe_device_is_in_reset() false, attempting forcewake and MMIO register reads for devcoredump capture as the device enters D3 and then calling xe_sched_submission_start() while GuC submission is stopped. > + > +/* > + * Allow callers to wake user-fence waiters without triggering the TDR. > + * The memory CAT handler uses this to avoid starting cleanup before > + * the context-reset notification arrives. > + */ > +static void xe_queue_trigger_cleanup_for_reset(struct xe_exec_queue *q, > + bool trigger_tdr) > +{ > + if (exec_queue_banned(q)) > + return; > + > + if (trigger_tdr) > + xe_guc_exec_queue_trigger_cleanup(q); > + else > + xe_guc_exec_queue_wake_ufence(q); > +} [Severity: Medium] Does waking ufence_wq here still allow TDR to run before the context-reset notification arrives when userspace is waiting in xe_wait_user_fence_ioctl()? Because EXEC_QUEUE_STATE_RESET is already set, a thread woken in xe_wait_user_fence_ioctl() sees q->ops->reset_status(q) return true and returns -EIO to userspace immediately. If userspace responds to -EIO by destroying the exec queue or closing the DRM fd, xe_exec_queue_kill() -> guc_exec_queue_kill() calls xe_guc_exec_queue_trigger_cleanup(q), queuing TDR and sending the scheduling-disable H2G before the context-reset notification arrives. [ ... ] > @@ -653,7 +675,8 @@ static void > xe_guc_exec_queue_group_trigger_cleanup(struct xe_exec_queue *q) > mutex_unlock(&group->list_lock); > } > > -static void xe_guc_exec_queue_reset_trigger_cleanup(struct xe_exec_queue *q) > +static void xe_guc_exec_queue_reset_trigger_cleanup(struct xe_exec_queue *q, > + bool trigger_tdr) > { > if (xe_exec_queue_is_multi_queue(q)) { > struct xe_exec_queue *primary = > xe_exec_queue_multi_queue_primary(q); > @@ -664,20 +687,17 @@ static void > xe_guc_exec_queue_reset_trigger_cleanup(struct xe_exec_queue *q) > WRITE_ONCE(group->banned, true); > > set_exec_queue_reset(primary); > - if (!exec_queue_banned(primary)) > - xe_guc_exec_queue_trigger_cleanup(primary); > + xe_queue_trigger_cleanup_for_reset(primary, trigger_tdr); > > mutex_lock(&group->list_lock); > list_for_each_entry(eq, &group->list, multi_queue.link) { > set_exec_queue_reset(eq); > - if (!exec_queue_banned(eq)) > - xe_guc_exec_queue_trigger_cleanup(eq); > + xe_queue_trigger_cleanup_for_reset(eq, trigger_tdr); > } > mutex_unlock(&group->list_lock); [Severity: Medium] When trigger_tdr is false, should xe_guc_exec_queue_reset_trigger_cleanup() still set group->banned and call xe_guc_exec_queue_wake_ufence() for every queue in the group? Setting group->banned to true causes xe_guc_exec_queue_group_stop() and guc_exec_queue_timedout_job() on any sibling queue to treat the group as banned and immediately trigger TDR and disable_scheduling() even though trigger_tdr was false. In addition, xe_guc_exec_queue_wake_ufence() wakes the device-wide xe->ufence_wq waitqueue, so calling it inside the group->list loop calls wake_up_all(&xe->ufence_wq) repeatedly for every secondary queue while holding group->list_lock. > } else { > set_exec_queue_reset(q); > - if (!exec_queue_banned(q)) > - xe_guc_exec_queue_trigger_cleanup(q); > + xe_queue_trigger_cleanup_for_reset(q, trigger_tdr); > } > } [Severity: High] Does setting EXEC_QUEUE_STATE_RESET without banning the queue or stopping the scheduler leave the queue in an inconsistent state where q->ops->reset_status(q) is true while exec_queue_killed_or_banned_or_wedged(q) is false? Because guc_exec_queue_suspend() only checks exec_queue_killed_or_banned_or_wedged(q), preempt_fence_enable_signaling() succeeds in calling q->ops->suspend(q) on the reset queue (which queues STATIC_MSG_SUSPEND and sends an early disable_scheduling() H2G via __guc_exec_queue_process_msg_suspend()). Then preempt_fence_work_func() checks !q->ops->reset_status(q), sees EXEC_QUEUE_STATE_RESET, skips q->ops->suspend_wait(q), and signals the preemption fence immediately while the context is still active on hardware: drivers/gpu/drm/xe/xe_preempt_fence.c:preempt_fence_work_func() { ... } else if (!q->ops->reset_status(q)) { int err = q->ops->suspend_wait(q); if (err) dma_fence_set_error(&pfence->base, err); } else { dma_fence_set_error(&pfence->base, -ENOENT); } dma_fence_signal(&pfence->base); ... } wait_for_existing_preempt_fences() ignores -ENOENT, allowing VM unbind or BO eviction to proceed while the GPU context is still running. In addition, because the scheduler is not stopped and guc_exec_queue_run_job() only checks exec_queue_killed_or_banned_or_wedged(q), queued jobs continue to be submitted to GuC after the CAT error, and a GT reset in guc_exec_queue_stop() masks q->guc->state without EXEC_QUEUE_STATE_RESET, losing the reset state on LR or idle queues. [ ... ] > @@ -3592,8 +3612,12 @@ int xe_guc_exec_queue_memory_cat_error_handler(struct > xe_guc *guc, u32 *msg, > > trace_xe_exec_queue_memory_cat_error(q); > > - /* Treat the same as engine reset */ > - xe_guc_exec_queue_reset_trigger_cleanup(q); > + /* > + * Starting the TDR here can send a scheduling-disable request before > + * the context-reset notification arrives. Mark the queue reset and > + * wake user-fence waiters, leaving that notification to trigger > cleanup. > + */ > + xe_guc_exec_queue_reset_trigger_cleanup(q, false); > > return 0; > } [Severity: High] Can skipping TDR here leave long-running (LR) queues stuck forever if GuC does not follow the memory CAT error with a context-reset notification? In guc_exec_queue_init(), queues on an LR-mode VM set timeout to MAX_SCHEDULE_TIMEOUT, so drm_sched_start_timeout() never arms a fallback TDR timer. If no XE_GUC_ACTION_CONTEXT_RESET_NOTIFICATION arrives, guc_exec_queue_timedout_job() never runs: the context is never banned or disabled in GuC, pending xe_sched_job entries remain on pending_list holding their exec queue and runtime PM references, and callers waiting in xe_wait_user_fence_ioctl() without an exec_queue_id (where q is NULL and reset_status() is not checked) wake once on ufence_wq and go back to sleep until their full timeout expires. [1] https://sashiko.dev/#/bug/linux-449968a5-0b4a-4c5d-acd3-7ced29f0129b -- Sashiko AI review ยท https://sashiko.dev/#/patchset/[email protected]?part=1
