On 2026-10-15 11:02, William Palacek wrote:
kgd2kfd_interrupt() only checks init_complete and the per-node
interrupts_active flag, and interrupts_active is cleared only in
kfd_interrupt_exit(). During a GPU reset KFD therefore keeps queueing
IH entries for its interrupt worker, and the worker processes them
together with entries queued just before the reset started. These
entries describe state that the reset is discarding, and processing
them can act on queues and processes that are being torn down.
Discard KFD IH entries while the reset domain is held, using the
existing amdgpu_in_reset() check:
- kgd2kfd_interrupt() drops new entries;
- the interrupt worker discards entries it dequeues;
- kgd2kfd_pre_reset() flushes the interrupt workqueue after suspending
KFD, so entries queued before the reset are discarded before the
ASIC is reset, rather than whenever the worker next runs.
Nothing is drained through the GPU, which may not respond at this point.
The only entry KFD waits for is the drain fence that
kfd_process_drain_interrupts() sends when a debugger disables traps or
detaches. If that happens during a reset, the fence is discarded and
the wait ends after AMDGPU_FENCE_JIFFIES_TIMEOUT (HZ/2) with a
ratelimited warning.
Suggested-by: Felix Kuehling <[email protected]>
Signed-off-by: William Palacek <[email protected]>
---
v4: new approach, replaces "drm/amdgpu: preempt KFD queues before taking
the reset domain lock" (v1-v3)
- Drop the queue preemption and every reset-ordering change: queues
cannot be unmapped once a reset has started (Christian)
- Ignore KFD interrupts while amdgpu_in_reset() (Felix)
- Discard IH entries already queued when the reset starts: the worker
drops them and kgd2kfd_pre_reset() flushes the KFD IH workqueue
(Felix: drain or discard)
- Drop the v3 amdgpu_amdkfd_pre_reset() device-list change and the
unmap_queues_cpsch() -EIO path along with the approach
- Retitle to drm/amdkfd, the change is KFD-only
---
drivers/gpu/drm/amd/amdkfd/kfd_device.c | 7 +++++++
drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c | 6 ++++--
2 files changed, 11 insertions(+), 2 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
index bb4c6695f3cf9..ceee52eee7a38 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
@@ -1019,6 +1019,9 @@ int kgd2kfd_pre_reset(struct kfd_dev *kfd,
kgd2kfd_suspend(kfd, true);
+ /* Reset is in progress, so the worker discards what is still queued */
+ flush_workqueue(kfd->ih_wq);
+
This doesn't do what you think it does. It waits for any running worker
to finish. But the worker can schedule itself again, and it won't wait
for that rescheduled worker.
for (i = 0; i < kfd->num_nodes; i++)
kfd_signal_reset_event(kfd->nodes[i]);
@@ -1176,6 +1179,10 @@ void kgd2kfd_interrupt(struct kfd_dev *kfd, const void *ih_ring_entry)
if (!kfd->init_complete)
return;
+ /* IH entries raised while the ASIC is being reset are stale */
+ if (amdgpu_in_reset(kfd->adev))
+ return;
+
if (kfd->device_info.ih_ring_entry_size > sizeof(patched_ihre)) {
dev_err_once(kfd_device, "Ring entry too small\n");
return;
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c
b/drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c
index 783c2f5a04e4b..abc0e5c583cee 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c
@@ -140,8 +140,10 @@ static void interrupt_wq(struct work_struct *work)
unsigned long start_jiffies = jiffies;
while (dequeue_ih_ring_entry(dev, &ih_ring_entry)) {
- dev->kfd->device_info.event_interrupt_class->interrupt_wq(dev,
- ih_ring_entry);
+ /* Discard entries queued before the reset started */
+ if (!amdgpu_in_reset(dev->adev))
+
dev->kfd->device_info.event_interrupt_class->interrupt_wq(dev,
+
ih_ring_entry);
kfifo_skip_count(&dev->ih_fifo,
dev->kfd->device_info.ih_ring_entry_size);
if (time_is_before_jiffies(start_jiffies + HZ)) {
Just below is where the rescheduling happens. You'll need to prevent
that when we're draining work in the reset case. It should be no problem
because it shouldn't take a lot of time when we're just skipping over
all the IH entries anyway.
Regards,
Felix
