kgd2kfd_interrupt() only checks init_complete and the per-node interrupts_active flag, and interrupts_active is cleared only in kfd_interrupt_exit(). During a GPU reset KFD therefore keeps queueing IH entries for its interrupt worker, and the worker processes them together with entries queued just before the reset started. These entries describe state that the reset is discarding, and processing them can act on queues and processes that are being torn down.
Discard KFD IH entries while the reset domain is held, using the existing amdgpu_in_reset() check: - kgd2kfd_interrupt() drops new entries; - the interrupt worker discards entries it dequeues; - kgd2kfd_pre_reset() flushes the interrupt workqueue after suspending KFD, so entries queued before the reset are discarded before the ASIC is reset, rather than whenever the worker next runs. Nothing is drained through the GPU, which may not respond at this point. The only entry KFD waits for is the drain fence that kfd_process_drain_interrupts() sends when a debugger disables traps or detaches. If that happens during a reset, the fence is discarded and the wait ends after AMDGPU_FENCE_JIFFIES_TIMEOUT (HZ/2) with a ratelimited warning. Suggested-by: Felix Kuehling <[email protected]> Signed-off-by: William Palacek <[email protected]> --- v4: new approach, replaces "drm/amdgpu: preempt KFD queues before taking the reset domain lock" (v1-v3) - Drop the queue preemption and every reset-ordering change: queues cannot be unmapped once a reset has started (Christian) - Ignore KFD interrupts while amdgpu_in_reset() (Felix) - Discard IH entries already queued when the reset starts: the worker drops them and kgd2kfd_pre_reset() flushes the KFD IH workqueue (Felix: drain or discard) - Drop the v3 amdgpu_amdkfd_pre_reset() device-list change and the unmap_queues_cpsch() -EIO path along with the approach - Retitle to drm/amdkfd, the change is KFD-only --- drivers/gpu/drm/amd/amdkfd/kfd_device.c | 7 +++++++ drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c | 6 ++++-- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c index bb4c6695f3cf9..ceee52eee7a38 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c @@ -1019,6 +1019,9 @@ int kgd2kfd_pre_reset(struct kfd_dev *kfd, kgd2kfd_suspend(kfd, true); + /* Reset is in progress, so the worker discards what is still queued */ + flush_workqueue(kfd->ih_wq); + for (i = 0; i < kfd->num_nodes; i++) kfd_signal_reset_event(kfd->nodes[i]); @@ -1176,6 +1179,10 @@ void kgd2kfd_interrupt(struct kfd_dev *kfd, const void *ih_ring_entry) if (!kfd->init_complete) return; + /* IH entries raised while the ASIC is being reset are stale */ + if (amdgpu_in_reset(kfd->adev)) + return; + if (kfd->device_info.ih_ring_entry_size > sizeof(patched_ihre)) { dev_err_once(kfd_device, "Ring entry too small\n"); return; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c b/drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c index 783c2f5a04e4b..abc0e5c583cee 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c @@ -140,8 +140,10 @@ static void interrupt_wq(struct work_struct *work) unsigned long start_jiffies = jiffies; while (dequeue_ih_ring_entry(dev, &ih_ring_entry)) { - dev->kfd->device_info.event_interrupt_class->interrupt_wq(dev, - ih_ring_entry); + /* Discard entries queued before the reset started */ + if (!amdgpu_in_reset(dev->adev)) + dev->kfd->device_info.event_interrupt_class->interrupt_wq(dev, + ih_ring_entry); kfifo_skip_count(&dev->ih_fifo, dev->kfd->device_info.ih_ring_entry_size); if (time_is_before_jiffies(start_jiffies + HZ)) { -- 2.43.0
