On 22-Jul-26 12:01 AM, Sunday Clement wrote:
Host kernel panics (Oops in free_swap_cache) when GPU reset is
triggered under multi-process load. The crash is caused by SDMA engines
writing to memory while munmap clears page table entries during process
eviction.
The race condition:
amdgpu_device_halt_activities() loops over all devices in the reset
list and calls amdgpu_amdkfd_pre_reset() on each device. The first call
to amdgpu_amdkfd_pre_reset() triggers global process eviction across
all GPUs via kgd2kfd_suspend_process(), which starts munmap clearing
PTEs. However, SDMA engines on other devices in the reset list still
have in-flight DMA transactions that have not been drained.
Fix by draining ALL SDMA rings on ALL devices in the reset list BEFORE
calling amdgpu_amdkfd_pre_reset() on any device. Also flush the HDP
write cache to ensure SDMA writes reach system memory before PTEs are
cleared. This guarantees no in-flight DMAs exist when process eviction
begins.
The processes are signaled during pre_reset. Without
suspending/notifying any process, how do you expect rings to drain?
Thanks,
Lijo
Signed-off-by: Sunday Clement <[email protected]>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 31 +++++++++++++++++++++-
1 file changed, 30 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 472e96ae884e..0cd3dc4cffb2 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -5340,7 +5340,36 @@ static void amdgpu_device_halt_activities(struct
amdgpu_device *adev,
bool need_emergency_restart)
{
struct amdgpu_device *tmp_adev = NULL;
- int i;
+ int i, r;
+ struct amdgpu_ring *ring;
+
+ /*
+ * Drain all SDMA on all devices before any KFD pre_reset.
+ *
+ * amdgpu_amdkfd_pre_reset() triggers global process eviction across
+ * all GPUs, which starts munmap clearing PTEs. We must ensure no
+ * in-flight SDMA DMAs on any device when eviction begins, otherwise
+ * SDMA writes race with munmap, corrupting PTEs.
+ */
+ list_for_each_entry(tmp_adev, device_list, reset_list) {
+ for (i = 0; i < tmp_adev->sdma.num_instances; i++) {
+ ring = &tmp_adev->sdma.instance[i].ring;
+ if (ring->sched.timeout == MAX_SCHEDULE_TIMEOUT)
+ continue;
+
+ r = amdgpu_fence_wait_empty(ring);
+ if (r)
+ dev_warn(tmp_adev->dev,
+ "SDMA%d ring drain timeout before reset,
proceeding\n", i);
+ }
+
+ /*
+ * Flush HDP write cache to ensure SDMA writes reach system
+ * memory before munmap. Prevents cached writes from arriving
+ * after pages are freed.
+ */
+ amdgpu_device_flush_hdp(tmp_adev, NULL);
+ }
/* block all schedulers and reset given job's ring */
list_for_each_entry(tmp_adev, device_list, reset_list) {
