Host kernel panics (Oops in free_swap_cache) when GPU reset is triggered under multi-process load. The crash is caused by SDMA engines writing to memory while munmap clears page table entries during process eviction.
The race condition: amdgpu_device_halt_activities() loops over all devices in the reset list and calls amdgpu_amdkfd_pre_reset() on each device. The first call to amdgpu_amdkfd_pre_reset() triggers global process eviction across all GPUs via kgd2kfd_suspend_process(), which starts munmap clearing PTEs. However, SDMA engines on other devices in the reset list still have in-flight DMA transactions that have not been drained. Fix by draining ALL SDMA rings on ALL devices in the reset list BEFORE calling amdgpu_amdkfd_pre_reset() on any device. Also flush the HDP write cache to ensure SDMA writes reach system memory before PTEs are cleared. This guarantees no in-flight DMAs exist when process eviction begins. Signed-off-by: Sunday Clement <[email protected]> --- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 31 +++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c index 472e96ae884e..0cd3dc4cffb2 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c @@ -5340,7 +5340,36 @@ static void amdgpu_device_halt_activities(struct amdgpu_device *adev, bool need_emergency_restart) { struct amdgpu_device *tmp_adev = NULL; - int i; + int i, r; + struct amdgpu_ring *ring; + + /* + * Drain all SDMA on all devices before any KFD pre_reset. + * + * amdgpu_amdkfd_pre_reset() triggers global process eviction across + * all GPUs, which starts munmap clearing PTEs. We must ensure no + * in-flight SDMA DMAs on any device when eviction begins, otherwise + * SDMA writes race with munmap, corrupting PTEs. + */ + list_for_each_entry(tmp_adev, device_list, reset_list) { + for (i = 0; i < tmp_adev->sdma.num_instances; i++) { + ring = &tmp_adev->sdma.instance[i].ring; + if (ring->sched.timeout == MAX_SCHEDULE_TIMEOUT) + continue; + + r = amdgpu_fence_wait_empty(ring); + if (r) + dev_warn(tmp_adev->dev, + "SDMA%d ring drain timeout before reset, proceeding\n", i); + } + + /* + * Flush HDP write cache to ensure SDMA writes reach system + * memory before munmap. Prevents cached writes from arriving + * after pages are freed. + */ + amdgpu_device_flush_hdp(tmp_adev, NULL); + } /* block all schedulers and reset given job's ring */ list_for_each_entry(tmp_adev, device_list, reset_list) { -- 2.43.0
