Hi all,
Not a patch review, just a field data point on this fd-close/VM-teardown
race, since the discussion here (and on Vitaly's "wait for HW fences before
VM teardown on fd close") turned on whether dma_resv/TTM already keeps the
PDs/PTs alive long enough on kernel queues.
On GFX11 it does not appear to, in practice.
Hardware / software:
- Navi 32 (RDNA3, GFX11), PCI 1002:747e, sdma_v6_0 / mes_v11_0
- Linux 7.1.4, Mesa 26.1.3, amdgpu defaults (vm_update_mode=-1)
In normal desktop use (closing games/browsers) I hit, ~once a day:
amdgpu: ring sdma0 timeout, signaled seq=354165, emitted seq=354166
amdgpu: Starting sdma0 ring reset -> Ring sdma0 reset succeeded
amdgpu: [drm] device wedged, but no recovery needed
i.e. exactly one in-flight SDMA job never signals, and it is always
immediately preceded by VM-teardown for an exiting process:
amdgpu: VM memory stats for proc <steam/Xwayland/vesktop/...> is
non-zero when fini
Building Vitaly's IGT amd_close_race and running only the mildest subtest
(close-race-low, 2 racer threads) reproduces it deterministically within
seconds, and on this GFX11 part it escalates hard:
[gfxhub] page fault (src_id:0 ring:64 vmid:7 pasid:2931)
GCVM_L2_PROTECTION_FAULT_STATUS:0x00700881
Faulty UTCL2 client ID: CPF (0x4)
PERMISSION_FAULTS: 0x8
... GPU reset succeeded / VRAM is lost due to GPU reset! (x6)
... MES failed to respond to msg=RESET / REMOVE_QUEUE (x20)
reset via MES failed and try pipe reset -110
Ring gfx_0.0.0 reset failed
The GPU only self-recovered after the 12th reset; a couple more rounds and
it would have needed a power cycle. The CPF client + PERMISSION_FAULTS look
like the GPU walking PTEs that teardown has already invalidated - which is
the point in question here.
Full details, dmesg, environment and a captured devcoredump are in:
https://gitlab.freedesktop.org/drm/amd/-/work_items/5512
Happy to run further instrumentation (tracepoints, IB dumps, specific
debugfs) on this hardware if it helps narrow down why the fence doesn't
gate teardown on GFX11.
This investigation, reproduction and write-up were prepared with AI
assistance.
Thanks,
Liav
