Hi Christian,
> It's based on the latest amd-staging-drm-next development branch. I
> have no idea if that will even apply to 7.2.6.
It did, with a small compat layer (begin/end_critical naming, hmm_range
field plumbing, trace arg order, NUM_LEVELS). Re-test results on
gfx906 (MI50 32GB, Vega 20, TB3 eGPU), kernel 7.2.6 + your Sep-30
8-patch set including amdgpu_vm_pt_leaves() v3:
- The gap-slot overshoot regression I reported on 06/13 is gone.
- The previously *deterministic* reproducer (vLLM/ROCm staging a
10-shard GLM-4-9B, i.e. heavy SVM map/unmap/evict/restore churn)
now runs to completion.
- Sustained sessions with llama.cpp (qwen3-32b, 2 parallel slots)
plus batch inference workers: zero "STALE PDE", zero
bo_update failures, zero MMHUB faults.
So for the series:
Tested-by: Olivier Kaloudoff <[email protected]>
The one remaining crash signature we've seen since is unrelated to the
VM rework: the TB3 link drops under sustained ~100 degC junction load
(Razer Core X thermal/electrical marginality), which is the surprise
removal path below.
> If you could give a bit more data (e.g. what timing) then Vitaly will
> probably be able to create a CI test case for that.
Here is a captured timeline (EFI pstore, 7.2.6-k726). Two independent
drops produced byte-identical stack shapes; this is the cleaner one:
t+0.000 amdgpu 05:00.0: Failed to send message 0x19, response 0xffffffff
(SMU already gone - link electrically dead)
t+0.001 thunderbolt 0-2: device disconnected
t+0.11 "finishing device", amdgpu_job_timedout skips recovery
t+0.44 thunderbolt 0-2: Razer Core X found again (link flapped)
t+1.21 thunderbolt 0-2: device disconnected (gone for good)
t+9.38 qcm fence wait timeout -> GPU reset begin ->
"device lost from bus!", reset ret -19
t+9.63 ring kiq test fails; WARNING in kfd DQM uninitialize
(queues still attached to the dead device)
t+9.63 kgd2kfd_device_exit() frees adev->kfd.dev
<- amdgpu_amdkfd_device_fini_sw
<- amdgpu_device_fini_hw <- pciehp_unconfigure_device
<- pciehp_disable_slot <- pciehp_ist (irq/39-pciehp)
t+9.64 BUG: NULL deref at 0x1b4, kgd2kfd_teardown_processes+0xde
same pciehp_ist thread, ~3ms after kfd.dev was freed.
So no race is needed for CI: within amdgpu_device_fini_hw(),
device_fini_sw runs kgd2kfd_device_exit() (which releases kfd.dev)
*before* amdgpu_amdkfd_teardown_processes() runs - it dereferences the
already-freed pointer unconditionally when KFD processes are attached.
An igt-side equivalent would be: open KFD queues on a device, hot-unplug
the parent (or kset unregister), expect teardown_processes on a live
kfd.dev. In our case llama-server/vLLM held the queues.
Local workaround we carry (happy to post properly if useful):
guard adev->kfd.dev for NULL in kgd2kfd_teardown_processes() and bound
the "wait for process count" loop (30s) so pciehp can't wedge forever
on a dead device. Same spirit in the userptr/VM update paths: fail fast
with -ENODEV once the device is unplugged.
> Yeah that is already very well known problem. The old KFD SVM code has
> a whole bunch of design bugs. Because of that we are working on
> completely replacing that with a new implementation based on the
> drm_gpusvm component.
Acknowledged, that matches what we see. Happy to keep exercising the
interim fixes on gfx906 and to run the drm_gpusvm series once it's
usable - this machine produces SVM invalidation churn (evict/restore
under real memory pressure) more reliably than anything else I have.
Regards,
Olivier
