Hi Felix,
Retest done on the same machine (MI50 gfx906 over TB3, Strix Halo host,
28GB RAM).
Test kernel: 7.3.0-rc5 plus
- 25f0b322ffb5 (evict SVM BOs synchronously from TTM eviction)
- ec4d432b (vm range hole fix)
- 46109de0 (skip migration when fault window already in VRAM)
- our local series from the original report (restore convergence,
non-xnack lock-bounded prefault, unplug/teardown guards), plus two
migration-error propagation changes still being baked.
Reproducer: llama-server -ngl 99 with ~19GB mmap'd model
(SVM-registered), health-polled while a second allocator saturated the
remaining VRAM, then a managed-memory bounce workload
(hipMallocManaged, 2GB, 20 CPU<->GPU rounds) on ranges that dmesg
reports "over locked budget ... left evictable" — i.e. genuinely
reclaimable and migrated under pressure.
Results, 40 min sustained:
- ftrace count of svm_range_cpu_invalidate_pagetables: 0 for the
entire run, including a window where MemAvailable bottomed at
~470MB and direct reclaim was hammering the host.
- llama-server health=200 on every poll; real inference verified.
- Managed bounce completed all 20 rounds, no migration errors.
Baseline for comparison: on this same hardware at 7.3-rc5 + our series
without 46109de0 + copy-done propagation, the bounce workload under
VRAM eviction panic'd on Oct 3 — BUG_ON(start + size > res->size) at
amdgpu_res_cursor.h:65 inside svm_migrate_copy_to_vram, called from
svm_migrate_ram_to_vram via kfd_ioctl_svm prefetch. (EFI pstore
capture; happy to send the trace.)
So on this stack the storm is gone and the migrate path survives
forced eviction.
One honest caveat: our non-xnack locking makes registered ranges
unevictable up to a lock budget, so "inv=0" is partly by construction —
the over-budget bounce run is the part that exercises the evictable
path, and that's where the Oct-3 BUG_ON used to live.
So I ran the clean A/B: vanilla 7.3.0-rc5 with only your three commits
(25f0b322ffb5, ec4d432b, 46109de0), none of our local SVM changes,
same reclaim-storm reproducer:
- 94.3M svm_range_cpu_invalidate_pagetables hits in 30 min
(~4M/min, same rate as baseline)
- llama-server never converged (health 503, RSS churning, threads
parked in kfd_wait_on_events)
- an earlier run went fully dark ~13 min in — no console, no ICMP,
journal just ends (this run did not reproduce that)
So on vanilla the three commits do not resolve the livelock. 46109de0
only skips migration for windows that are already VRAM-resident; under
reclaim the faulted ranges are genuinely RAM-side, so the residency
check never engages and the evict -> restore -> re-invalidate loop is
unchanged. The failure is restore-worker convergence itself — the
invalid cmpxchg aborts that leave evicted_ranges never draining —
which none of the three commits touch.
The only thing that suppresses the storm here is our restore-worker
locking, which is a workaround with its own tradeoffs, not an
upstreamable fix — but it confirms where the loop lives.
Regards,
Olivier
