On 2026-10-13 14:53, Olivier Kaloudoff wrote:
Hi Felix,Retest done on the same machine (MI50 gfx906 over TB3, Strix Halo host, 28GB RAM). Test kernel: 7.3.0-rc5 plus - 25f0b322ffb5 (evict SVM BOs synchronously from TTM eviction) - ec4d432b (vm range hole fix) - 46109de0 (skip migration when fault window already in VRAM) - our local series from the original report (restore convergence, non-xnack lock-bounded prefault, unplug/teardown guards), plus two migration-error propagation changes still being baked. Reproducer: llama-server -ngl 99 with ~19GB mmap'd model (SVM-registered), health-polled while a second allocator saturated the remaining VRAM, then a managed-memory bounce workload (hipMallocManaged, 2GB, 20 CPU<->GPU rounds) on ranges that dmesg reports "over locked budget ... left evictable" — i.e. genuinely reclaimable and migrated under pressure. Results, 40 min sustained: - ftrace count of svm_range_cpu_invalidate_pagetables: 0 for the entire run, including a window where MemAvailable bottomed at ~470MB and direct reclaim was hammering the host. - llama-server health=200 on every poll; real inference verified. - Managed bounce completed all 20 rounds, no migration errors. Baseline for comparison: on this same hardware at 7.3-rc5 + our series without 46109de0 + copy-done propagation, the bounce workload under VRAM eviction panic'd on Oct 3 — BUG_ON(start + size > res->size) at amdgpu_res_cursor.h:65 inside svm_migrate_copy_to_vram, called from svm_migrate_ram_to_vram via kfd_ioctl_svm prefetch. (EFI pstore capture; happy to send the trace.) So on this stack the storm is gone and the migrate path survives forced eviction. One honest caveat: our non-xnack locking makes registered ranges unevictable up to a lock budget, so "inv=0" is partly by construction — the over-budget bounce run is the part that exercises the evictable path, and that's where the Oct-3 BUG_ON used to live. So I ran the clean A/B: vanilla 7.3.0-rc5 with only your three commits (25f0b322ffb5, ec4d432b, 46109de0), none of our local SVM changes, same reclaim-storm reproducer: - 94.3M svm_range_cpu_invalidate_pagetables hits in 30 min (~4M/min, same rate as baseline) - llama-server never converged (health 503, RSS churning, threads parked in kfd_wait_on_events) - an earlier run went fully dark ~13 min in — no console, no ICMP, journal just ends (this run did not reproduce that) So on vanilla the three commits do not resolve the livelock. 46109de0 only skips migration for windows that are already VRAM-resident; under reclaim the faulted ranges are genuinely RAM-side, so the residency check never engages and the evict -> restore -> re-invalidate loop is unchanged. The failure is restore-worker convergence itself — the invalid cmpxchg aborts that leave evicted_ranges never draining — which none of the three commits touch.
I think you're expecting your workload to run successfully while memory is overcommitted. In theory the driver tries to prevent memory overcommitment so that the entire workload can fit into memory at the same time. This is a prerequisite for the HIP programming model where all GPU-mapped virtual memory is resident and GPU accessible all the time. (We can handle memory overcommitment with recoverable page faults, but that's only supported on some GFX9 GPUs at the moment.)
Strix Halo is an APU with all memory shared between CPU and GPU. Allocations that would go to a dedicated VRAM heap on a discrete GPU are allocated from system memory. That creates an accounting problem where your application sees two memory heaps (VRAM and system memory) that alias the same physical memory space. If you try to allocate too much memory from both heaps, you are overcommitting your memory and that can cause the type of problem you're seeing.
Ideally, our memory accounting would be a bit more conservative and prevent you from allocating too much memory for the GPU. But the result would be that your application would fail with a memory allocation error instead of stalling. The trouble is, that it also depends on what else is running on the system.
There is an experimental patch in the user mode stack to use a different allocation method that allows users or administrators to impose memory limits through cgroups that apply to all types of memory allocations. That could potentially help in your situation (by failing memory allocations instead of stalling). You can experiment by setting HSA_USE_UDMABUF=1 in your environment. It won't do anything useful unless you limit your application's memory allowance with cgroup.
Regards, Felix
The only thing that suppresses the storm here is our restore-worker locking, which is a workaround with its own tradeoffs, not an upstreamable fix — but it confirms where the loop lives. Regards, Olivier
