On 11/13/26 21:34, Thomas Hellström wrote: > Hi, David > > On Fri, 2026-11-13 at 19:27 +0100, David Hildenbrand (Arm) wrote: >> On 11/13/26 17:25, Matthew Auld wrote: >>> When automatic NUMA balancing marks a PTE or PMD as protnone >>> (PAGE_NONE), hmm_pte_need_fault() requests a fault to restore >>> access >>> permissions, including _PAGE_RW when write access is needed. >>> >>> Calling handle_mm_fault() from hmm_do_fault() currently triggers >>> NUMA >>> balancing in do_numa_page() and do_huge_pmd_numa_page(). When >>> hmm_range_fault() runs from a device driver or kworker context >>> (such as >>> a GPU SVM or userptr rebind worker), folios get migrated toward the >>> CPU >>> node executing the remote fault. >>> >>> Also, each folio migration invokes try_to_migrate_one() -> >>> mmu_notifier_invalidate_range_start(), which invalidates the >>> caller's mmu_interval_notifier while holding mmap_lock. That causes >>> hmm_range_fault() to abort with -EBUSY after every 2 MiB PMD and >>> exhaust >>> its retry timeout on large buffers, which has been observed with >>> large >>> imported Xe userptr buffers. >>> >>> Add FAULT_FLAG_NO_NUMA and set it in hmm_do_fault() so protnone >>> entries >>> restore PTE/PMD permissions in-place without performing NUMA >>> balancing, >>> matching GUP's default !FOLL_HONOR_NUMA_FAULT behaviour. >> >> Ehm, that's not how it works. >> >> GUP will not trigger a "special" fault and will try not triggering a >> fault and >> not restore permissions. >> >> It will try continuing, leaving the protnone pte as it is. >> >> See gup_can_follow_protnone(). >> >> For read-only access that should always work. >> >> For write-access we always trigger an ordinary fault and do NUMA >> migration etc. >> >> >> So FAULT_FLAG_NO_NUMA is doing something completely different and I >> don't agree >> with it. >> > > There are two major problems with the current behaviour. > > 1) The work item executing hmm_range_fault() might have no affinity at > all to the GPU about to access the memory. > > 2) If migration happens, then hmm_range_fault() will stop and retry > leading to excessive execution time. > > While perhaps Matt's commit message doesn't correctly describe the GUP > behaviour, do you have any alternative suggestion than to avoid > migration in this case? I think it's a reasonable idea given the > problems with the current code?
Adding fault reasons that hack around NUMA hinting such that a actual CPU accessing it later will not trigger ... NUMA hinting? I might be missing the "reasonable" in here :P I recall that John (cc) kept saying "GPUs and NUMA balancing is inherently incompatible, you have to disable NUMA balancing". You could likely implement something similar to GUP, where you would allow for bypassing protnone for reading. That wouldn't give you access to the writing side. There might be ways to sort out some of the writing (similar to FOLL_FORCE handling in GUP), but I'd rather not go down that path. KVM also doesn't seem to need that. -- Cheers, David
