On 2026-11-17 at 10:57 +1100, "Gregory Price (Meta)" <[email protected]>
wrote...
> On 2026-11-17 10:24 +1100, Alistair Popple wrote:
> > On 2026-11-14 at 07:53 +1100, "Gregory Price (Meta)" <[email protected]>
> > wrote...
> > > On 2026-11-13 21:43 +0100, David Hildenbrand (Arm) wrote:
> > > > On 11/13/26 21:34, Thomas Hellström wrote:
> > >
> > > > I recall that John (cc) kept saying "GPUs and NUMA balancing is
> > > > inherently
> > > > incompatible, you have to disable NUMA balancing".
> > > >
> > >
> > > If this is a NUMA-hotplugged GPU, then this is resolved by a private
> > > node, because this:
> > >
> > > @@ -6404,6 +6404,9 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
> > > if (!folio || folio_is_zone_device(folio))
> > >
> > >
> > > changes to
> > >
> > > if (!folio || folio_is_common_memory(folio))
> > >
> > > And all folios on the GPU node are never subject to NUMA Balancing
> > >
> > > so balancing can stay enabled while the GPU memory is not subject to
> > > PROT_NONE spew.
> >
> > Hi Gregory, just to hopefully help clarify understanding a bit here this
> > isn't
> > related to folio location and therefore private node won't help much.
> >
> > This about a driver getting a snapshot of current PTE state to program a
> > secondary MMU (in this case on a GPU, but can equally be a NIC or other
> > device).
> >
> > Even CPU memory being subject to NUMA balancing can have quite an adverse
> > impact
> > on GPU performance due to increased incidence of GPU faults, hence why John
> > says
> > GPUs (using SVM) are inherently incompatible with NUMA balancing.
> >
>
> Ah, I follow. I'm still working through the performance implications of
> mmu_notifiers in the presence of PROT_NONE faults, but if i'm
> understanding correctly it tracks that you'd say it's incompatible.
Incompatible is probably not the correct term as it still works. It's just that
GPU faults are really expensive, and to be avoided at all costs. So having NUMA
balancing running around randomly unmapping pages the GPU is using (regardless
of whether or not they are on the device) results in more GPU faults and
therefore a huge perf impact for basically no (or in fact sometimes negative)
gain.
> I presume here the driver needs a global view of memory (or at least all
> memory it can potentially DMA), rather than simply just a subset
> (some set of VMAs)?
The basic process is this:
* GPU needs to access some user space virtual address (eg. allocated with
mmap/malloc)
* Therefore GPU driver needs to setup GPU page tables to point to the physical
memory[1] backing the user space virtual address
* To find the physical address the GPU driver uses hmm_range_fault() on the
range, possibly faulting pages in if required (ie. if CPU PTE permissions
don't match what's required).
* hmm_range_fault() relies on mmu_notifiers for synchronisation with CPU PTE
state.
* An mmu_notifier firing causes a driver to do two things:
a) Must invalidate any GPU page tables / TLBs mapping the notified virtual
address.
b) Increments a sequence number for any concurrent hmm_range_faults(), which
must
retry as their view of CPU PTEs is now out-dated. This is the problem
this patch series hits - the migration on fault causes an mmu_notifier to
fire
resulting in a hmm_range_fault() retry and associated overhead.
So the driver doesn't need a global view of memory, just a view of the memory
the GPU needs, or likely needs, to access. Determining that depends on the
driver - some might require pre-registration of the ranges (eg. via an ioctl)
prior to launching work whilst others might wait until the GPU work faults on an
address not in the GPU page tables. Or some combination thereof.
- Alistair
[1] - Well technically the IOVA / DMA address mapping physical system memory
but that's not relevant for this discussion. We can assume there are no IOMMUs
present.
> ~Gregory
>
> --
> ~Gregory
>
>