On Fri, Aug 21, 2026 at 5:06 PM Felix Kuehling <[email protected]> wrote: > > On 2026-08-21 15:52, Alex Deucher wrote: > > Document the details of UALink on the GPU. > > > > Cc: [email protected] > > Cc: [email protected] > > Signed-off-by: Alex Deucher <[email protected]> > > --- > > Documentation/gpu/amdgpu/index.rst | 1 + > > Documentation/gpu/amdgpu/ualink.rst | 74 +++++++++++++++++++++++++++++ > > 2 files changed, 75 insertions(+) > > create mode 100644 Documentation/gpu/amdgpu/ualink.rst > > > > diff --git a/Documentation/gpu/amdgpu/index.rst > > b/Documentation/gpu/amdgpu/index.rst > > index b2ab182236efb..ba2ee73278672 100644 > > --- a/Documentation/gpu/amdgpu/index.rst > > +++ b/Documentation/gpu/amdgpu/index.rst > > @@ -23,4 +23,5 @@ Next (GCN), Radeon DNA (RDNA), and Compute DNA (CDNA) > > architectures. > > debugfs > > process-isolation > > amdgpu-glossary > > + ualink > > ptl > > diff --git a/Documentation/gpu/amdgpu/ualink.rst > > b/Documentation/gpu/amdgpu/ualink.rst > > new file mode 100644 > > index 0000000000000..1e0a6c52b7b7a > > --- /dev/null > > +++ b/Documentation/gpu/amdgpu/ualink.rst > > @@ -0,0 +1,74 @@ > > +============== > > +UALink Support > > +============== > > + > > +Overview > > +======== > > + > > +Connected GPUs in a pod can directly access the remove memory on another > > GPU > > +over UALink. Unlike RMDA, there is no copy involved; it is direct > > loads/stores > > +over the fabric. Shared memory can only be accessed by a remote GPU if the > > +memory was exported and the importer has been authorized. For the memory > > to be > > +shared, it must be part of a unified physical address space shared between > > +nodes. This address space is called NPA (Nework Physical Address) space. > > This > > +address space is partitioned between the GPUs so that each GPU has its own > > +segment of the address space in which to export its memory. Each GPU > > maintains > > +a dedicated set of page tables for their NPA space similar to GPUVM. Note > > that > > +this mechanism only allows for GPU access to remote memory. The remote > > memory > > +is not CPU accessible. > > + > > +Exported memory is not pinned. The exporter may need to move memory or > > make it > > +unavailable to the importers. This is handled via remote TLB shootdowns > > on the > > +exporter in the case of TTM evictions or MMU notifiers and remote presense > > +checks with retry on the importer. > > This is no longer true. We ended up having to pin exported memory. This > was also discussed with Christian already. In principle it's similar to > how DMABufs in VRAM are pinned for RDMA access. > > Remote TLB shootdown will only be used when exported memory is freed in > order to remove access by remote GPUs. > > > > + > > +To access remote memory, the driver can map NPA addresses into its per > > process > > +GPUVM page tables just like local memory. Applications use opaque handles > > to > > +represent remote memory. GPUs in a pod communicate with eachother > > directly to > > +exchange NPA addresses between importers and exporters. If a node goes > > offline > > +or is reset, their peers will clean up any remaining refrences that are > > lost > > +when that happens. > > One important point here is, that NPA addresses are exchanged between > the kernel mode drivers (through the scale-up fabric). NPA addresses are > never given to user mode. > > > > + > > +On the importer, the NPA space is like another physical address space. NPA > > +addresses can be used as physical addresses for GPUVM to provide GPU > > virtual > > +addresses to the memory for processes using the GPU. > > + > > +On the exporter, the NPA space provides a way to expose discontiguous local > > +memory as a contiguous address range for remote GPUs. This allows the > > exporter > > +to locally manage the pages mapped into the NPA space. > > + > > +Remote NPAs are managed like another device specific TTM pool similar to > > +doorbells or VRAM, however they cannot be CPU mapped. > > + > > + > > +User Interface > > +============== > > +Two IOCTLs are provided to export and import remote memory. > > + > > +Export Memory > > +------------- > > +To export memory, a UALINK handle must be created for an allocation that > > can be > > +shared with another node in the pod. To do this the exporter calls the GEM > > +UALink IOCTL with the GEM handle to the buffer it wants to export. The > > IOCTL > > +returns a unique 128 bit handle which can be shared with the remote host. > > +Calling export on the same GEM handle always returns the same UALink > > handle. > > +The UALink handle is destroyed when the GEM object is freed. > > More precisely, the UALink handle is destroyed when the GEM handle count > reaches 0.
Thanks. I've incorporated your comments. Alex > > Regards, > Felix > > > > + > > +Import Memory > > +------------- > > +To import remote memory, the UALink handle from the remote node must be > > +converted from a UALink handle to a local GEM object which represents the > > local > > +reference to the NPA space on the importer. If the memory has already been > > +imported, it just returns a new reference to the existing GEM object. If > > not, > > +the importer queries the exporter to get the NPA address. Once it has > > that, the > > +importer can create the GEM to represent the NPA space used by the > > allocation. > > +The GEM object is then exported to the caller as a dma-buf. The dma-buf is > > +leveraged for dynamic attachment which provides the ability to revoke > > access > > +when necessary. > > + > > + > > +Device to Device Communications > > +=============================== > > + > > +Devices communicate via a protocol implemented in firmware. Mesages sent > > to a > > +remote node generate an interrupt on that node for servicing.
