On 05/08/26 21:46, Kuehling, Felix wrote:
On 2026-08-05 03:09, Donet Tom wrote:
On 8/5/26 3:57 AM, Felix Kuehling wrote:
On 2026-08-04 05:52, Donet Tom wrote:
This series fixes topology device creation and proximity domain
mappings in
AMDKFD when a system does not provide a CRAT table and the driver
generates a
Virtual CRAT (VCRAT).
The current implementation assumes that CPU NUMA node IDs are
contiguous and
that every NUMA node contains CPUs. During VCRAT generation, CPU
topology
entries and proximity domains are created only for NUMA nodes that
have CPUs.
GPU proximity domains are then allocated immediately after the CPU
proximity
domains, and the GPU I/O link (proximity_domain_to) is initialized
using the
NUMA node ID to which the GPU is attached, implicitly assuming that
NUMA node
IDs and proximity domains have a one-to-one mapping.
These assumptions break on systems with:
Sparse (non-contiguous) NUMA node IDs
CPU-less NUMA nodes
CPU-less and memory-less NUMA nodes
For example:
available: 3 nodes (0,2-3)
node 0: CPUs present
node 2: CPU-less
node 3: CPU-less
In this case, the driver creates a CPU topology device only for
node 0 and
assigns a single CPU proximity domain (0). However, the GPU VCRAT
still
references the NUMA node ID to which the GPU is attached (for
example, node 3)
in the proximity_domain_to field. Since no corresponding CPU
proximity domain
exists for node 3, the parser cannot find a matching proximity
domain during
VCRAT parsing, causing topology initialization to fail.
Hi Tom,
Hi Felix,
Thank you for the explanation and the patch series. I may have some
gaps in my understanding that I would like to clarify. In my mind, I
was using "proximity domain" and "NUMA node" interchangeably. You're
demonstrating that they are not the same thing. Is that just a
different way of labeling the same thing, or are NUMA nodes and
proximity domains fundamentally different concepts.
Your code in patch 5 (kfd_proximity_domain_to_numa_node and
kfd_numa_node_to_proximity_domain) seems to imply that there is, in
fact, a 1:1 mapping, as it assumes that there is a unique
translation in both directions. Am I missing something?
Thanks for the comment.
IIUC, NUMA node IDs and proximity domains are different numbering
schemes. We have proximity domains for both CPU and GPU devices. CPU
proximity domains start from 0, and GPU proximity domains start after
the last CPU proximity domain.
If the NUMA node IDs are contiguous, the CPU proximity domains happen
to match the NUMA node IDs. However, if the NUMA node IDs are
discontiguous, the CPU proximity domains and NUMA node IDs no longer
have a one-to-one mapping.
If they are just different numbering systems, do we really need to
keep track of both proximity domains and NUMA nodes in the KFD
topology? Or would it be sufficient to only track NUMA nodes, if
that's what we really care about in the uAPI (KFD sysfs)?
Thanks for the suggestion. I also think we don't need to keep track
of both the proximity domains and the NUMA node IDs in KFD. Do you
think the approach below would be reasonable?
Just to make sure I understand correctly, for CPU devices the
proximity domain will be the same as the NUMA node ID, and the GPU
proximity domains will start after the last CPU proximity domain.
In that case, the CPU proximity domains and NUMA node IDs will always
have a one-to-one mapping, and the GPU proximity domains will follow
after them.
For example, if a system has three NUMA nodes and two GPUs:
NUMA node IDs: 0 2 3
CPU proximity domains: 0 2 3
GPU proximity domains: 4 5
Would it be okay to proceed with this approach?
Yes, this looks good to me.
Hi Felix,
I have been looking into this issue along with Donet.
From my understanding, there are three related but distinct concepts
involved here:
* NUMA node: A system can have sparse NUMA node IDs. For example,
online nodes can be 0, 2, and 3, with node 1 absent.
* Proximity domain: The proximity domain is the identifier
associated with a KFD topology device. In the current implementation,
proximity domains for CPU topology devices are assigned using a
sequential counter, and GPU proximity domains are allocated after the
CPU proximity domains.
* KFD sysfs topology node: KFD exposes each topology device through
a node under its sysfs topology interface. Userspace reads these sysfs
nodes to discover the CPU/GPU topology and their relationships. In the
current implementation, the sysfs node numbering corresponds to the
proximity domain numbering.
IIUC, the proximity-domain numbering and KFD sysfs topology-node
numbering are expected to correspond, while the Linux NUMA node ID can
be a separate identifier that is mapped to the corresponding topology
or proximity domain.
As discussed in the thread, we implemented the
'NUMA node ==Proximity Domain' approach. However, after implementing
this for sparse nodes, we found that the userspace libraries using the
KFD sysfs nodes have an assumption that the topology devices are contiguous.
With this approach, discontiguous NUMA nodes result in discontiguous
proximity-domain and sysfs node numbers. This would break userspace
libraries that assume the topology devices are contiguous.
To support this approach, we would need to make corresponding changes
in the userspace libraries, which would break applications using the
existing userspace and introduce backward-compatibility concerns.
With this in mind, we think the current RFC v1 approach would be the
better way to proceed, keeping the NUMA node IDs separate from the
proximity-domain/sysfs numbering.
Would you agree with this approach? If so, I can rebase the RFC v1
series onto the latest kernel and post a new version for review.
Thanks,
Dhruv B
Regards,
Felix
I think this approach should also resolve the driver loading issue.
Thanks
Donet Tom
Thanks,
Felix
The failure is observed as:
amdgpu: Virtual CRAT table created for GPU
amdgpu: Error parsing VCRAT
kfd: amdgpu: Error adding device to topology
kfd: amdgpu: Error initializing KFD node
Since every online NUMA node is a valid topology object and can
contain
CPUs, memory, I/O links, or any combination of these, topology devices
and proximity domains should be created for every online NUMA node
rather than only for NUMA nodes that contain CPUs.
To address this, this series introduces a new VCRAT subtype that
records the mapping between the NUMA node ID and the generated VCRAT
proximity domain. When topology devices are created, this information
is stored in the corresponding topology device, allowing the driver to
translate a NUMA node ID into its associated proximity domain whenever
required.
Returning to the previous example, the system contains three online
NUMA nodes, so three CPU topology devices and three proximity domains
are created, even though only one NUMA node contains CPUs. The NUMA
node ID is stored in each topology device together with its generated
proximity domain.
Later, when the GPU VCRAT is generated, the driver only knows the NUMA
node ID to which the GPU is attached (for example, node 3). Instead of
assuming that the NUMA node ID is equal to the proximity domain, the
driver walks the existing topology devices to locate the corresponding
NUMA node and retrieves its generated proximity domain. In this
example, NUMA node 3 maps to proximity domain 2, so
proximity_domain_to is populated with the correct value.
Since the GPU I/O link now references a valid proximity domain, VCRAT
parsing completes successfully and topology initialization proceeds
without errors on systems with sparse NUMA node IDs, CPU-less NUMA
nodes, and CPU-less/memory-less NUMA nodes.
This series consists of the following patches:
Patch 1 removes an unused argument from
kfd_create_crat_image_virtual() as a preparatory cleanup.
Patch 2 introduces a new VCRAT NUMA affinity subtype that stores the
NUMA node ID and its corresponding proximity domain.
Patch 3 populates the NUMA affinity entries during VCRAT generation
for every online NUMA node.
Patch 4 parses the NUMA affinity entries from the VCRAT and stores the
NUMA node ID and proximity domain in the corresponding topology
device.
Patch 5 fixes GPU VCRAT proximity domain mappings by translating the
GPU's NUMA node ID to the corresponding CPU proximity domain before
programming proximity_domain_to.
Patch 6 creates proximity domains and VCRAT entries for all online
NUMA nodes, including CPU-less and memory-less nodes.
Patch 7 adds a numa_node sysfs attribute for CPU and GPU topology
devices, exposing the associated NUMA node through the topology sysfs
interface.
Please note that the changes in this series are on a best effort
basis from our
end. Therefore, requesting the amd-gfx community (who have deeper
knowledge of the
HW & SW stack) to kindly help with the review and provide feedback
/ comments on
these patches
Donet Tom (7):
drm/amdgpu: Remove unused argument from
kfd_create_crat_image_virtual
drm/amdgpu: Add VCRAT NUMA affinity entry
drm/amdgpu: Populate NUMA affinity entries in VCRAT
drm/amdgpu: Parse NUMA affinity entries from VCRAT
drm/amdgpu: Fix VCRAT proximity domain mappings for GPU nodes
drm/amdgpu: Create proximity domains and VCRAT entries for CPU-less
and memory-less NUMA nodes
drm/amdgpu: Add numa_node in cpu/gpu topology device sysfs entry
drivers/gpu/drm/amd/amdkfd/kfd_crat.c | 152
+++++++++++++++-------
drivers/gpu/drm/amd/amdkfd/kfd_crat.h | 19 ++-
drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 49 +++++--
drivers/gpu/drm/amd/amdkfd/kfd_topology.h | 4 +
4 files changed, 168 insertions(+), 56 deletions(-)
