On 2026-10-05 10:00, Dhruv B wrote:
On systems with discontiguous NUMA nodes, the GPU driver fails to load
with the following errors:
[ 5.609767] amdgpu 0038:12:00.0: Error parsing VCRAT
[ 5.609781] kfd kfd: Error adding device to topology
[ 5.609848] kfd kfd: Error initializing KFD node
In the current KFD code, proximity domains are created contiguously
when generating the VCRAT. With discontiguous NUMA node IDs, this
creates a mismatch between the proximity domain ID and the NUMA node ID.
For example, if NUMA nodes 0, 2 and 3 are online, the current code
assigns proximity domains 0, 1 and 2 to these nodes respectively,
instead of domains 0, 2 and 3.
The KFD topology uses the NUMA node and proximity domain information to
create topology devices representing the CPU and memory resources
associated with each NUMA node. This topology is exposed to userspace
through sysfs, where the NUMA node and proximity domain information are
used to identify the topology devices. The mismatch between the NUMA
node ID and proximity domain therefore results in an inconsistent KFD
topology and causes topology parsing to fail.
Make the proximity domain ID correspond directly to the NUMA node ID by
iterating through the complete NUMA node ID range up to the last online
node. Account for every node ID when assigning proximity domains, while
skipping resource entry creation for offline or unpopulated nodes.
This preserves the NUMA-node-to-proximity-domain mapping and allows the
KFD topology and its sysfs representation to correctly reflect
discontiguous NUMA topologies.
Co-developed-by: Donet Tom <[email protected]>
Signed-off-by: Donet Tom <[email protected]>
Signed-off-by: Dhruv B <[email protected]>
---
drivers/gpu/drm/amd/amdkfd/kfd_crat.c | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index 99b0c7f825fe..d4739db1c14c 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -1899,7 +1899,7 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image,
size_t *size)
acpi_status status;
struct crat_subtype_generic *sub_type_hdr;
int avail_size = *size;
- int numa_node_id;
+ int numa_node_id, last_node;
#if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
uint32_t entries = 0;
#endif
@@ -1935,9 +1935,12 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image,
size_t *size)
crat_table->num_domains = 0;
sub_type_hdr = (struct crat_subtype_generic *)(crat_table+1);
+ last_node = find_last_bit(node_online_map.bits, MAX_NUMNODES);
- for_each_online_node(numa_node_id) {
- if (kfd_numa_node_to_apic_id(numa_node_id) == -1)
+ for (numa_node_id = 0; numa_node_id <= last_node; numa_node_id++) {
+ crat_table->num_domains++;
Doing this at the beginning of the loop body, doesn't that result in an
incorrect proximity_domain being used in kfd_fill_cu_for_cpu? E.g. the
first CPU will get proximity_domain 1 instead of 0.
Regards,
Felix
+
+ if (!node_online(numa_node_id))
continue;
/* Fill in Subtype: Compute Unit */
@@ -1982,8 +1985,6 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image,
size_t *size)
#else
pr_info("IO link not available for non x86 platforms\n");
#endif
-
- crat_table->num_domains++;
}
/* TODO: Add cache Subtype for CPU.
