gies?
The issue we see on a system with discontiguous NUMA nodes is that the
GPU VCRAT parsing fails when we load the GPU driver, resulting in the
following errors in dmesg:
amdgpu: Virtual CRAT table created for GPU
amdgpu: Error parsing VCRAT
kfd: amdgpu: Error adding device to topology
kfd: amdgpu: Error initializing KFD node
On our system, we have 3 NUMA nodes: 0, 2, and 3, with 2 GPUs attached
to NUMA node 3.
With the current upstream implementation, CPU proximity domains are
assigned sequentially. Thus, NUMA nodes 0, 2, and 3 get CPU proximity
domains 0, 1, and 2 respectively. The two GPUs then get proximity
domains 3 and 4.
However, when generating the GPU VCRAT, `proximity_domain_to` is
populated directly with the NUMA node ID. Since both GPUs are attached
to NUMA node 3, the VCRAT contains:
GPU0 -> proximity_domain_to = 3
GPU1 -> proximity_domain_to = 3
https://elixir.bootlin.com/linux/v7.1/source/drivers/gpu/drm/amd/amdkfd/kfd_crat.c#L2175
Here, the CPU proximity domain corresponding to NUMA node 3 is actually
2, while proximity domain 3 belongs to GPU0.
During VCRAT parsing, the I/O link parser looks up this `id_to`
proximity domain, which is 3, and returns `-ENODEV` because there is no
CPU topology device with proximity domain 3:
https://elixir.bootlin.com/linux/v7.1/source/drivers/gpu/drm/amd/amdkfd/kfd_crat.c#L1284
Thus, the GPU I/O link is referencing the NUMA node ID as if it were
the CPU proximity domain, which causes the VCRAT parsing to fail.
Since the parsing fails, the driver does not get loaded.
Thanks,
Dhruv B
Regards,
Felix
Would you agree with this approach? If so, I can rebase the RFC v1
series onto the latest kernel and post a new version for review.
Thanks,
Dhruv B
Regards,
Felix
I think this approach should also resolve the driver loading issue.
Thanks
Donet Tom
Thanks,
Felix
The failure is observed as:
amdgpu: Virtual CRAT table created for GPU
amdgpu: Error parsing VCRAT
kfd: amdgpu: Error adding device to topology
kfd: amdgpu: Error initializing KFD node
Since every online NUMA node is a valid topology object and can
contain
CPUs, memory, I/O links, or any combination of these, topology
devices
and proximity domains should be created for every online NUMA node
rather than only for NUMA nodes that contain CPUs.
To address this, this series introduces a new VCRAT subtype that
records the mapping between the NUMA node ID and the generated VCRAT
proximity domain. When topology devices are created, this
information
is stored in the corresponding topology device, allowing the
driver to
translate a NUMA node ID into its associated proximity domain
whenever
required.
Returning to the previous example, the system contains three online
NUMA nodes, so three CPU topology devices and three proximity
domains
are created, even though only one NUMA node contains CPUs. The NUMA
node ID is stored in each topology device together with its
generated
proximity domain.
Later, when the GPU VCRAT is generated, the driver only knows the
NUMA
node ID to which the GPU is attached (for example, node 3).
Instead of
assuming that the NUMA node ID is equal to the proximity domain, the
driver walks the existing topology devices to locate the
corresponding
NUMA node and retrieves its generated proximity domain. In this
example, NUMA node 3 maps to proximity domain 2, so
proximity_domain_to is populated with the correct value.
Since the GPU I/O link now references a valid proximity domain,
VCRAT
parsing completes successfully and topology initialization proceeds
without errors on systems with sparse NUMA node IDs, CPU-less NUMA
nodes, and CPU-less/memory-less NUMA nodes.
This series consists of the following patches:
Patch 1 removes an unused argument from
kfd_create_crat_image_virtual() as a preparatory cleanup.
Patch 2 introduces a new VCRAT NUMA affinity subtype that stores the
NUMA node ID and its corresponding proximity domain.
Patch 3 populates the NUMA affinity entries during VCRAT generation
for every online NUMA node.
Patch 4 parses the NUMA affinity entries from the VCRAT and
stores the
NUMA node ID and proximity domain in the corresponding topology
device.
Patch 5 fixes GPU VCRAT proximity domain mappings by translating the
GPU's NUMA node ID to the corresponding CPU proximity domain before
programming proximity_domain_to.
Patch 6 creates proximity domains and VCRAT entries for all online
NUMA nodes, including CPU-less and memory-less nodes.
Patch 7 adds a numa_node sysfs attribute for CPU and GPU topology
devices, exposing the associated NUMA node through the topology
sysfs
interface.
Please note that the changes in this series are on a best effort
basis from our
end. Therefore, requesting the amd-gfx community (who have deeper
knowledge of the
HW & SW stack) to kindly help with the review and provide feedback
/ comments on
these patches
Donet Tom (7):
drm/amdgpu: Remove unused argument from
kfd_create_crat_image_virtual
drm/amdgpu: Add VCRAT NUMA affinity entry
drm/amdgpu: Populate NUMA affinity entries in VCRAT
drm/amdgpu: Parse NUMA affinity entries from VCRAT
drm/amdgpu: Fix VCRAT proximity domain mappings for GPU nodes
drm/amdgpu: Create proximity domains and VCRAT entries for
CPU-less
and memory-less NUMA nodes
drm/amdgpu: Add numa_node in cpu/gpu topology device sysfs entry
drivers/gpu/drm/amd/amdkfd/kfd_crat.c | 152
+++++++++++++++-------
drivers/gpu/drm/amd/amdkfd/kfd_crat.h | 19 ++-
drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 49 +++++--
drivers/gpu/drm/amd/amdkfd/kfd_topology.h | 4 +
4 files changed, 168 insertions(+), 56 deletions(-)
