On 2/21/26 12:39 PM, Donet Tom wrote:
This is v3 of the patch series enabling 64 KB system page size support
in AMDGPU. v2, part 1 of this series [1] has already been merged
upstream and provides the minimal infrastructure required for 64 KB
page support.
This series addresses additional issues uncovered in AMDGPU when
running rccl unit tests and rocr-debug-agent tessts on 64KB page-size
systems.
With this series applied, all RCCL unit tests and rocr-debug-agent
tests pass on systems using a 64 KB system page size, across
multi-GPU configurations, with XNACK both enabled and disabled.
Patch 1 in this series (drm/amdgpu: Change AMDGPU_VA_RESERVED_TRAP_SIZE
to 2 * PAGE_SIZE) fixes a kernel crash observed when running rocminfo
on systems with a 64 KB page size. This patch is required to enable
minimal support for 64 KB system page sizes.
Since RFC v2, we observed AQL queue creation failures while running
certain workloads on 64K page-size systems due to an expected queue size
mismatch. This issue is addressed in patch 2 of this series.
The questions we had in this seres are:
=======================================
1 When the control stack size is aligned to 64 KB, we consistently
observe queue preemption or eviction failures on gfx9, on both
4 KB and 64 KB system page-size configurations.
The control stack size is calculated based on the number of CUs and
waves and is then aligned to PAGE_SIZE. On systems with a 64 KB
system page size, this alignment always results in a 64 KB-aligned
control stack size, after which queue preemption fails.
Is there any hardware-imposed limitation on gfx9 that prevents the
control stack size from being 64 KB? For gfx10, I see explicit
hardware limitations on the control stack size in the code [2].
Is there anything similar for gfx9?
What is the correct or recommended control stack size for gfx9?
With a 4 KB system page size, I observe a control stack size of
around 44 KB—can it grow beyond this? If the control stack size
is fixed for a given gfx version, do you see any issues with
aligning the control stack size to the GPU page size?
This series has 5 patches
=========================
1. AMDGPU_VA_RESERVED_TRAP_SIZE was hard-coded to 8 KB while
KFD_CWSR_TBA_TMA_SIZE is defined as 2 * PAGE_SIZE, which matches on
4 KB page-size systems but results in a size mismatch on 64 KB
systems, leading to kernel crashes when running rocminfo or RCCL
unit tests.
This patch updates AMDGPU_VA_RESERVED_TRAP_SIZE to 2 * PAGE_SIZE so
that the reserved trap area matches the allocation size across all
system page sizes. This is a must needed patch to enable minimal
support for 64 KB system page sizes.
2. Aligned expected_queue_size to PAGE_SIZE to fix AQL queue creation
failure.
3. Fix amdgpu page fault handler (for xnack) to pass the corresponding
system pfn (instead of gpu pfn) for restoring SVM range mapping.
4. Updated AMDGPU_GTT_MAX_TRANSFER_SIZE to always match the PMD size
across all page sizes.
5. On systems where the CPU page size is larger than the GPU’s 4 KB page
size, the MQD and control stack were aligned to the CPU PAGE_SIZE,
causing multiple GPU pages to incorrectly inherit the UC attribute.
This change aligns both regions to the GPU page size, ensuring that
the MQD is mapped as UC and the control stack as NC, restoring the
correct behavior.
6. Queue preemption fails when the control stack size is aligned to
64 KB. This patch fixes this issue by aligning the control stack
size to gpu page size.
Setup details:
============
System details: Power10 LPAR using 64K pagesize.
AMD GPU:
Name: gfx90a
Marketing Name: AMD Instinct MI210
[1] https://lore.kernel.org/all/[email protected]/
[2]
https://elixir.bootlin.com/linux/v6.19-rc5/source/drivers/gpu/drm/amd/amdkfd/kfd_queue.c#L457
RFC V2 -
https://lore.kernel.org/all/[email protected]/
RFC V1 -
https://lore.kernel.org/all/[email protected]/
Donet Tom (6):
drm/amdgpu: Change AMDGPU_VA_RESERVED_TRAP_SIZE to 2 PAGE_SIZE pages
drm/amdkfd: Align expected_queue_size to PAGE_SIZE
drm/amdgpu: Handle GPU page faults correctly on non-4K page systems
drm/amdgpu: Fix AMDGPU_GTT_MAX_TRANSFER_SIZE for non-4K page size
drm/amd: Fix MQD and control stack alignment for non-4K
drm/amdkfd: Fix queue preemption/eviction failures by aligning control
stack size to GPU page size
Hi All,
Gentle ping.
Could you please review this patch series and share your feedback?
We have tested this series on both 4K and 64K page-size kernels, and all
RCCL tests and ROCR debug agent tests are passing. If everything looks
good, could we get this merged in the next kernel release (7.1)?
Thanks for your review.
-Donet
drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c | 44 +++++++++++++++++++
drivers/gpu/drm/amd/amdgpu/amdgpu_gart.h | 2 +
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 24 ++++------
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 2 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 6 +--
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 2 +-
drivers/gpu/drm/amd/amdgpu/vce_v1_0.c | 3 +-
.../gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c | 23 ++++++----
drivers/gpu/drm/amd/amdkfd/kfd_queue.c | 11 ++---
9 files changed, 82 insertions(+), 35 deletions(-)
