Public bug reported:
Affects: linux-nvidia (Ubuntu), linux-nvidia-bos (Ubuntu)
---
On Vera Rubin systems, a GPU that writes through SVA (shared virtual
addressing) into anonymous memory freed with madvise(MADV_FREE) either
hangs or runs far below normal speed:
- With an older NVIDIA UVM driver in coherent-driver memory mode, a pageable
device-to-host copy into such a buffer never completes. Applications see this
as an NCCL broadcast hang.
- With current drivers the copy completes, but at about 0.2 GB/s instead of
about 360 GB/s.
MADV_FREE leaves the buffer's PTEs clean: read-only, with the Dirty Bit
Modifier (DBM) set. The CPU's next write marks the page dirty in
hardware. The SMMU can do the same for a shared process page table, but
only if the SVA context descriptor sets CD.HA and CD.HD. The 7.0
kernel's arm_smmu_make_sva_cd() never sets them, even though Vera's
SMMUs support hardware Access and Dirty updates (IDR0.HTTU=2). Every GPU
write to such a page therefore faults, and the device driver must fix up
the PTE in software, one page per fault.
Fix: backport mainline commit 74fa4c177ad0 ("iommu/arm-smmu-v3-sva:
Enable Hardware Access and Hardware Dirty bits"), from v7.2. Stable
7.0.y ended at 7.0.14, before it landed.
Testing on a Vera Rubin NVL72 compute tray (2 Vera CPUs, 4 Rubin GPUs),
linux-nvidia-bos 7.0 with the commit, against the stock
7.0.0-2023-nvidia-bos kernel:
- Older driver, coherent-driver mode, 60 runs: 54/60 hung on the stock kernel.
None hung with the commit.
- Current driver: the MADV_FREE device-to-host copy goes from 0.20 GB/s to
274-282 GB/s. A copy into already-dirty memory runs at 362 GB/s on both kernels.
- Data integrity: GPU writes into MADV_FREE'd pages, followed by forced memory
reclaim, kept all of their data in every configuration.
- Soak: 30 minutes on all 4 GPUs, racing GPU writes against reclaim with swap,
page migration, MADV_COLD/MADV_PAGEOUT, fork and NUMA balancing. No word out of
4.2e11 checked was wrong, nothing hung, and no SMMU or GPU errors were logged.
- Hardware without HTTU (Grace Hopper): the new bits are gated on the SMMU's
HA/HD feature flags, so nothing changes.
** Affects: linux-nvidia-bos (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2172334
Title:
arm-smmu-v3: SVA does not enable hardware Access/Dirty updates, so GPU
writes to MADV_FREE memory hang or crawl on Vera
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-nvidia-bos/+bug/2172334/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
