Hi Christian, Philip, Alex, Felix, and all, These series introduce a new Shared Virtual Memory (SVM) implementation for amdgpu that is built directly on top of the common DRM GPUSVM / drm_pagemap core (drivers/gpu/drm/drm_gpusvm.c, drm_pagemap.c) rather than on the existing KFD-private SVM code in amdkfd/kfd_svm.c.
The goal is to provide HMM-based unified memory through the same shared infrastructure that the Xe driver already uses, so that amdgpu and xe converge on one well-reviewed SVM/page-migration core instead of maintaining a separate amdgpu-specific stack. Dependencies ============ This series is rebased on top of Christian's amdgpu VM rework series and depends on it. The SVM locking work in patches 13-15 builds directly on the eviction-lock critical section and the amdgpu_vm_update_range() split introduced there, so this series does not apply or build without it: https://lore.kernel.org/amd-gfx/[email protected]/ Phased plan =========== This work is being upstreamed in phases to keep each submission reviewable: * Phase 1 (this series): single GPU, XNACK on (retry faults). - v9 https://lore.kernel.org/amd-gfx/[email protected]/ * Phase 2 (in progress): single GPU, XNACK off (no-retry / eager mapping). - https://lore.kernel.org/amd-gfx/[email protected]/ - https://lore.kernel.org/amd-gfx/[email protected]/ * Phase 3 (in progress): multiple GPUs (XGMI mapping, P2P device-to-device migration). - https://lore.kernel.org/amd-gfx/[email protected] Accordingly, this first submission targets the simplest useful configuration: * Single GPU with local VRAM migration: transparent RAM <-> VRAM page migration on one GPU via SDMA, with TTM-based eviction for overcommit. * XNACK on (retry faults): GPU page faults are retried after the driver lazily populates PTEs on demand. The XNACK-off/eager-mapping path is deferred to Phase 2. * Compute VMs only (amdgpu_vm_make_compute()), matching the KFD SVM use case. Everything is gated behind a new, default-n Kconfig option (CONFIG_DRM_AMDGPU_SVM) and is therefore opt-in and isolated from existing users until the feature matures. UAPI ==== A new render-node ioctl, DRM_IOCTL_AMDGPU_GEM_SVM, lets userspace describe SVM attributes over a CPU virtual-address interval (patch 1): - operations: SET_ATTR / GET_ATTR / RESET_ATTR - access modes: INACCESSIBLE / IN_PLACE / ALLOW_MIGRATE - location hints: SYSMEM / UNDEFINED / LOCAL / HIVE (preferred_loc, prefetch_loc). The location is a fixed enum rather than a device file descriptor on purpose: a descriptor has no stable meaning across a checkpoint/restore cycle. - per-range flags: HOST_ACCESS, COHERENT, EXT_COHERENT, GPU_RO, GPU_EXEC, GPU_READ_MOSTLY, plus a granularity hint GET_ATTR additionally returns, in @size, the length of the contiguous range starting at @start_addr over which the queried attributes are constant, so user space (e.g. CRIU) can walk the whole address space without knowing the range boundaries in advance. Attributes are stored in a per-VM interval tree; the GPU page tables are populated lazily on demand by retry faults, or eagerly on a prefetch request. User Space Work =============== - libdrm exposes the new SVM ioctl: https://gitlab.freedesktop.org/mesa/libdrm/-/merge_requests/485 - ROCm runtime adaptation for the new amdgpu DRM SVM API: https://github.com/ROCm/rocm-systems/pull/8318 Design Overview =============== The implementation is split into clearly layered modules: amdgpu_svm.c Core context (struct amdgpu_svm embeds struct drm_gpusvm), kref lifecycle, PASID lookup, drm_gpusvm_ops, the GEM_SVM ioctl entry point, and the GC workqueue. amdgpu_svm_attr.c The attribute interval tree: validation, gap/overlap split-merge, SET/GET/RESET, and change-trigger classification that decides whether an attribute change needs PTE invalidation or a remap. amdgpu_svm_range.c Per-range GPU mapping: PTE-flag computation per GC IP version, DMA-segment coalescing, MMU-notifier begin/end handling, PTE zapping, and the garbage collector. amdgpu_svm_fault.c The retry-fault entry point amdgpu_svm_handle_fault() and the fault_map_range() pipeline. amdgpu_migrate.c drm_pagemap / ZONE_DEVICE VRAM migration backend. amdgpu_svm_range_migrate.c Per-range RAM<->VRAM migration helpers. Fault path (XNACK on): amdgpu_vm_handle_fault() routes a compute-VM retry fault to amdgpu_svm_handle_fault() when the VM has an SVM context. After PASID lookup and a checkpoint-timestamp filter that drops stale retry faults left over from a recent unmap, it looks up (or synthesizes a default) attribute range and runs fault_map_range(): garbage-collect -> VMA permission check -> find/insert range -> short-circuit if recently validated or already valid -> drm_gpusvm get_pages (HMM) -> program GPU PTEs under the notifier lock with a notifier-sequence re-check. Locking (patches 13-15): Following the v9 review, amdgpu_vm's eviction lock is now a selectable read/write semaphore (struct rw_semaphore *eviction_lock) instead of a private mutex. On an SVM VM it is pointed at the drm_gpusvm notifier lock, so page-table updates, MMU-notifier invalidation and TTM eviction all serialize on one lock rather than on separate, unordered ones. Updates made with the root PD reserved take the read side; updates without it, and eviction, take the write side. Because page-table allocation can enter reclaim - and reclaim re-enters the MMU notifier that wants the same lock - the lock is dropped around amdgpu_vm_pt_create() and the range sequence number is rechecked on reacquire, returning -EAGAIN if it moved. Plain graphics VMs keep their own default lock and are unaffected. Invalidation / GC: MMU-notifier callbacks zap the affected PTEs (batched, single heavyweight TLB flush) and, for unmap events, queue the range to a high-priority GC workqueue that removes it via drm_gpusvm_range_remove() outside notifier context. VRAM migration (patches 16-22): A drm_pagemap is registered over the GPU's VRAM as a ZONE_DEVICE region at device-init / reset-restore time. struct amdgpu_bo_svm (a new BO subtype) backs migrated ranges, with SDMA-based copy_to_devmem / copy_to_ram callbacks using a GART window. TTM eviction of SVM BOs synchronously migrates pages back to system memory, which keeps VRAM overcommit working. The fault and prefetch paths call amdgpu_svm_range_migrate_to_vram() before mapping when migration is requested, with a single -EBUSY retry that evicts conflicting pages first. Patch 16 ("drm/pagemap: fix page undercount for compound src entries") is a prerequisite fix in the shared drm_pagemap core, found while enabling THP migration on amdgpu. It is independent of this series and can be applied on its own. Test Results ============ Tested on gfx943 (MI300X) and gfx906 (MI60) with XNACK on: - KFD test: 95%+ passed. - ROCR test: all passed. - HIP catch test: gfx943 (MI300X): 99% passed. gfx906 (MI60): 99% passed. These numbers come from running the existing KFD and ROCR test suites over the new DRM SVM path, which is enabled by: https://github.com/ROCm/rocm-systems/pull/13719 IGT test suites covering amdgpu DRM SVM directly will follow. Changes from Old Version ======================== In previous V8 version, we had consolidated all implementations for the single-GPU XNACK-on mode. This approach followed Christian's suggestion to make the code review process more straightforward. Previously, the XNACK-on implementation was split into two parts: - Basic V7 https://lore.kernel.org/amd-gfx/[email protected]/ - Migration V5 https://lore.kernel.org/amd-gfx/[email protected]/ Above two parts are no longer needed to be reviewed separately, please focus on this series. Changes from V9 to V10: - Rebased on top of Christian's amdgpu VM rework series (see Dependencies above), which this series now requires. The GEM_SVM ioctl number moved from 0x1B to 0x1C. - eviction_lock: the V9 limitation is resolved by three new patches (13-15). amdgpu_vm's eviction lock becomes a selectable rwsem aliased to the drm_gpusvm notifier lock on an SVM VM, so page table updates, MMU notifier invalidation and eviction serialize on one lock. It is dropped around amdgpu_vm_pt_create() to avoid the reclaim/notifier recursion, with an -EAGAIN recheck on reacquire. - Added "drm/pagemap: fix page undercount for compound src entries" as a prerequisite, and made the SDMA copy path folio aware, so THP-backed ranges migrate correctly. - UAPI: the location enum gained LOCAL/HIVE and UNDEFINED changed from 0xffffffffU to 1, replacing the AMDGPU_SVM_ATTR_HIVE_LOCAL flag. GET_ATTR now returns in @size the length of the range over which the queried attributes are constant, so CRIU can walk the address space. - Review fixes: the GEM_SVM ioctl no longer creates the SVM context lazily, the missing root BO reservation around context create/destroy was added, the mapping publication moved inside the notifier lock, two range-queueing bugs were fixed, and several error paths now report failure (-EPERM on a write to a read-only range, -EFAULT on a missing VMA, unknown attribute values, TLB flush failures) instead of being silently ignored. - Cleanups: dropped the private log wrappers in favour of pr_debug() and pr_err_ratelimited(), partition-scoped TLB flushing, kzalloc_obj(), and removed the leftover clean-svm Makefile target. Changes from V8 to V9: - This series has been rebased on top of the amdgpu vm fix. - https://lore.kernel.org/amd-gfx/[email protected]/ - amdgpu_svm.h: dropped the AMDGPU_SVM_KMEM_CACHE_* macros; SVM ranges use plain kzalloc()/kfree(). - amdgpu_svm.h: dropped the flush_tlb callback; amdgpu_svm_flush_tlb() is called directly. - amdgpu_svm_attr.h: added kerneldoc for the interval tree and list. - amdgpu_svm_attr.c: dropped the attr slab cache; uses kzalloc()/kfree(). - amdgpu_svm_attr.c: squashed the types header into its implementation patch - amdgpu_svm_attr.c: renamed the local "flags" to "unsupported_vm_flags". - amdgpu_svm_attr.c: dropped amdgpu_svm_attr_validate_range_vma() from the fault path; hmm_range_fault() covers it. A lighter sync check is kept only at the set/get attr ioctl entry for fix regression. - amdgpu_svm_range.h: AMDGPU_SVM_RANGE_DEBUG removed, replaced by a tracepoint. - amdgpu_svm_range.h: dropped the UNMAP_WORK macro. - amdgpu_svm_range.c: the fast range valid check is now lock free. - amdgpu_svm_range.c: zap_ptes always drains the fence regardless of the update return value. - per IP PTE flags: moved the switch out of the SVM core into gmc_v9_0.c/gmc_v12_0.c behind a new gmc_funcs->get_svm_pte_flags() callback. - eviction_lock: drm_gpusvm framwork's limitation blocks using the eviction_lock kept as is. - SVM/BO overlap: dropped the scan-based amdgpu_svm_attr_check_vm_bo() helper. - Added drm_pagemap refcount-based deferred destroy for amdgpu_pagemap: the ZONE_DEVICE mapping is torn down from a destroy worker (amdgpu_pagemap_destroy_work) once the drm_pagemap refcount reaches zero, using drm_dev_enter()/exit() to guard against device teardown - Added pre_migrate_fence handling in the SDMA copy callbacks: wait for outstanding async VRAM operations (DMA_RESV_USAGE_KERNEL) to complete Thanks, Ray/Honglei/Junhua Honglei Huang (15): drm/amdgpu: add SVM ioctl UAPI definitions drm/amdgpu: add SVM core header and VM integration drm/amdgpu: implement SVM attribute tree and helper functions drm/amdgpu: implement SVM attribute set/get/clear operations drm/amdgpu: add SVM range types and work queue interface drm/amdgpu/gmc: add get_svm_pte_flags callback drm/amdgpu: implement SVM range GPU mapping core drm/amdgpu: implement SVM range notifier and GC helpers drm/amdgpu: add SVM notifier invalidate callback and checkpoint drm/amdgpu: implement SVM initialization and lifecycle drm/amdgpu: add SVM ioctl entry and fault handler module drm/amdgpu: integrate SVM into build system and VM fault path drm/amdgpu: make VM eviction lock read/write to match notifier lock drm/amdgpu: recheck range validity after page table allocation drm/amdgpu: use drm_gpusvm notifier lock for VM eviction Junhua Shen (7): drm/pagemap: fix page undercount for compound src entries drm/amdgpu: add VRAM migration infrastructure for drm_pagemap drm/amdgpu: implement drm_pagemap SDMA migration callbacks drm/amdgpu: implement synchronous TTM eviction for SVM BOs drm/amdgpu: hook up ZONE_DEVICE registration in device init and reset drm/amdgpu: add SVM range migration helpers for drm_pagemap drm/amdgpu: integrate VRAM migration into SVM fault and prefetch paths drivers/gpu/drm/amd/amdgpu/Kconfig | 10 + drivers/gpu/drm/amd/amdgpu/Makefile | 4 + drivers/gpu/drm/amd/amdgpu/amdgpu.h | 8 + .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 + drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 4 +- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h | 24 + drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c | 899 ++++++++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h | 107 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 4 +- drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c | 2 + drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c | 858 +++++++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h | 196 ++++ drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c | 977 ++++++++++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h | 202 ++++ drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c | 476 +++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.h | 39 + drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 812 +++++++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h | 191 ++++ .../drm/amd/amdgpu/amdgpu_svm_range_migrate.c | 120 +++ .../drm/amd/amdgpu/amdgpu_svm_range_migrate.h | 35 + drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h | 30 + drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 20 + drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 24 +- drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 111 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 19 +- .../gpu/drm/amd/amdgpu/amdgpu_vm_internal.h | 30 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c | 32 +- drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c | 69 ++ drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c | 119 +++ drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 7 +- drivers/gpu/drm/drm_pagemap.c | 60 +- include/uapi/drm/amdgpu_drm.h | 127 +++ 36 files changed, 5541 insertions(+), 85 deletions(-) create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.h create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.c create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.h -- 2.53.0
