The backing pages of an Xe system memory BO are allocated on the device local node (dev_to_node()), or on the allocating CPU's node when the device has none. That is fine when CPU and GPU access a buffer from the same place, but not in general:
- On multi-socket systems, and where the GPU's PCIe/CXL link attaches to a different node than the threads using a buffer, the cost of accessing a BO depends on which node backs its pages. The device local node is not always the best choice. - A runtime often knows where a BO should live: the node of the CPU threads producing or consuming it, the node closest to the link the GPU uses, or a node chosen to balance memory bandwidth. Today it has no way to express that for driver allocated system memory. - mbind()/set_mempolicy() do not help: BO pages are allocated by the driver, not through the task's fault path, and BO CPU mappings are VM_PFNMAP. This series lets userspace select the host NUMA node per BO at creation time. UAPI: - New DRM_XE_GEM_CREATE_SET_PROPERTY_NUMA_NODE property on the existing GEM_CREATE set-property extension. - The node is a preference (best effort, like MPOL_PREFERRED); the allocation falls back to other nodes under memory pressure. - A negative value, or not setting the property, keeps the current default placement. - Invalid or offline nodes, and BOs that can't be placed in system memory, are rejected with -EINVAL. - The placement is preserved across shrinker backup / restore. - A richer placement policy can be added later as a separate extension without changing this one. Why a node aware TTM allocation API rather than one ttm_pool per node in the driver: The NUMA node is a property of each allocation, not of the pool. TTM already tracks pages per node (per-node LRUs in the pool caches, per-node shrinker, per-node cache limits) and __ttm_pool_alloc() already takes a node internally. ttm_pool_alloc_node() and ttm_tt_restore_node() only let a caller pass that node in, following the alloc_pages_node() / kmalloc_node() pattern, with no new state. One pool per node in the driver would work for Xe today, but: - TTM core helpers (ttm_tt_restore(), ttm_tt_backup(), pool debugfs) all use the device pool. The driver would have to bypass them for populate, restore, backup and free, and re-check that whenever TTM adds or changes a helper. The restore issue fixed in this series is an example: the placement was silently lost after backup. - It relies on non dma_alloc pools having no per-pool state. For dma_alloc pools, N pools mean N separate page caches that can't share pages. - Every TTM driver that wants per-BO placement would have to duplicate the same per-node pool handling. The existing ttm_pool_alloc() and ttm_tt_restore() become wrappers passing NUMA_NO_NODE, so current users are unaffected. Himal Prasad Ghimiray (7): drm/ttm/pool: Add ttm_pool_alloc_node() for caller-selected NUMA node drm/ttm: Add NUMA node aware restore helpers drm/ttm/pool: enforce per-node cache limits on free drm/xe/bo: Plumb a per-BO host NUMA node hint into tt populate drm/xe/bo: Preserve the host NUMA node across backup / restore drm/xe/uapi: Add GEM_CREATE NUMA node placement property drm/xe/uapi: Wire up GEM_CREATE NUMA node placement handler drivers/gpu/drm/ttm/ttm_pool.c | 104 +++++++++++++++++++--- drivers/gpu/drm/ttm/ttm_pool_internal.h | 4 + drivers/gpu/drm/ttm/ttm_tt.c | 20 ++++- drivers/gpu/drm/xe/tests/xe_bo.c | 12 +-- drivers/gpu/drm/xe/tests/xe_dma_buf.c | 4 +- drivers/gpu/drm/xe/tests/xe_migrate.c | 8 +- drivers/gpu/drm/xe/xe_bo.c | 109 +++++++++++++++++++----- drivers/gpu/drm/xe/xe_bo.h | 5 +- drivers/gpu/drm/xe/xe_bo_types.h | 7 ++ drivers/gpu/drm/xe/xe_dma_buf.c | 3 +- include/drm/ttm/ttm_pool.h | 2 + include/drm/ttm/ttm_tt.h | 2 + include/uapi/drm/xe_drm.h | 7 ++ 13 files changed, 238 insertions(+), 49 deletions(-) -- 2.43.0
