In most memory tiering scenarios, the memory to be demoted is expected to be cold and most likely out of the node's last-level cache (as well as target pages in the target node). Using non-temporal stores instead of a standard memcpy path can reduce the cache pollution in the local node and the bandwidth overhead to the target node. Furthermore, for certain types of CXL devices that support in-line memory compression, the last-level cache eviction patterns can negatively affect the bandwidth of the device. Non-temporal stores can mitigate this.
This patch-set introduces a new migrate_mode flag for using non-temporal stores that is used only in the demotion path. Patch 1 adds some helpers in x86 and mm to bring non-temporal stores support to a respective folio_copy function. Patch 2 adds the new flag and necessary changes for compatibility with the existing behavior. Patch 3 uses the new flag for demotions and guards this by a Kconfig option. Experimental data: in a CXL system with 1 memory expander, a microbenchmark that allocates N=64 GB memory in the local node and then triggers demotion using memory.reclaim, shows a practically complete elimination of read traffic on the device, i.e. write traffic is N GB with and without the patch, while read traffic drops from N to almost 0 with the patch. Opens: 1. There is some "duplication" in the x86 tree and a bit in mm. Can we do something better there? As it is now in copy_mc_to_kernel_nt we duplicate the machine check functionality, which if available will override the non-temporal. We were not sure how to prioritize these two and what's the best approach here. Can we completely skip the machine checked for this path? 2. We've hidden the use of the new flag behind a Kconfig option. Is this the right way to add this? 3. We have not carefully considered how this should be structured so that it is easily adopted in other architecture trees (e.g. aarch64). Arm support is currently out of our scope but any input is appreciated. Signed-off-by: Yiannis Nikolakopoulos <[email protected]> --- Alirad Malek (3): mm, x86: support copying a folio using non-temporal stores mm: new migrate_mode flag for async using non-temporal stores mm: use non-temporal stores for demotion arch/x86/include/asm/uaccess.h | 4 ++++ arch/x86/lib/copy_mc.c | 26 ++++++++++++++++++++++++++ fs/nfs/write.c | 2 +- include/linux/highmem.h | 32 ++++++++++++++++++++++++++++++++ include/linux/migrate_mode.h | 9 +++++++++ include/linux/mm.h | 1 + include/trace/events/migrate.h | 1 + mm/Kconfig | 8 ++++++++ mm/compaction.c | 18 +++++++++--------- mm/migrate.c | 21 ++++++++++++++------- mm/util.c | 17 +++++++++++++++++ 11 files changed, 122 insertions(+), 17 deletions(-) --- base-commit: 1f318b96cc84d7c2ab792fcc0bfd42a7ca890681 change-id: 20260526-rfc-nt-demote-0fafadfdd006 Best regards, -- Yiannis Nikolakopoulos <[email protected]>
