In most memory tiering scenarios, the memory to be demoted is expected to be cold and most likely out of the node's last-level cache (as well as target pages in the target node). Using non-temporal stores instead of a standard memcpy path can reduce the cache pollution in the local node and the bandwidth overhead to the target node. Furthermore, for certain types of CXL devices that support in-line memory compression, the last-level cache eviction patterns can negatively affect the bandwidth of the device. Non-temporal stores can mitigate this.
This patch-set introduces a new migrate_mode flag for using non-temporal stores that is used only in the demotion path. Patch 1 adds some helpers in x86 and mm to bring non-temporal stores support to a respective folio_copy function. Patch 2 adds the new flag and necessary changes for compatibility with the existing behavior. Patch 3 uses the new flag for demotions. Experimental data: in a CXL system with 1 memory expander, a microbenchmark that allocates N=64 GB memory in the local node and then triggers demotion using memory.reclaim, shows a practically complete elimination of read traffic on the device, i.e. write traffic is N GB with and without the patch, while read traffic drops from N to almost 0 with the patch. Opens: 1. There is some "duplication" in the x86 tree and a bit in mm. Can we do something better there? As it is now in copy_mc_to_kernel_nt we duplicate the machine check functionality, which if available will override the non-temporal. We were not sure how to prioritize these two and what's the best approach here. Can we completely skip the machine checked for this path? Huan Nguyen has some ideas here that we will align for the next version. 2. I am not sure how this should be structured so that it is easily adopted in other architecture trees (e.g. aarch64). We rely on memcpy_flushcache for x86_64 but this does not use non temporal stores in ARM. ARM support is currently out of our scope but any input is appreciated. Signed-off-by: Yiannis Nikolakopoulos <[email protected]> --- Changes in v2: - Remove the KConfig option that was guarding this feature (patch 3) and use it always in x86_64 and memcpy_flushcache (non-temporal stores should be there). - Link to v1: https://patch.msgid.link/[email protected] To: Thomas Gleixner <[email protected]> To: Ingo Molnar <[email protected]> To: Borislav Petkov <[email protected]> To: Dave Hansen <[email protected]> To: [email protected] To: "H. Peter Anvin" <[email protected]> To: Andrew Morton <[email protected]> To: David Hildenbrand <[email protected]> To: Lorenzo Stoakes <[email protected]> To: "Liam R. Howlett" <[email protected]> To: Vlastimil Babka <[email protected]> To: Mike Rapoport <[email protected]> To: Suren Baghdasaryan <[email protected]> To: Michal Hocko <[email protected]> To: Trond Myklebust <[email protected]> To: Anna Schumaker <[email protected]> To: Zi Yan <[email protected]> To: Matthew Brost <[email protected]> To: Joshua Hahn <[email protected]> To: Rakie Kim <[email protected]> To: Byungchul Park <[email protected]> To: Gregory Price <[email protected]> To: Ying Huang <[email protected]> To: Alistair Popple <[email protected]> To: Steven Rostedt <[email protected]> To: Masami Hiramatsu <[email protected]> To: Mathieu Desnoyers <[email protected]> To: Brendan Jackman <[email protected]> To: Johannes Weiner <[email protected]> Cc: [email protected] Cc: [email protected] Cc: [email protected] Cc: [email protected] --- Alirad Malek (3): mm, x86: support copying a folio using non-temporal stores mm: new migrate_mode flag for async using non-temporal stores mm: use non-temporal stores for demotion arch/x86/include/asm/uaccess.h | 4 ++++ arch/x86/lib/copy_mc.c | 26 ++++++++++++++++++++++++++ fs/nfs/write.c | 2 +- include/linux/highmem.h | 32 ++++++++++++++++++++++++++++++++ include/linux/migrate_mode.h | 9 +++++++++ include/linux/mm.h | 1 + include/trace/events/migrate.h | 1 + mm/compaction.c | 18 +++++++++--------- mm/migrate.c | 22 +++++++++++++++------- mm/util.c | 17 +++++++++++++++++ 10 files changed, 115 insertions(+), 17 deletions(-) --- base-commit: 1f318b96cc84d7c2ab792fcc0bfd42a7ca890681 change-id: 20260526-rfc-nt-demote-0fafadfdd006 Best regards, -- Yiannis Nikolakopoulos <[email protected]>
