From: Alirad Malek <[email protected]> Memory demoted to a lower tier is assumed to be cold and most likely out of the CPU's last level cache. Additionally, in certain demotion targets (e.g. CXL devices with compressed memory) the bandwidth can be negatively impacted by the eviction patterns of the last level cache when standard memcpy is used. In supported architectures (currently X86_64), use the MIGRATE_ASYNC_NON_TEMPORAL_STORES flag in demotions to trigger the folio copy path using non-temporal stores.
Signed-off-by: Alirad Malek <[email protected]> Co-developed-by: Yiannis Nikolakopoulos <[email protected]> Signed-off-by: Yiannis Nikolakopoulos <[email protected]> --- mm/migrate.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/mm/migrate.c b/mm/migrate.c index ff6cf50e7b0b..c7c62700d541 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -862,7 +862,10 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst, if (folio_ref_count(src) != expected_count) return -EAGAIN; - rc = folio_mc_copy(dst, src); + if (mode == MIGRATE_ASYNC_NON_TEMPORAL_STORES) + rc = folio_mc_copy_nt(dst, src); + else + rc = folio_mc_copy(dst, src); if (unlikely(rc)) return rc; @@ -2081,6 +2084,11 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio, LIST_HEAD(split_folios); struct migrate_pages_stats stats; + if (IS_ENABLED(CONFIG_X86_64) && + IS_ENABLED(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) && + reason == MR_DEMOTION && mode == MIGRATE_ASYNC) + mode = MIGRATE_ASYNC_NON_TEMPORAL_STORES; + trace_mm_migrate_pages_start(mode, reason); memset(&stats, 0, sizeof(stats)); -- 2.43.0
