From: Alirad Malek <[email protected]> Memory demoted to a lower tier is assumed to be cold and most likely out of the CPU's last level cache. Additionally, in certain demotion targets (e.g. CXL devices with compressed memory) the bandwidth can be negatively impacted by the eviction patterns of the last level cache when standard memcpy is used. When the feature is enabled, use the MIGRATE_ASYNC_NON_TEMPORAL_STORES flag in demotions to trigger the folio copy path using non-temporal stores.
Signed-off-by: Alirad Malek <[email protected]> Co-developed-by: Yiannis Nikolakopoulos <[email protected]> Signed-off-by: Yiannis Nikolakopoulos <[email protected]> --- mm/Kconfig | 8 ++++++++ mm/migrate.c | 9 ++++++++- 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/mm/Kconfig b/mm/Kconfig index ebd8ea353687..4b7a75b57f6e 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -645,6 +645,14 @@ config MIGRATION pages as migration can relocate pages to satisfy a huge page allocation instead of reclaiming. +config DEMOTION_WITH_NON_TEMPORAL_STORES + bool "Use non-temporal stores for demotion" + default n + depends on MIGRATION + help + Enable non-temporal stores when migrating pages due to demotion. + If disabled, demotion uses regular migration copy paths. + config DEVICE_MIGRATION def_bool MIGRATION && ZONE_DEVICE diff --git a/mm/migrate.c b/mm/migrate.c index ff6cf50e7b0b..368d40dc8772 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -862,7 +862,10 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst, if (folio_ref_count(src) != expected_count) return -EAGAIN; - rc = folio_mc_copy(dst, src); + if (mode == MIGRATE_ASYNC_NON_TEMPORAL_STORES) + rc = folio_mc_copy_nt(dst, src); + else + rc = folio_mc_copy(dst, src); if (unlikely(rc)) return rc; @@ -2081,6 +2084,10 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio, LIST_HEAD(split_folios); struct migrate_pages_stats stats; + if (IS_ENABLED(CONFIG_DEMOTION_WITH_NON_TEMPORAL_STORES) && + reason == MR_DEMOTION && mode == MIGRATE_ASYNC) + mode = MIGRATE_ASYNC_NON_TEMPORAL_STORES; + trace_mm_migrate_pages_start(mode, reason); memset(&stats, 0, sizeof(stats)); -- 2.43.0
