On Mon, Aug 10, 2026 at 08:20:57PM +0800, Li Zhe wrote: > The x86 memcpy_nontemporal() helper maps to memcpy_flushcache(), and the > ZONE_DEVICE template-copy path uses it to copy one struct page at a > time. > > The relevant copy size is sizeof(struct page). On x86_64, the base struct > page layout is 64 bytes. Adding either the KMSAN metadata pointers or an > out-of-flags last_cpupid field can make it 80 bytes after alignment, and > enabling both can make it 96 bytes. > > memcpy_flushcache() currently only has inline fixed-size cases for 4, 8, > and 16 bytes. As a result, these constant-sized struct page copies fall > through to __memcpy_flushcache() even though the compiler knows the copy > size at the call site. > > Add fixed-size MOVNTI cases up to 96 bytes so the ZONE_DEVICE > template-copy path can keep these struct page copies in the inline > memcpy_flushcache() path. > > This matters for ZONE_DEVICE memmap initialization because the copy > happens once per initialized struct page. For a 100 GB fsdax namespace > with map=dev, this is about 25 million struct page copies during nd_pmem > binding or rebinding. > > Tested in a VM with a 100 GB fsdax namespace device configured with > map=dev and a 100 GB devdax namespace (align=2097152) on Intel Ice Lake > server. > > Test procedure: > Rebind the nd_pmem and dax_pmem drivers 30 times and collect the memmap > initialization time from the pr_debug() output of > memmap_init_zone_device(). > > With memcpy_nontemporal() used by the ZONE_DEVICE template-copy path: > Average of rebinds for nd_pmem driver: 150.83 ms > Average of rebinds for dax_pmem driver: 153.55 ms > > With this x86 fixed-size fastpath patch applied: > Average of rebinds for nd_pmem driver: 96.79 ms > Average of rebinds for dax_pmem driver: 119.04 ms > > This further reduces the average memmap initialization time measured > during rebind by about 35.8% for nd_pmem and 22.5% for dax_pmem. > > Suggested-by: Borislav Petkov <[email protected]> > Signed-off-by: Li Zhe <[email protected]> > --- > arch/x86/include/asm/string_64.h | 71 +++++++++++++++++++++++++------- > 1 file changed, 56 insertions(+), 15 deletions(-)
Acked-by: Borislav Petkov (AMD) <[email protected]> -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
