When post_alloc_hook() needs to zero a page for an explicit __GFP_ZERO allocation for a user page (user_addr is set), use folio_zero_user() instead of kernel_init_pages(). This zeros near the faulting address last, keeping those cachelines hot for the impending user access.
folio_zero_user() is only used for explicit __GFP_ZERO, not for init_on_alloc. On architectures with virtually-indexed caches (e.g., ARM), clear_user_highpage() performs per-line cache operations; using it for init_on_alloc would add overhead that kernel_init_pages() avoids (the page fault path flushes the cache at PTE installation time regardless). No functional change yet: current callers do not pass __GFP_ZERO for user pages (they zero at the callsite instead). Subsequent patches will convert them. Signed-off-by: Michael S. Tsirkin <[email protected]> Assisted-by: Claude:claude-opus-4-6 --- mm/page_alloc.c | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3108ac9061ce..ad0655387e9d 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1864,9 +1864,38 @@ inline void post_alloc_hook(struct page *page, unsigned int order, for (i = 0; i != 1 << order; ++i) page_kasan_tag_reset(page + i); } - /* If memory is still not initialized, initialize it now. */ - if (init) - kernel_init_pages(page, 1 << order); + /* + * On architectures with cache aliasing, pages zeroed via the + * kernel direct map (e.g. init_on_free) must be re-zeroed + * through a user-congruent mapping. Host-zeroed pages + * (zeroed flag) don't need this: physical RAM is clean. + */ + if (!init && (gfp_flags & __GFP_ZERO) && + user_addr != USER_ADDR_NONE && + user_alloc_needs_zeroing()) + init = true; + /* + * If memory is still not initialized, initialize it now. + * When __GFP_ZERO was explicitly requested and user_addr is set, + * use folio_zero_user() which zeros near the faulting address + * last, keeping those cachelines hot. For init_on_alloc, use + * kernel_init_pages() to avoid unnecessary cache flush overhead + * on architectures with virtually-indexed caches. + */ + if (init) { + if ((gfp_flags & __GFP_ZERO) && user_addr != USER_ADDR_NONE) { + /* + * folio_zero_user relies on folio_nr_pages which + * requires __GFP_COMP for order > 0. All user folio + * allocations set __GFP_COMP via __folio_alloc. + * user_addr != USER_ADDR_NONE implies sleepable + * context (user page fault). + */ + VM_WARN_ON_ONCE(order && !(gfp_flags & __GFP_COMP)); + folio_zero_user(page_folio(page), user_addr); + } else + kernel_init_pages(page, 1 << order); + } set_page_owner(page, order, gfp_flags); page_table_check_alloc(page, order); -- MST
