The common vmemmap population path cannot yet handle optimized Device DAX mappings on its own. It uses pfn_to_zone() to find the shared tail page, but Device DAX populates its vmemmap at runtime before the ZONE_DEVICE span is initialized.
Teach the common path to use device_zone() for runtime optimized vmemmap population while retaining pfn_to_zone() for early boot. This allows the same path to support both early boot mappings and Device DAX. The backing PFN supplied by the Device DAX-specific population path is no longer used, allowing the redundant lookup and population code to be removed later. Signed-off-by: Muchun Song <[email protected]> --- mm/sparse-vmemmap.c | 44 +++++++++++++++++++------------------------- 1 file changed, 19 insertions(+), 25 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 878d29a4e862..e83821768c12 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -208,18 +208,27 @@ static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, struct page *page; const unsigned int order = pfn_to_section_compound_order(pfn); - /* - * Device DAX still relies on vmemmap_populate_compound_pages() for - * head/first-tail allocation and tail-page reuse. - */ if (!vmemmap_optimizable_pfn(pfn)) return vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); - zone = pfn_to_zone(pfn, node); + /* + * At runtime (slab available), only ZONE_DEVICE pages trigger vmemmap + * optimization, so device_zone() suffices. Note that pfn_to_zone() + * cannot be used at runtime because the zone span is not set up now. + */ + zone = slab_is_available() ? device_zone(node) : pfn_to_zone(pfn, node); page = vmemmap_shared_tail_page(order, zone); if (!page) return NULL; + /* + * When a PTE entry is freed, a free_pages() call occurs. This get_page() + * pairs with put_page_testzero() on the freeing path. This can only occur + * when slab is available. + */ + if (slab_is_available()) + get_page(page); + return page_address(page); } @@ -231,27 +240,12 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in if (pte_none(ptep_get(pte))) { pte_t entry; + void *p = vmemmap_alloc_pte(pfn, node, altmap); - if (ptpfn == (unsigned long)-1) { - void *p = vmemmap_alloc_pte(pfn, node, altmap); - - if (!p) - return NULL; - ptpfn = PHYS_PFN(__pa(p)); - } else { - /* - * When a PTE/PMD entry is freed from the init_mm - * there's a free_pages() call to this page allocated - * above. Thus this get_page() is paired with the - * put_page_testzero() on the freeing path. - * This can only called by certain ZONE_DEVICE path, - * and through vmemmap_populate_compound_pages() when - * slab is available. - */ - if (slab_is_available()) - get_page(pfn_to_page(ptpfn)); - } - entry = pfn_pte(ptpfn, PAGE_KERNEL); + if (!p) + return NULL; + + entry = pfn_pte(PHYS_PFN(__pa(p)), PAGE_KERNEL); set_pte_at(&init_mm, addr, pte, entry); } else if (WARN_ON_ONCE(vmemmap_optimizable_pfn(pfn))) return NULL; -- 2.54.0
