On Wed, Oct 22, 2025 at 12:37:09PM -0600, Nico Pache wrote: > Pass an order and offset to collapse_huge_page to support collapsing anon > memory to arbitrary orders within a PMD. order indicates what mTHP size we > are attempting to collapse to, and offset indicates were in the PMD to > start the collapse attempt. > > For non-PMD collapse we must leave the anon VMA write locked until after > we collapse the mTHP-- in the PMD case all the pages are isolated, but in
NIT but is this -- a typo? > the mTHP case this is not true, and we must keep the lock to prevent > changes to the VMA from occurring. > > Signed-off-by: Nico Pache <[email protected]> > --- > mm/khugepaged.c | 108 ++++++++++++++++++++++++++++++------------------ > 1 file changed, 67 insertions(+), 41 deletions(-) > > diff --git a/mm/khugepaged.c b/mm/khugepaged.c > index 286c3a7afdee..75e7ebdccc36 100644 > --- a/mm/khugepaged.c > +++ b/mm/khugepaged.c > @@ -1142,43 +1142,50 @@ static int alloc_charge_folio(struct folio **foliop, > struct mm_struct *mm, > return SCAN_SUCCEED; > } > > -static int collapse_huge_page(struct mm_struct *mm, unsigned long address, > - int referenced, int unmapped, > - struct collapse_control *cc) > +static int collapse_huge_page(struct mm_struct *mm, unsigned long > pmd_address, Presumably pmd_address is the PMD-aligned address? > + int referenced, int unmapped, struct collapse_control *cc, > + bool *mmap_locked, unsigned int order, unsigned long offset) It'd be nice to pass through a helper struct at this point having so many params but perhaps we can deal with that in a follow up series. If PMD address is the PMD-aligned address, and mthp_address = pmd_address + offset * PAGE_SIZE, couldn't we just pass in the mthp address and get the PMD address by aligning down to PMD size and reduce the number of args by 1? > { > LIST_HEAD(compound_pagelist); > pmd_t *pmd, _pmd; > - pte_t *pte; > + pte_t *pte = NULL, mthp_pte; mthp_pte is only used in a single if () branch and can be declared there AFAICT? > pgtable_t pgtable; > struct folio *folio; > spinlock_t *pmd_ptl, *pte_ptl; > int result = SCAN_FAIL; > struct vm_area_struct *vma; > struct mmu_notifier_range range; > + bool anon_vma_locked = false; > + const unsigned long nr_pages = 1UL << order; > + unsigned long mthp_address = pmd_address + offset * PAGE_SIZE; Do we ever update this? If not we can const-ify. > > - VM_BUG_ON(address & ~HPAGE_PMD_MASK); > + VM_BUG_ON(pmd_address & ~HPAGE_PMD_MASK); NIT: Be nice to convert this to a VM_WARN_ON_ONCE(), as VM_BUG_ON() is not right here. > > /* > * Before allocating the hugepage, release the mmap_lock read lock. > * The allocation can take potentially a long time if it involves > * sync compaction, and we do not need to hold the mmap_lock during > * that. We will recheck the vma after taking it again in write mode. > + * If collapsing mTHPs we may have already released the read_lock. > */ > - mmap_read_unlock(mm); > + if (*mmap_locked) { > + mmap_read_unlock(mm); > + *mmap_locked = false; > + } > > - result = alloc_charge_folio(&folio, mm, cc, HPAGE_PMD_ORDER); > + result = alloc_charge_folio(&folio, mm, cc, order); > if (result != SCAN_SUCCEED) > goto out_nolock; > > mmap_read_lock(mm); > - result = hugepage_vma_revalidate(mm, address, true, &vma, cc, > - HPAGE_PMD_ORDER); > + *mmap_locked = true; > + result = hugepage_vma_revalidate(mm, pmd_address, true, &vma, cc, > order); > if (result != SCAN_SUCCEED) { > mmap_read_unlock(mm); I don't really love the semantics of 'sometimes we set *mmap_locked false when we unlock, sometimes we rely on out_nolock doing it'. Let's just set it false when we unlock and VM_WARN_ON_ONCE(*mmap_locked) in out_nolock. > goto out_nolock; > } > > - result = find_pmd_or_thp_or_none(mm, address, &pmd); > + result = find_pmd_or_thp_or_none(mm, pmd_address, &pmd); > if (result != SCAN_SUCCEED) { > mmap_read_unlock(mm); > goto out_nolock; > @@ -1190,13 +1197,14 @@ static int collapse_huge_page(struct mm_struct *mm, > unsigned long address, > * released when it fails. So we jump out_nolock directly in > * that case. Continuing to collapse causes inconsistency. > */ > - result = __collapse_huge_page_swapin(mm, vma, address, pmd, > - referenced, > HPAGE_PMD_ORDER); > + result = __collapse_huge_page_swapin(mm, vma, mthp_address, pmd, > + referenced, order); > if (result != SCAN_SUCCEED) > goto out_nolock; > } > > mmap_read_unlock(mm); > + *mmap_locked = false; > /* > * Prevent all access to pagetables with the exception of > * gup_fast later handled by the ptep_clear_flush and the VM > @@ -1206,20 +1214,20 @@ static int collapse_huge_page(struct mm_struct *mm, > unsigned long address, > * mmap_lock. > */ > mmap_write_lock(mm); > - result = hugepage_vma_revalidate(mm, address, true, &vma, cc, > - HPAGE_PMD_ORDER); > + result = hugepage_vma_revalidate(mm, pmd_address, true, &vma, cc, > order); > if (result != SCAN_SUCCEED) > goto out_up_write; > /* check if the pmd is still valid */ > vma_start_write(vma); > - result = check_pmd_still_valid(mm, address, pmd); > + result = check_pmd_still_valid(mm, pmd_address, pmd); > if (result != SCAN_SUCCEED) > goto out_up_write; > > anon_vma_lock_write(vma->anon_vma); > + anon_vma_locked = true; > > - mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, address, > - address + HPAGE_PMD_SIZE); > + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, mthp_address, > + mthp_address + (PAGE_SIZE << order)); > mmu_notifier_invalidate_range_start(&range); > > pmd_ptl = pmd_lock(mm, pmd); /* probably unnecessary */ > @@ -1231,24 +1239,21 @@ static int collapse_huge_page(struct mm_struct *mm, > unsigned long address, > * Parallel GUP-fast is fine since GUP-fast will back off when > * it detects PMD is changed. > */ > - _pmd = pmdp_collapse_flush(vma, address, pmd); > + _pmd = pmdp_collapse_flush(vma, pmd_address, pmd); Not your fault but so hate this _p** convention. One for a follow up I suppose. > spin_unlock(pmd_ptl); > mmu_notifier_invalidate_range_end(&range); > tlb_remove_table_sync_one(); > > - pte = pte_offset_map_lock(mm, &_pmd, address, &pte_ptl); > + pte = pte_offset_map_lock(mm, &_pmd, mthp_address, &pte_ptl); > if (pte) { > - result = __collapse_huge_page_isolate(vma, address, pte, cc, > - HPAGE_PMD_ORDER, > - &compound_pagelist); > + result = __collapse_huge_page_isolate(vma, mthp_address, pte, > cc, > + order, > &compound_pagelist); > spin_unlock(pte_ptl); > } else { > result = SCAN_PMD_NULL; > } > > if (unlikely(result != SCAN_SUCCEED)) { > - if (pte) > - pte_unmap(pte); OK I guess we drop this because it's handled in out_up_write. I assume no issue keeping PTE mapped here? > spin_lock(pmd_ptl); > BUG_ON(!pmd_none(*pmd)); > /* > @@ -1258,21 +1263,21 @@ static int collapse_huge_page(struct mm_struct *mm, > unsigned long address, > */ > pmd_populate(mm, pmd, pmd_pgtable(_pmd)); > spin_unlock(pmd_ptl); > - anon_vma_unlock_write(vma->anon_vma); > goto out_up_write; > } > > /* > - * All pages are isolated and locked so anon_vma rmap > - * can't run anymore. > + * For PMD collapse all pages are isolated and locked so anon_vma > + * rmap can't run anymore. For mTHP collapse we must hold the lock > */ > - anon_vma_unlock_write(vma->anon_vma); > + if (order == HPAGE_PMD_ORDER) { > + anon_vma_unlock_write(vma->anon_vma); > + anon_vma_locked = false; > + } > > result = __collapse_huge_page_copy(pte, folio, pmd, _pmd, > - vma, address, pte_ptl, > - HPAGE_PMD_ORDER, > - &compound_pagelist); > - pte_unmap(pte); > + vma, mthp_address, pte_ptl, > + order, &compound_pagelist); Looking through __collapse_huge_page_copy() there doesn't seem to be any issue with holding anon lock here. > if (unlikely(result != SCAN_SUCCEED)) > goto out_up_write; > > @@ -1282,20 +1287,42 @@ static int collapse_huge_page(struct mm_struct *mm, > unsigned long address, > * write. > */ > __folio_mark_uptodate(folio); > - pgtable = pmd_pgtable(_pmd); > + if (order == HPAGE_PMD_ORDER) { > + pgtable = pmd_pgtable(_pmd); > > - spin_lock(pmd_ptl); > - BUG_ON(!pmd_none(*pmd)); > - pgtable_trans_huge_deposit(mm, pmd, pgtable); > - map_anon_folio_pmd_nopf(folio, pmd, vma, address); > - spin_unlock(pmd_ptl); > + spin_lock(pmd_ptl); > + WARN_ON_ONCE(!pmd_none(*pmd)); > + pgtable_trans_huge_deposit(mm, pmd, pgtable); > + map_anon_folio_pmd_nopf(folio, pmd, vma, pmd_address); > + spin_unlock(pmd_ptl); > + } else { /* mTHP collapse */ As per above, let's just declare mthp_pte here. > + mthp_pte = mk_pte(folio_page(folio, 0), vma->vm_page_prot); Hm, so we make a PTE that references the first page of the folio? I guess the folio will be an mTHP folio so we're just creating essentially a > + mthp_pte = maybe_mkwrite(pte_mkdirty(mthp_pte), vma); In set_pte_range() we have a whole host of other checks like dirty, uffd_wp, etc. I wonder if we need to consider those? > + > + spin_lock(pmd_ptl); We're duplicating this in both branches, why not do outside if/else? > + WARN_ON_ONCE(!pmd_none(*pmd)); Hmm so the PMD entry will still always be empty on mTHP collapse? Surely we could be collapsing more than one mTHP into an existing PTE table no? I may be missing something here/confused :) > + folio_ref_add(folio, nr_pages - 1); If we're setting the refcount here, where is the ref count being set in the PMD path? > + folio_add_new_anon_rmap(folio, vma, mthp_address, > RMAP_EXCLUSIVE); > + folio_add_lru_vma(folio, vma); > + set_ptes(vma->vm_mm, mthp_address, pte, mthp_pte, nr_pages); > + update_mmu_cache_range(NULL, vma, mthp_address, pte, nr_pages); Prior to this change the only user of this are functions in memory.c, I do wonder if this is the wrong abstraction here. But maybe that's _yet another_ thing for a follow up (the THP code is a mess). > + > + smp_wmb(); /* make PTEs visible before PMD. See pmd_install() */ Feels like we could avoid open-coding this by just using pmd_install()? Also are we therefore missing a mm_inc_nr_ptes() invocation here, or do we update mm->pgtables_bytes elsewhere? > + pmd_populate(mm, pmd, pmd_pgtable(_pmd)); Why are we referencing pmd in PMD branch and _pmd here? > + spin_unlock(pmd_ptl); The PMD case does this stuff in map_anon_pmd_nopf(), could we add one for mTHP? This function is already horribly overwrought (not your fault) so I'd like to avoid adding open-coded blocks as much as possible. > + } > > folio = NULL; > > result = SCAN_SUCCEED; > out_up_write: > + if (anon_vma_locked) > + anon_vma_unlock_write(vma->anon_vma); > + if (pte) > + pte_unmap(pte); > mmap_write_unlock(mm); > out_nolock: > + *mmap_locked = false; See above comment about setting this prior to jumping to out_nolock. > if (folio) > folio_put(folio); > trace_mm_collapse_huge_page(mm, result == SCAN_SUCCEED, result); > @@ -1463,9 +1490,8 @@ static int collapse_scan_pmd(struct mm_struct *mm, > pte_unmap_unlock(pte, ptl); > if (result == SCAN_SUCCEED) { > result = collapse_huge_page(mm, start_addr, referenced, > - unmapped, cc); > - /* collapse_huge_page will return with the mmap_lock released */ > - *mmap_locked = false; > + unmapped, cc, mmap_locked, > + HPAGE_PMD_ORDER, 0); > } > out: > trace_mm_khugepaged_scan_pmd(mm, folio, referenced, > -- > 2.51.0 >
