On Sun, Aug 16, 2026 at 11:45:20PM +0100, Kiryl Shutsemau wrote:
>From: "Kiryl Shutsemau (Meta)" <[email protected]>
>
>Fill in the scan. Walk the range and set a bit in cc->eligible_ptes for
>every PTE a collapse may take as a source: present, anonymous, not
>uffd-armed, on the LRU and unlocked. The bit is set last, so a PTE that
>failed anything leaves it clear.
>
>The walk takes no page table lock. What it produces is advice: the
>freeze settles every question the scan asks, by re-reading the table
>under the lock and freezing each source to the count it expects. A racy
>read can only cost a candidate the freeze then refuses, or miss one the
>next pass finds. What it buys is that a fault in the range does not wait
>for a walk of the whole table.
>
>pte_offset_map() holds rcu_read_lock() until pte_unmap(), which keeps the
>table from being freed underneath the walk. mmap_lock keeps the VMA
>attached, without which free_pgtables() could free it without waiting for
>RCU at all.
>
>The verdict is two-sided, which is the point:
>
> - A PTE that disqualifies only itself leaves the bitmap clear there and
> drops the PMD order, since a PMD candidate needs the whole table.
> Selection still gets the smaller windows that avoid it.
> - What refuses the table as a unit -- a limit the whole range exceeds,
> or sources spread across nodes too distant for one folio to serve --
> leaves no order eligible at all.
>
>Limits on swapped-out and shared PTEs are stated per PMD and scaled to
>what was actually scanned, so a partial table is held to the same density
>as a whole one.
>
>A folio whose reference count its mappings do not account for -- a GUP
>pin, say -- is left to the freeze rather than refused here.
>folio_expected_ref_count() wants a folio that cannot change order while
>it is read. This walk holds no page table lock and no folio lock, so a
>folio splitting underneath it would have its count read for the wrong
>size. A reference of its own would not help: that stops a folio being
>freed, not split.
>
>Whether a range has to look used at all is the caller's policy, so only a
>caller that asks gathers the young/referenced evidence.
>
>Assisted-by: Claude-Code:claude-opus-5
>Signed-off-by: Kiryl Shutsemau (Meta) <[email protected]>
>---
> mm/collapse.c | 239 +++++++++++++++++++++++++++++++++++++++++++++++-
> mm/collapse.h | 7 ++
> mm/khugepaged.c | 8 +-
> 3 files changed, 249 insertions(+), 5 deletions(-)
>
>diff --git a/mm/collapse.c b/mm/collapse.c
>index 0e6c3c68b44c..66931ef6a6d0 100644
>--- a/mm/collapse.c
>+++ b/mm/collapse.c
>@@ -86,6 +86,20 @@
> * replaces, and is switched over once both halves are complete.
> */
>
>+/*
>+ * Is @count past a limit stated per PMD, when only part of a table was
>scanned?
>+ * Scale the comparison to the table so a partial scan is held to the same
>+ * density as a whole one.
>+ */
>+static bool collapse_exceeds_limit(unsigned int count, unsigned int
>max_per_pmd,
>+ unsigned long start, unsigned long end)
>+{
>+ const unsigned long nr_scanned = (end - start) >> PAGE_SHIFT;
>+
>+ return (unsigned long)count * HPAGE_PMD_NR >
>+ (unsigned long)max_per_pmd * nr_scanned;
>+}
>+
> /*
> * Scan the PTEs between @start and @end and record what a collapse could
> use: a
> * bit in cc->eligible_ptes for every PTE that may be a source. Returns
>@@ -97,7 +111,230 @@ static enum scan_result collapse_scan_table(struct
>vm_area_struct *vma,
> unsigned long end,
> struct collapse_control *cc)
> {
>- return SCAN_SUCCEED;
>+ const unsigned long pmd_addr = start & HPAGE_PMD_MASK;
>+ unsigned int max_ptes_none, max_ptes_swap, max_ptes_shared;
>+ int none_or_zero = 0, shared = 0, referenced = 0, unmapped = 0;
>+ enum scan_result result, pmd_result = SCAN_SUCCEED;
>+ unsigned int first_offset;
>+ unsigned long addr;
>+ pte_t *pte;
>+ int i;
>+
>+ max_ptes_none = collapse_max_ptes_none(cc, vma, HPAGE_PMD_ORDER);
>+ max_ptes_swap = collapse_max_ptes_swap(cc, HPAGE_PMD_ORDER);
>+ max_ptes_shared = collapse_max_ptes_shared(cc, HPAGE_PMD_ORDER);
>+
>+ /*
>+ * No page table lock: what this builds is advice, and the freeze
>settles
>+ * every question it asks by re-reading the table under the lock and
>+ * freezing each source to the count it expects. A racy read can only
>+ * cost a candidate that the freeze then refuses, or miss one that the
>+ * next pass finds. What it buys is that a fault in this range does not
>+ * wait for a scan of the whole table.
>+ *
>+ * pte_offset_map() holds rcu_read_lock() until pte_unmap(), which is
>+ * what keeps the table itself from being freed underneath the walk;
>+ * mmap_lock keeps the VMA attached, without which free_pgtables() could
>+ * free it without waiting for RCU at all. Nothing below here sleeps.
>+ */
>+ pte = pte_offset_map(pmd, start);
>+ if (!pte) {
>+ cc->progress++;
>+ result = SCAN_NO_PTE_TABLE;
>+ goto out_no_table;
>+ }
>+
>+ /*
>+ * The bitmap and the selection offsets stay relative to the table:
>+ * natural-alignment math needs the table-absolute position, not the
>+ * position within an arbitrarily placed VMA.
>+ */
>+ first_offset = (start - pmd_addr) >> PAGE_SHIFT;
>+ for (i = first_offset, addr = start; addr < end;
>+ i++, addr += PAGE_SIZE) {
>+ pte_t pteval = ptep_get(pte + (i - first_offset));
Hmm, ptep_get() does not look right for a lockless scan ...
On arm64, a contiguous PTE sends ptep_get() to contpte_ptep_get():
static inline pte_t ptep_get(pte_t *ptep)
{
...
if (likely(!pte_valid_cont(pte)))
return pte;
return contpte_ptep_get(ptep, pte);
}
contpte_ptep_get() explicitly assumes PTL is held and therefore has no
consistency retry:
pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte)
{
/*
* Gather access/dirty bits, which may be populated in any of the ptes
* of the contig range. We are guaranteed to be holding the PTL, so any
* contiguous range cannot be unfolded or otherwise modified under our
* feet.
*/
...
}
The lockless accessor uses the matching implementation:
static inline pte_t ptep_get_lockless(pte_t *ptep)
{
...
if (likely(!pte_valid_cont(pte)))
return pte;
return contpte_ptep_get_lockless(ptep);
}
pte_t contpte_ptep_get_lockless(pte_t *orig_ptep)
{
/*
* The ptep_get_lockless() API requires us to read and return *orig_ptep
* so that it is self-consistent, without the PTL held, so we may be
* racing with other threads modifying the pte. Usually a READ_ONCE()
...
* and we can't read all of those neighbouring ptes atomically, so any
* contiguous range may be unfolded/modified/refolded under our feet.
* Therefore we ensure we read a _consistent_ contpte range by checking
* that all ptes in the range are valid and have CONT_PTE set, that all
* pfns are contiguous and that all pgprots are the same (ignoring
* access/dirty). If we find a pte that is not consistent, then we must
* be racing with an update so start again. If the target pte does not
...
*/
...
retry:
orig_pte = __ptep_get(orig_ptep);
if (!pte_valid_cont(orig_pte))
return orig_pte;
...
for (i = 0; i < CONT_PTES; i++, ptep++, pfn++) {
pte = __ptep_get(ptep);
if (!contpte_is_consistent(pte, pfn, orig_prot))
goto retry;
...
}
The later freeze can reject a stale candidate, but the earlier PTE read
is still lockless. Should the read use ptep_get_lockless() so arm64 can
retry if it finds an inconsistent PTE in the contpte range?
Cheers, Lance
>+ struct folio *folio;
>+ struct page *page;
>+ int node;
>+
>+ cc->progress++;
>+
>+ if (pte_none_or_zero(pteval)) {
>+ if (++none_or_zero > max_ptes_none &&
>+ pmd_result == SCAN_SUCCEED) {
>+ pmd_result = SCAN_EXCEED_NONE_PTE;
>+ count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
>+ count_mthp_stat(HPAGE_PMD_ORDER,
>+ MTHP_STAT_COLLAPSE_EXCEED_NONE);
>+ }
>+ continue;
>+ }
>+ if (!pte_present(pteval)) {
>+ unmapped++;
>+ if (collapse_exceeds_limit(unmapped, max_ptes_swap,
>+ start, end)) {
>+ result = SCAN_EXCEED_SWAP_PTE;
>+ count_vm_event(THP_SCAN_EXCEED_SWAP_PTE);
>+ count_mthp_stat(HPAGE_PMD_ORDER,
>+ MTHP_STAT_COLLAPSE_EXCEED_SWAP);
>+ goto out_table_refused;
>+ }
>+ /* Swap entries armed with uffd-wp are refused too */
>+ if (pte_swp_uffd_any(pteval) &&
>+ pmd_result == SCAN_SUCCEED)
>+ pmd_result = SCAN_PTE_UFFD;
>+ continue;
>+ }
>+ if (pte_uffd(pteval)) {
>+ /*
>+ * The huge PMD could be marked write protected when any
>+ * of the small ones is, but that could deliver
>+ * userfaults outside the registered range. Keep it
>+ * simple and refuse the PTE.
>+ */
>+ if (pmd_result == SCAN_SUCCEED)
>+ pmd_result = SCAN_PTE_UFFD;
>+ continue;
>+ }
>+
>+ page = vm_normal_page(vma, addr, pteval);
>+ if (unlikely(!page) || unlikely(is_zone_device_page(page))) {
>+ if (pmd_result == SCAN_SUCCEED)
>+ pmd_result = SCAN_PAGE_NULL;
>+ continue;
>+ }
>+ folio = page_folio(page);
>+
>+ /*
>+ * A VM_DROPPABLE VMA keeps the lazyfree property across the
>+ * collapse, so there is nothing to preserve by skipping.
>+ */
>+ if (cc->policy.skip_lazyfree &&
>+ !(vma->vm_flags & VM_DROPPABLE) &&
>+ folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
>+ if (pmd_result == SCAN_SUCCEED)
>+ pmd_result = SCAN_PAGE_LAZYFREE;
>+ continue;
>+ }
>+
>+ if (!folio_test_anon(folio)) {
>+ if (pmd_result == SCAN_SUCCEED)
>+ pmd_result = SCAN_PAGE_ANON;
>+ continue;
>+ }
>+
>+ /*
>+ * A page counts as shared if any part of its folio is, which
>+ * bounds the cost of CoW-breaking rather than the count of it:
>+ * collapse_faultin() unshares on !PageAnonExclusive(), a
>broader
>+ * test -- a page whose fork co-mapper has exited is
>+ * single-mapped, so not counted here, yet stays non-exclusive
>+ * until a write reuses it. Those are the cheap ones, reused in
>+ * place. A page that has to be copied is one this test
>catches,
>+ * so the limit does bound the copying it is there to bound.
>+ */
>+ if (folio_maybe_mapped_shared(folio)) {
>+ shared++;
>+ if (collapse_exceeds_limit(shared, max_ptes_shared,
>+ start, end)) {
>+ result = SCAN_EXCEED_SHARED_PTE;
>+ count_vm_event(THP_SCAN_EXCEED_SHARED_PTE);
>+ count_mthp_stat(HPAGE_PMD_ORDER,
>+
>MTHP_STAT_COLLAPSE_EXCEED_SHARED);
>+ goto out_table_refused;
>+ }
>+ }
>+
>+ /*
>+ * Which node the sources are on decides where the destination
>is
>+ * allocated: the one with the most of them wins.
>+ */
>+ node = folio_nid(folio);
>+ if (collapse_scan_abort(node, cc)) {
>+ result = SCAN_SCAN_ABORT;
>+ goto out_table_refused;
>+ }
>+ cc->node_load[node]++;
>+
>+ /*
>+ * Usually a folio somebody else is already isolating, whose
>+ * reference the freeze would refuse anyway. Not exact: one
>+ * still on a per-CPU add batch reads the same, and the freeze
>+ * drains those before it starts.
>+ */
>+ if (!folio_test_lru(folio)) {
>+ if (pmd_result == SCAN_SUCCEED)
>+ pmd_result = SCAN_PAGE_LRU;
>+ continue;
>+ }
>+ if (folio_test_locked(folio)) {
>+ if (pmd_result == SCAN_SUCCEED)
>+ pmd_result = SCAN_PAGE_LOCK;
>+ continue;
>+ }
>+
>+ /*
>+ * A folio whose reference count its mappings do not account for
>+ * -- a GUP pin, say -- is refused by the freeze, not here.
>+ * folio_expected_ref_count() wants a folio that cannot change
>+ * order while it is read, and this walk holds no page table
>lock
>+ * and no folio lock, so a folio splitting underneath it would
>+ * have the count read for the wrong size. A reference of our
>+ * own would not help: it stops the folio being freed, not
>split.
>+ *
>+ * So leave it to the freeze, which reads the table under the
>+ * lock and settles the question by freezing each source to the
>+ * count it expects. What it costs is a window selected here
>and
>+ * refused there.
>+ */
>+
>+ /*
>+ * Every check passed: this PTE can be a collapse source. The
>+ * bit is set last, so a disqualified PTE leaves it clear.
>+ */
>+ __set_bit(i, cc->eligible_ptes);
>+
>+ /*
>+ * Whether a range has to look used at all is the caller's
>+ * policy, so only a caller that asks gathers the evidence.
>+ */
>+ if (cc->policy.require_referenced &&
>+ (pte_young(pteval) || folio_test_young(folio) ||
>+ folio_test_referenced(folio) ||
>+ mmu_notifier_test_young(vma->vm_mm, addr)))
>+ referenced++;
>+ }
>+
>+ if (cc->policy.require_referenced &&
>+ (!referenced || (unmapped && referenced < HPAGE_PMD_NR / 2)))
>+ result = SCAN_LACK_REFERENCED_PAGE;
>+ else
>+ result = pmd_result;
>+ pte_unmap(pte);
>+ goto out;
>+
>+out_table_refused:
>+ /*
>+ * The table is refused as a unit -- a limit the whole range exceeds, or
>+ * pages on nodes too distant for one folio to serve them all -- so no
>+ * window inside it is eligible either.
>+ */
>+ pte_unmap(pte);
>+out_no_table:
>+ cc->select_orders = 0;
>+out:
>+ /*
>+ * A PMD candidate needs the whole table, so anything that disqualified
>a
>+ * single PTE rules it out. Smaller windows that avoid the offending
>+ * PTEs are still collapsible, so drop just that order and leave the
>rest
>+ * to selection -- dropping it also lowers the order selection roots its
>+ * windows at. MADV_COLLAPSE has no other order enabled, so it is left
>+ * with none.
>+ */
>+ if (result != SCAN_SUCCEED)
>+ cc->select_orders &= ~BIT(HPAGE_PMD_ORDER);
>+
>+ return result;
> }
>
> /* Everything a table is judged on starts empty for each table */
>diff --git a/mm/collapse.h b/mm/collapse.h
>index e2af4c47cb60..ad88b91d9a72 100644
>--- a/mm/collapse.h
>+++ b/mm/collapse.h
>@@ -130,5 +130,12 @@ unsigned long collapse_possible_orders(struct
>vm_area_struct *vma,
> vm_flags_t vm_flags, enum tva_type tva_flags);
> enum scan_result find_pmd_or_thp_or_none(struct mm_struct *mm,
> unsigned long address, pmd_t **pmd);
>+bool collapse_scan_abort(int nid, struct collapse_control *cc);
>+unsigned int collapse_max_ptes_none(struct collapse_control *cc,
>+ struct vm_area_struct *vma, unsigned int order);
>+unsigned int collapse_max_ptes_swap(struct collapse_control *cc,
>+ unsigned int order);
>+unsigned int collapse_max_ptes_shared(struct collapse_control *cc,
>+ unsigned int order);
>
> #endif /* __MM_COLLAPSE_H */
>diff --git a/mm/khugepaged.c b/mm/khugepaged.c
>index 26d25093260b..9823884a83c9 100644
>--- a/mm/khugepaged.c
>+++ b/mm/khugepaged.c
>@@ -305,7 +305,7 @@ struct attribute_group khugepaged_attr_group = {
> *
> * Return: Maximum number of empty/shared zeropage PTEs for the collapse
> operation
> */
>-static unsigned int collapse_max_ptes_none(struct collapse_control *cc,
>+unsigned int collapse_max_ptes_none(struct collapse_control *cc,
> struct vm_area_struct *vma, unsigned int order)
> {
> const unsigned int max_ptes_none = cc->policy.max_ptes_none;
>@@ -341,7 +341,7 @@ static unsigned int collapse_max_ptes_none(struct
>collapse_control *cc,
> * Return: Maximum number of PTEs that map shared anonymous pages for the
> * collapse operation
> */
>-static unsigned int collapse_max_ptes_shared(struct collapse_control *cc,
>+unsigned int collapse_max_ptes_shared(struct collapse_control *cc,
> unsigned int order)
> {
> /*
>@@ -362,7 +362,7 @@ static unsigned int collapse_max_ptes_shared(struct
>collapse_control *cc,
> * Return: Maximum number of non-present PTEs or the maximum allowed
> non-present
> * pagecache entries for the collapse operation.
> */
>-static unsigned int collapse_max_ptes_swap(struct collapse_control *cc,
>+unsigned int collapse_max_ptes_swap(struct collapse_control *cc,
> unsigned int order)
> {
> /*
>@@ -934,7 +934,7 @@ static struct collapse_control khugepaged_collapse_control
>= {
> .is_khugepaged = true,
> };
>
>-static bool collapse_scan_abort(int nid, struct collapse_control *cc)
>+bool collapse_scan_abort(int nid, struct collapse_control *cc)
> {
> int i;
>
>--
>2.54.0
>
>
