From: "Kiryl Shutsemau (Meta)" <[email protected]>
Yes, I know, this is a lot of changes. But I'm happy with the overall state
of the patchset and the only reason I tag it as RFC is that it is tricky
to get 57 patches upstream.
I wanted to give a view of the end state first. I will suggest a possible
way to split it below.
I would appreciate any feedback.
TL;DR
=====
This replaces khugepaged's anonymous collapse with an engine that
can collapse sub-PMD ranges. It is built around migration entries and
frozen folios instead of heavy locking and isolation, aiming for better
scalability and less disruption to the workload being collapsed.
Why
===
mTHP collapse landed in khugepaged in 7.2 and I was glad to see it. We
at Meta run arm64 with 64K base pages, where a PMD is 512M: PMD-order THP
is of limited use at that size, and mTHP is exactly what we want.
It turned out not to help us.
khugepaged only ever looks at PMD-aligned windows, and it is not an easy
limitation to lift.
Fixing the alignment is a one-line change, but what it feeds assumes the
PMD everywhere that matters: collapse_huge_page() clears and flushes the
whole PMD whatever order it is collapsing, installs a PMD leaf because
that is the only thing it can produce, and keeps everyone out with
mmap_write_lock, anon_vma_lock_write() and an IPI broadcast while it
does.
Which is why hugepage_vma_revalidate() demands that the VMA span the
whole PMD even for an mTHP order -- "we'd need to lock all VMAs in the
PMD range to support this", as the comment there puts it. A PMD-granular
operation is only safe when one VMA owns the PMD, and that is exactly the
restriction in the way. The alignment is the symptom; the PMD is the
design.
So both roots have to go.
Design
======
The old mechanism holds the address space still because it has nothing
else stopping the sources from moving under the copy. The new engine
makes the sources themselves inert instead, with the two barriers
migration already uses, raised in that order:
1. migration entries replace the source PTEs. Faults and GUP-slow
now wait on the source folio's lock, which is taken before the
first entry becomes visible.
2. the source folio's refcount is frozen to its expected value.
GUP-fast, pfn walkers, reclaim, compaction and memory-failure all
fail folio_try_get() and back off.
Between the two, nothing can reach a source, so the copy runs with no
lock held at all -- and the address space is left alone while it does.
What that removes from every collapse path:
mmap_write_lock -> mmap_read
anon_vma_lock_write() -> nothing: an rmap walk needs the folio
locked, and the engine holds that lock
from freeze to putback
tlb_remove_table_sync_one() -> nothing: one ranged flush per round
LRU isolation -> nothing: sources are inert in place
Working in windows rather than whole PMDs takes care of the other root.
A sub-PMD window is collapsed under the page table lock, so a collapse
disturbs only the window it collapses, and each candidate is validated
at its own order -- a window need only fit its own VMA. A PMD-order
candidate still has to own the whole PMD, which is the old rule kept
where it is still needed.
Candidates are carried through the passes a batch at a time rather than
one window at a time, so a round pays for its flush and its lock
acquisitions once.
With the barriers holding the sources still, which read lock the engine
takes stops being part of the design. A round works inside a single
VMA, so patches 43-49 switch it from mmap_read to per-VMA locking: an
mmap_write elsewhere in the mm then stops waiting for a collapse that
has nothing to do with it. That block is the only part of the series
that needs per-VMA locking to be unconditional, and it is a separate
dependency (see below); everything before it runs under mmap_read and
does not care.
Patch 7 sketches the engine as a comment naming every pass, what lock it
takes and what it may sleep on; the details are there rather than here.
What falls out beyond the lock diet:
- mTHP collapse in VMAs smaller than a PMD, which is the arm64 case
above: a 2M VMA on an arm64/64K machine collapses nothing today at
any order, and collapses to mTHP here.
- Hole and zeropage population at every order, so partially populated
windows collapse to mTHP under the same max_ptes_none policy as PMD.
- Sources come in spans -- any stretch of consecutive PTEs mapping
consecutive pages of one folio -- so partially mapped and scrambled
compound sources (the PTE-mapped-THP re-collapse class) work at
every order.
- A table that cannot become one huge page still yields the largest
windows inside it, where before a single disqualified PTE gave up
the whole table.
Reading the series
==================
57 patches is a lot to land on a list. They go in blocks:
1-6 helpers and shared state: pte_folio(), pte_none_or_zero(),
mm/collapse.h, and the policy that replaces asking whether
khugepaged started a collapse
7-8 the engine's shape: entry points, a call-tree comment naming
every pass, and the scan filled in
9-23 the collapse half, top down: the round frame, then each pass
in turn, then selection and the retry store
24 per-candidate tracing, before the switch takes the old
tracepoints away
25-28 the switch: point the anon path at the engine, widen coverage
to sub-PMD VMAs, delete the mechanism it replaces
29-35 move what is left of collapse out of khugepaged.c, and
MADV_COLLAPSE into madvise.c
36-42 tracing: the engine's own events and trace header
43-49 per-VMA locking, and the mm reference that makes it safe
50-56 selftests for what the engine can now do
57 MAINTAINERS
The two patches worth reading first if you read nothing else are 7 (the
design, as a comment naming the whole call tree) and 16 (the freeze,
which is where the safety argument lives).
A possible split, if that helps:
1-2 two mm helpers, pte_folio() and pte_none_or_zero(). Both
convert callers outside collapse and are useful on their own
3-27 the engine and the switch-over. This is the smallest unit
that does anything: stop earlier and the tree carries an
engine nothing calls
28 remove the mechanism the engine replaces
29-42 moving what is left of collapse out of khugepaged.c, and the
engine's own tracepoints
43-49 per-VMA locking
50-57 selftests and MAINTAINERS
Keeping the removal separate leaves both engines in the tree with only
the new one reachable, so the switch can be reverted on its own if
something turns up. The old mechanism is already carried that way for
three patches inside the series, so this costs nothing but 975 lines of
unreferenced code until 28 lands. That safety net only lasts until the
blocks after it land, though: once collapse has moved out of
khugepaged.c and the locking has changed, reverting the switch no longer
gives back a working old engine.
Base and dependencies
=====================
This applies on the selftests series, not on plain mm-new:
[PATCH v4 00/19] selftests/mm: improve khugepaged coverage
https://lore.kernel.org/all/[email protected]/
which is on mm-new 33f61b12d297.
Patches 43-49 depend on Suren's unconditional per-VMA locks:
[PATCH v6 0/5] mm: Unconditional per-VMA locks and cleanups
https://lore.kernel.org/all/[email protected]/
That series is not in mm-new yet, and with patch 46 applied SMP=n does
not build without it: lock_next_vma() is behind CONFIG_PER_VMA_LOCK in
mmap_lock.h. Everything up to patch 42 builds and runs on mm-new as it
stands. There is no fallback path by choice -- adding one would mean
carrying two locking models through every pass.
Both branches are available at
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git collapse/rfc-v1
and the benchmark used for the numbers below, which is unposted and not a
dependency, at
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git perf/bench-usemem
Performance
===========
Measuring khugepaged is awkward. It is a background daemon, so what
matters is what a workload feels while it runs, not what the daemon
reports about itself -- and the usual coverage instrument is no help
below the PMD: smaps AnonHugePages only counts PMD-order folios, so it
reads zero however much mTHP has been collapsed.
So I wrote "perf bench mem usemem" for this. It touches a region while
khugepaged works on it and reports the workload's own latency
percentiles and throughput, against per-size counters that can see
sub-PMD folios. The branch is above; it is unposted and not a
dependency.
x86-64, production configs (no KASAN, no lockdep, no DEBUG_VM, no
PAGE_TABLE_CHECK), interleaved rounds on an idle host, equal work on
every arm.
I measured three kernels, so the two halves of the series can be told
apart in the numbers below:
A the base
B the new engine, still under mmap_read
C B plus per-VMA locking -- what this series ends up with
The engine: a sub-PMD collapse stops blanking the surrounding 2M
-----------------------------------------------------------------
base routes sub-PMD collapse through collapse_huge_page(), whose
pmdp_collapse_flush() and tlb_remove_table_sync_one() are not gated on
order: to collapse an order-4 window of 16 pages it clears and flushes
the whole 512-page PMD and IPIs, then repopulates. The engine does the
window under the PTL.
A thread reading and writing a 32G region while it is collapsed at
order-4, 16384 collapses on every arm:
A B C
read p99 (ns) 3071 1023 1023 -66.7%
write p99 (ns) 3071 927 927 -69.8%
and the workload's read rate rises by 68% on both engine kernels.
B == C, so this is the engine, not the locking.
At PMD order the same workload is flat, and that is expected rather
than disappointing: it is the one configuration where both mechanisms
disturb exactly the same 2M. Read it as no regression at PMD order.
Per-VMA locking: address-space operations stop waiting on the scan
-------------------------------------------------------------------
MADV_HUGEPAGE/MADV_NOHUGEPAGE toggling against a scanning mm, which is
what jemalloc does with its arenas. 4096 collapses on every arm:
A B C
ops/sec 340656 340820 603305 +77%
p99 (ns) 77823 86015 3327 -96%
p99.9 (ns) 86015 94207 9215 -89%
B is about 11% worse than base at p99 here, consistently across runs:
the engine alone slightly worsens hint-toggle latency, and per-VMA
locking is what turns it into a win. Both halves are in this series, so
C is what a reviewer gets, but the middle column is the honest one.
The trade is real in the other direction too. On settled memory with
nothing to collapse and scan_sleep_millisecs=0, per-VMA locking costs
about 47% of scan throughput against one mmap_read for the whole walk.
That is a synthetic worst case -- the daemon wraps 8000 times a second
there, where production defaults to 10s between passes -- and it buys
mmap/munmap p99 of 56us against 1.4us.
Collapse itself is not slower
-----------------------------
One complete pass over a 32G region, 16384 collapses, khugepaged CPU
from /proc/<pid>/stat, 7 repeats:
A base median 5760 ms spread 12.7%
B engine median 4990 ms spread 3.8%
C pervma median 5020 ms spread 12.4%
The base arm is bimodal, so its median moves with sampling and the
percentage is soft. The distribution-free statement is better: every
engine run used less CPU than every base run.
The engine also allocates one destination per folio installed, where
base allocates 5.25 and frees the rest again: nothing is allocated until
the sources are frozen and the collapse can no longer be refused.
A measurement note, since an earlier version of this series quoted worse
figures. khugepaged CPU has to be measured per collapse or per
completed pass, never over a fixed window with scan_sleep_millisecs=0:
the daemon never sleeps, so whichever kernel finishes the work sooner
spends the rest of the window scanning settled memory and is charged for
it. Measured that way the engine appeared to cost 10% more CPU;
measured per unit of work it costs less.
Costs
=====
At PMD order the engine issues two TLB flushes per collapse where the
old mechanism issues one: the freeze's ranged flush plus the terminal
layer's pmdp_collapse_flush(). A PMD candidate is alone in its round,
so nothing amortizes the first. Dropping the old per-collapse
tlb_remove_table_sync_one() IPI presumably pays for it, but that was not
measured and is not claimed here.
There may be a way out -- a PMD migration entry over the table during
the window, so the CPU never caches a walk to shoot down -- but that
means teaching every pmd-level walker a new kind of entry, and I have
not tried it.
PMD collapse deposits a freshly allocated page table instead of
redepositing the detached one. Whoever withdraws a deposited table
frees it immediately, with nothing to hold a lockless walker off first,
and under a read lock the detached table may still be traversed by
GUP-fast or an RCU pte walk. It goes to pte_free_defer() instead,
exactly as retract_page_tables() does. One transient table page per PMD
collapse buys the IPI's absence.
That cost goes away if zap_deposited_table() -- the only site that frees
a deposited table outright, the others redeposit it or repopulate the
PMD with it -- used pte_free_defer(). The deposit would no longer have
to be quiescent and the detached table could go straight back. It would
defer every THP zap's table free, and I have not tried it.
A shared source now costs an extra copy. The freeze needs every page
exclusive to this mm, so the fault-in pass breaks CoW first -- an
allocation and a copy -- and the collapse then copies that page into the
destination; the old mechanism copied a shared page straight into the
new folio and broke the sharing that way. It is bounded by
max_ptes_shared, which khugepaged holds at zero below the PMD order, so
in practice this is PMD-order collapse and MADV_COLLAPSE.
Size
====
mm/ grows by 1915 lines net: 4475 added against 2560 deleted.
That is not a claim that this is less code, but it is less than it
looks. khugepaged.c goes from 3283 lines to 908. The new engine is
4052 lines across mm/collapse.c and mm/collapse.h, of which 1344 --
about a third -- are comments, which is where the pipeline's invariants
are written down. What replaces three install paths with their own
isolate/copy/rollback is one engine and one contract.
Testing
=======
Both matrices run the mm selftests plus a race harness, on the
validation config: KASAN, lockdep, PROVE_LOCKING, DEBUG_VM and
PAGE_TABLE_CHECK, 16G of guest memory, swap active so the swap-in
prepass is exercised rather than skipped.
x86-64 433 pass, 0 fail, 12 skip
arm64/64K 581 pass, 0 fail, 18 skip
dmesg clean on both. The arm64 skips are a pre-existing shmem
MADV_COLLAPSE -EINVAL on 64K pages, confirmed against the base by A/B.
Every one of the 57 patches builds with no new warnings; !NUMA and !MMU
(arm nommu) build clean. SMP=n does not build, for the reason in the
dependencies section above.
The race harness also gets longer soaks -- 1800s per driver mode, with
memory pressure and swap -- and the engine is fuzzed with syzkaller on a
KCOV+KASAN build. That found two bugs the selftests could not reach: a
teardown that dropped rmap while the source was still frozen, where
removing an mlocked mapping munlocks and munlock_folio() takes a
reference a frozen folio forbids; and a whole-table MADV_DONTNEED racing
the copy window under CONFIG_PT_RECLAIM, which freed the table and left
the sources frozen and locked. Both are fixed, and both gained coverage
-- the mlocked case is patch 53.
Kiryl Shutsemau (Meta) (57):
mm: add pte_folio()
mm: add pte_none_or_zero()
mm/collapse: add collapse.h for the shared collapse state
mm/collapse: rename mthp_present_ptes to eligible_ptes
mm/collapse: state what a collapse may do in the policy
mm/collapse: move the smallest collapse order to collapse.h
mm/collapse: sketch the new anonymous collapse engine
mm/collapse: scan a table for what a collapse could use
mm/collapse: collect candidate windows into a round
mm/collapse: run a round and feed the outcomes back
mm/collapse: sketch the passes of a round
mm/collapse: allocate a destination per candidate
mm/collapse: revalidate a round against the VMA
mm/collapse: fault the sources in before the freeze
mm/collapse: check what a candidate would freeze
mm/collapse: freeze the sources behind migration entries
mm/collapse: copy the sources into the destinations
mm/collapse: install the destinations at PTE level
mm/collapse: install a PMD leaf as the terminal layer
mm/collapse: put the sources back
mm/collapse: settle whatever the round reached
mm/collapse: walk a table with a selection cursor
mm/collapse: give a refused region a second chance
mm/collapse: report each candidate's outcome to tracing
mm/collapse: collapse anonymous memory with the new engine
mm/collapse: give collapse_single_pmd() the range to work on
mm/collapse: scan the windows a VMA can hold
mm/collapse: remove the mechanism the engine replaces
mm/collapse: move what a collapse is judged on into collapse.c
mm/collapse: name the max_ptes ceiling after collapse
mm/khugepaged: count collapses where khugepaged makes them
mm/collapse: move the file collapse into collapse.c
mm/collapse: split collapse into a scan and a run
mm/collapse: implement MADV_COLLAPSE in madvise.c
mm/madvise: drop MADV_COLLAPSE's redundant mm reference
mm/collapse: report what the scan found
mm/collapse: report what the fault-in pass paid
mm/collapse: report the round, and what it made faulters wait
mm/collapse: name the file collapse's tracepoints after collapse
mm/collapse: remove the tracepoints of the mechanism that is gone
mm/collapse: give collapse its own trace header
mm/collapse: allow error injection into the freeze
mm/khugepaged: check the scan budget before the work, not after
mm/khugepaged: hold the address space open across a scan
mm/collapse: take a per-VMA read lock for the round
mm/khugepaged: scan under a per-VMA read lock
mm/madvise: collapse under a per-VMA read lock
mm/collapse: assert the mm reference the engine relies on
mm/khugepaged: drop the mmap_lock barrier from __khugepaged_exit()
selftests/mm: attribute collapses by candidate event alone
selftests/mm: cover collapse inside a sub-PMD VMA
selftests/mm: cover a hole-y window in a sub-PMD VMA
selftests/mm: cover collapse of mlocked ranges
selftests/mm: cover collapse beside a MADV_FREE'd page
selftests/mm: cover collapse beside a pinned page
selftests/mm: cover the scaled max_ptes_shared limit
MAINTAINERS: add an entry for collapse
MAINTAINERS | 19 +-
fs/proc/task_mmu.c | 4 +-
include/linux/huge_mm.h | 9 -
include/linux/mm.h | 14 +
include/linux/pgtable.h | 17 +
.../events/{huge_memory.h => collapse.h} | 176 +-
kernel/bpf/btf.c | 8 +-
mm/Makefile | 2 +-
mm/collapse.c | 3808 +++++++++++++++++
mm/collapse.h | 244 ++
mm/hugetlb.c | 8 +-
mm/khugepaged.c | 2711 +-----------
mm/madvise.c | 251 +-
mm/migrate_device.c | 9 +-
mm/mremap.c | 2 +-
tools/testing/selftests/mm/khugepaged.c | 346 ++
tools/testing/selftests/mm/khugepaged_race.c | 29 +-
.../selftests/mm/khugepaged_sync_check.c | 65 +-
tools/testing/selftests/mm/vm_util.c | 2 +-
19 files changed, 5022 insertions(+), 2702 deletions(-)
rename include/trace/events/{huge_memory.h => collapse.h} (60%)
create mode 100644 mm/collapse.c
create mode 100644 mm/collapse.h
base-commit: 8b76faf42c5d342b3bf0b1fd97bdaf603ee57354
--
2.54.0
