On Tue, Aug 25, 2026 at 08:23:54PM +0800, Lance Yang wrote:
> >The detached one is not: GUP-fast and RCU pte walks that read the old PMD
> >may still be inside it, and on broadcast-TLBI architectures the flush
> >expels nobody. Quiescing it would need an IPI, which has nowhere to go
> >here -- outside the pmd lock it opens the pmd_none() window this design
> >does not have, inside it is a broadcast under a spinlock. So the
> >detached table goes to pte_free_defer(), which holds the free until those
> >walkers finish. One transient table page per PMD collapse is the cost.
>
> Well, git history spells out why pmdp_get_lockless_sync() is needed here.
One more good catch, thanks!
> Could we keep pmdp_get_lockless_sync() right after pmdp_collapse_flush(),
> before map_anon_folio_pmd_nopf() (and while the locks are still held)?
Yes, I will put pmdp_get_lockless_sync() there.
Here's what I got to my tree:
/*
* Nothing fallible sits past here. No anon_vma_lock_write either: rmap
* walks on the sources are unreachable -- refcounts frozen, folio locks
* held from freeze to putback -- non-rmap pte walkers see migration
* entries, pmd-level observers see the old table or the leaf and never an
* intermediate, and fork, mremap and munmap take a write lock where the
* round holds a read lock.
*
* The flush inside pmdp_collapse_flush() is the round's second over this
* range: the freeze displaced every leaf here and flushed before dropping
* the ptl, and the verify above proved nothing has been mapped since.
* What it covers is the paging-structure caches -- a CPU may still hold
* the pmd-to-table link, for a table that is about to be freed -- which
* is why the helper shoots down a pte range rather than a pmd.
*
* If pmd_t is too wide to load in one access, a lockless walker reads
* it half at a time. Such a walker holds interrupts off, so an
* interrupt between two present values is what keeps it from assembling
* halves of both; the flush above does not always send one.
* pmdp_get_lockless_sync() does, and is an empty inline wherever the
* entry loads atomically.
*/
old_pmd = pmdp_collapse_flush(vma, cand->addr, pmd);
pmdp_get_lockless_sync();
old_table = pmd_pgtable(old_pmd);
/*
* The smp_wmb() in __folio_mark_uptodate() orders the copied data before
* the install below publishes it.
*/
__folio_mark_uptodate(cand->new_folio);
/*
* Deposit a freshly allocated table, not the one just detached: a
* deposited table has to be quiescent, because whoever withdraws it frees
* it immediately (zap_huge_pmd()) with nothing to hold a lockless walker
* off first. A table that has never been reachable is quiescent by
* construction, which is why collapse_alloc() secured one.
*
* The detached table is not quiescent. GUP-fast and RCU pte walks that
* read the old PMD before pmdp_collapse_flush() may still be inside it,
* and on broadcast-TLBI arches that flush expels nobody. Quiescing it
* would take an IPI in a pmd_none window, which this design does not
* have. So the table goes to pte_free_defer(), which holds the free
* until those walkers finish, as retract_page_tables() does. One
* transient table page per PMD collapse is what that costs.
*/
pgtable_trans_huge_deposit(mm, pmd, cand->deposit);
map_anon_folio_pmd_nopf(cand->new_folio, pmd, vma, cand->addr);
Any objections?
--
Kiryl Shutsemau / Kirill A. Shutemov
