On Fri, 7 Aug 2026 at 23:52, Ackerley Tng via B4 Relay <[email protected]> wrote: > > From: "David Hildenbrand (Arm)" <[email protected]> > > KVM with guest_memfd wants to remove any folio references due to LRU > caches, as it really must only allow to convert folios from shared to > private when there are no unexpected folio references (e.g., from GUP > references). > > So, to drive the refcount down, it needs a way to flush the LRU caches. > Let's factor out what we have in lru_cache_drain_for_folio(). Document > it, and also mention that concurrent folio (un)mapping might, in theory, > miss detecting LRU cache references. Keep obtaining the expected refcount > twice to minimize the possibility. For the current and future user that > should work, and we don't really have a better alternative: we could > detect if the mapcount changed, but it would still be racy and add more > complexity with questionable benefit. > > Maybe there is a chance to avoid the draining entirely in the future, > by avoiding extra references from the LRU cache: Hugh thinks there might > be a way. But for the time being, this handling is unfortunately > required. > > Make folio_may_be_lru_cached() accept a const pointer so > lru_cache_drain_for_folio() can accept a const pointer as well. > > Signed-off-by: David Hildenbrand (Arm) <[email protected]> > Signed-off-by: Ackerley Tng <[email protected]>
Reviewed-by: Fuad Tabba <[email protected]> Cheers, /fuad > --- > include/linux/swap.h | 11 ++++++++++- > mm/gup.c | 15 ++------------- > mm/swap.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ > 3 files changed, 58 insertions(+), 14 deletions(-) > > diff --git a/include/linux/swap.h b/include/linux/swap.h > index 8f0f68e245baa..e625d64c9e13d 100644 > --- a/include/linux/swap.h > +++ b/include/linux/swap.h > @@ -317,7 +317,7 @@ void folio_add_lru_vma(struct folio *, struct > vm_area_struct *); > void mark_page_accessed(struct page *); > void folio_mark_accessed(struct folio *); > > -static inline bool folio_may_be_lru_cached(struct folio *folio) > +static inline bool folio_may_be_lru_cached(const struct folio *folio) > { > /* > * Holding PMD-sized folios in per-CPU LRU cache unbalances > accounting. > @@ -344,6 +344,15 @@ extern void lru_add_drain(void); > extern void lru_add_drain_cpu(int cpu); > extern void lru_add_drain_cpu_zone(struct zone *zone); > extern void lru_add_drain_all(void); > + > +enum lru_cache_drained { > + LRU_CACHE_NOT_DRAINED, > + LRU_CACHE_DRAINED, > + LRU_CACHE_DRAINED_ALL, > +}; > +void lru_cache_drain_for_folio(const struct folio *folio, > + unsigned int extra_refs, enum lru_cache_drained *drained); > + > void folio_deactivate(struct folio *folio); > void folio_mark_lazyfree(struct folio *folio); > extern void swap_setup(void); > diff --git a/mm/gup.c b/mm/gup.c > index bde05664fe9a9..1bd8915517196 100644 > --- a/mm/gup.c > +++ b/mm/gup.c > @@ -2266,9 +2266,9 @@ static unsigned long collect_longterm_unpinnable_folios( > struct list_head *movable_folio_list, > struct pages_or_folios *pofs) > { > + enum lru_cache_drained drained = LRU_CACHE_NOT_DRAINED; > unsigned long collected = 0; > struct folio *folio; > - int drained = 0; > long i = 0; > > for (folio = pofs_get_folio(pofs, i); folio; > @@ -2293,18 +2293,7 @@ static unsigned long > collect_longterm_unpinnable_folios( > * but also to remove any other folio references from LRU > * caches. > */ > - if (drained == 0 && folio_may_be_lru_cached(folio) && > - folio_ref_count(folio) != > - folio_expected_ref_count(folio) + pin_refs) { > - lru_add_drain(); > - drained = 1; > - } > - if (drained == 1 && folio_may_be_lru_cached(folio) && > - folio_ref_count(folio) != > - folio_expected_ref_count(folio) + pin_refs) { > - lru_add_drain_all(); > - drained = 2; > - } > + lru_cache_drain_for_folio(folio, pin_refs, &drained); > > if (!folio_isolate_lru(folio)) > continue; > diff --git a/mm/swap.c b/mm/swap.c > index 588f50d8f1a8c..8e965c8ce9aa9 100644 > --- a/mm/swap.c > +++ b/mm/swap.c > @@ -950,6 +950,52 @@ void lru_add_drain_all(void) > } > #endif /* CONFIG_SMP */ > > +/** > + * lru_cache_drain_for_folio() - drain LRU caches if the caches might hold > + * folio references > + * @folio: The folio. > + * @extra_refs: Extra folio references held by the caller. > + * @drained: Drain status for batch folio processing. > + * > + * Drain LRU caches if the caches might hold folio references. Start > + * with a local LRU cache drain, to then drain LRU caches on all CPUs if > + * local draining was insufficient. > + * > + * This function detects LRU cache references by comparing the folio refcount > + * with the sum of the expected folio refcount + extra references held by the > + * caller. Note that we cannot rely on PG_lru to reliably detect all LRU > + * cache references, and there are rare scenarios (concurrent folio > (un)mapping) > + * where this function might miss detecting LRU cache references. > + * > + * If @drained is not NULL, the function will avoid re-draining LRU caches > + * when processing multiple folios in a row. In that case, the variable > + * @drained points at must be initialized to LRU_CACHE_NOT_DRAINED before > + * the first invocation by the caller. > + */ > +void lru_cache_drain_for_folio(const struct folio *folio, > + unsigned int extra_refs, enum lru_cache_drained *drained) > +{ > + if (!folio_may_be_lru_cached(folio)) > + return; > + > + if (!drained || *drained == LRU_CACHE_NOT_DRAINED) { > + if (folio_ref_count(folio) == > + folio_expected_ref_count(folio) + extra_refs) > + return; > + lru_add_drain(); > + if (drained) > + *drained = LRU_CACHE_DRAINED; > + } > + if (!drained || *drained == LRU_CACHE_DRAINED) { > + if (folio_ref_count(folio) == > + folio_expected_ref_count(folio) + extra_refs) > + return; > + lru_add_drain_all(); > + if (drained) > + *drained = LRU_CACHE_DRAINED_ALL; > + } > +} > + > atomic_t lru_disable_count = ATOMIC_INIT(0); > > /* > > -- > 2.55.0.654.g21b8a5bc05-goog > >
