In memory management we've managed to manufacture a great deal of confusion around the concept of anonymous memory. We have:
1. 'Pure anon' memory - anonymous VMAs whose folios are anonymous and swap-backed (thus for reclaim purposes, treated as anonymous). These are simple enough. 2. shmem - file-backed VMAs, file-backed folios (from rmap perspective) so present in the page cache and mapped by an address_space object, but whose folios are also swap-backed (thus treated as anonymous for reclaim purposes). 3. MAP_PRIVATE-mapped /dev/zero - a strange beast whose VMAs have vma->vm_file set, but which clears vma->vm_ops to satisfy vma_is_anonymous(), resulting in VMAs that were mmap()'d referencing a file, but are in every other sense anonymous, including the folios. 4. Other MAP_PRIVATE-file backed mappings - These possess file-backed VMAs and have file-backed folios until CoW'd, at which point those CoW'd folios are anonymous. This series fixes issues 3 and 4. In order for us to traverse VMAs using the reverse mapping, we require two fields - folio->mapping and folio->index. The first tells the rmap code where to look for VMAs, and the second tells it at which offset the folio starts within the referenced object. For anonymous folios, folio->mapping points at an anon_vma object. For file-backed folios, it points at an address_space. And: * For file-backed folios folio->index is simply the page offset of the start of the folio within the file. * For anonymous folios belonging to pure anon mappings, folio->index is equal to the anonymous page offset of the folio. * For anonymous folios belonging to file-backed mappings (i.e. CoW'd folios of a MAP_PRIVATE file-backed mapping), folio->index is equal to the file page offset. This series establishes a new anonymous page offset property of VMAs to allow us to map anonymous folios at their anonymous page offset, consistent with pure anon. The purpose of doing so is to lay the foundations for the scalable CoW work. This is necessary because scalable CoW looks in the maple tree for the VMA located at folio->index << PAGE_SHIFT, before falling back to looking up tracked remaps if necessary. The MAP_PRIVATE file-backed case means that folio indices will very often conflict with one another and this remap tracking becomes substantially more contended, and of course the fast path can never be used. This also makes it possible, in future, to unshare anonymously mapped folios with deep fork hierarchies on remap, eliminating the need for remap tracking in the vast majority of cases. Similar to page offset of pure anonymous VMAs, we update the anonymous page offset of unfaulted file-backed VMAs on remap, but do not once CoW'd (i.e. vma->anon_vma is non-NULL). Overall, there is little impact on mergeability, which remains exactly the same for pure anonymous and shared file-backed mappings, with the only impact being on MAP_PRIVATE-mapped file-backed mappings, which must now match on anonymous page offset as well as file page offset to be merged. To fail to merge like this would require CoW'ing the mapping, then finding another VMA with identical file and compatible page offset to remap next to. This is therefore very much an edge case that should have very little impact (and which scalable CoW may very well address in any case). We also address the outlier case of MAP_PRIVATE-/dev/zero mappings which have file page offset but satisfy vma_is_anonymous() by making them truly anonymous. This is low-risk as for anything meaningful these mappings behave anonymously, but it is very beneficial to do so as we eliminate an odd corner case, and those are notorious for attracting subtle bugs. Signed-off-by: Lorenzo Stoakes (ARM) <[email protected]> --- v3: - Renamed virtual page offset to anonymous page offset across the series as per David and adjusted prose to reflect it. - As part of the rename, eliminated vma_anon_pgoff_addr() and the conflicting vma_[start, end]_anon_pgoff() functions by passing anon pgoff to the merge logic and having that figure out which page offset to use, significantly simplifying things. - Introduced needs_adjacent_anon_pgoff() to be really clear about what the merge logic is doing. - Passing through the anon pgoff fixes the issue Sashiko raised with shared file-backed mappings having incorrect anon pgoff (inconsequential but a wrinkle nonetheless). - Dropped unnecessary needs_rmap_locks change in 9/15 - this checks to see if rmap locks need to be taken due to the range being moved backwards. Since both file-backed and anon page offset are updated on such a move, it suffices to check only the former. Updated commit message to reflect it. - Sashiko complained about a couple missing .is_anon_walk entries in pvmw's - page_mapped_in_vma() and migrate_vma_collect_huge_pmd() - however neither impact anything - the field is only meaningful for vma_address_end() if nr_pages > 1 and neither site is impacted. Moreover, neither site sets pgoff either. Update the commit message to explain this. - Renamed is_anon_walk to pgoff_is_anon and document that pvmw->pgoff is only meaningful if pvmw->nr_pages > 1. - Highlighted the user-visible changes to /dev/zero being true-anon in the relevant commit message as per Yang. - Fixed a bug in the proc-self-map-files-00[1,2].c procfs selftests - the code was trying to MAP_PRIVATE 'an arbitrary file' then asserts that file-backed procfs entries exist, but happened to choose /dev/zero. Updated to the guaranteed-available /proc/self/exe, as reported by Mark. - Fixed an issue with drivers that intentionally mark vma->vm_ops as NULL (using the legacy ->mmap callback). If they do this set dummy ops, which is what they meant. The mmap_prepare case is fine as nobody does this there and this will be fixed when all drivers are finally converted to mmap_prepare. As reported by Sashiko. - Added some missed VMA selftest vma_start_[anon]_pgoff() conversions in the merge test commit as per Sashiko. - Various small prose/comment fixups. v2: - Removed incorrect assert on always-NULL folio from 7/15, as per syzbot. - Updated 2/15 so linear_virt_page_index() checks for vma_is_anonymous() as well to be cautious about 'special' (VDSO, VVAR, etc.) VMAs accidentally being asserted when CONFIG_DEBUG_VM is set, as per Sashiko. - Updated commit message of 9/15 to mention the subtle change in NUMA interleaving behaviour, as per Sashiko. - Updated 10/15 to assert virtual page offset for adjacent VMAs for various VMA userland tests, as per Sashiko. - Updated 13/15 to remove the !vma->vm_file check altogether after MAP_PRIVATE-/dev/zero is made pure anon in vma_start_virt_pgoff(). - Updated 12/15 to check that the /dev/zero device is a character device since it turns out that block and character devices have their own separate major/minor device number namespaces... :) as per Sashiko. - Updated 12/15 to fix a bisection hazard where vma->vm_ops would be overwritten by vma_dummy_vm_ops for MAP_PRIVATE-/dev/zero, as per Sashiko. - Updated 12/15 to fix another bisection hazard (...!) due to ordering of vma_set_anonymous(). Removed in 13/15. - Added comments to __vm_virt_pgoff[lo, hi] fields referencing vma_start_virt_pgoff()'s comment to be clearer what these are as per Xu Xin in 1/15. - Various typo fixes + cleanups in prose. - Updated the cover letter to point out that the MAP_PRIVATE-/dev/zero issue is addressed in this series too. https://patch.msgid.link/[email protected] v1: - Rebased onto mm-new. - Dependent series heavily reviewed and looks highly likely to land, so un-RFC. - Added explicit check for /dev/zero and removed ability for arbitrary mmap/mmap_prepare hooks to make themselves anonymous. - Made MAP_PRIVATE-/dev/zero mappings truly anonymous. - Added MAP_PRIVATE file-backed mapping merge test to selftests. - Added a MAP_PRIVATE-/dev/zero VMA userland test to assert that the VMA really is made anonymous. - Added MAP_PRIVATE-/dev/zero merge tests to selftests. - Fixed missed virtual page index site in try_to_merge_with_ksm_page(). - Updated folio_within_range() to use virtual page offset for anon folio. This had no impact as it is only called for large folios at the moment (and MAP_PRIVATE-file backed mappings can't currently be backed by a large folio) but making the change now protects us for the future. - Fixed typos etc. https://patch.msgid.link/[email protected] RFC: https://patch.msgid.link/[email protected] To: Andrew Morton <[email protected]> To: David Hildenbrand <[email protected]> To: "Liam R. Howlett" <[email protected]> To: Vlastimil Babka <[email protected]> To: Mike Rapoport <[email protected]> To: Suren Baghdasaryan <[email protected]> To: Michal Hocko <[email protected]> To: Jann Horn <[email protected]> To: Pedro Falcato <[email protected]> To: "Matthew Wilcox (Oracle)" <[email protected]> To: Jan Kara <[email protected]> To: Miaohe Lin <[email protected]> To: Naoya Horiguchi <[email protected]> To: Rik van Riel <[email protected]> To: Harry Yoo <[email protected]> To: Lance Yang <[email protected]> To: Kees Cook <[email protected]> To: Zi Yan <[email protected]> To: Baolin Wang <[email protected]> To: Nico Pache <[email protected]> To: Ryan Roberts <[email protected]> To: Dev Jain <[email protected]> To: Barry Song <[email protected]> To: Usama Arif <[email protected]> To: Matthew Brost <[email protected]> To: Joshua Hahn <[email protected]> To: Rakie Kim <[email protected]> To: Byungchul Park <[email protected]> To: Gregory Price <[email protected]> To: Ying Huang <[email protected]> To: Alistair Popple <[email protected]> To: Peter Xu <[email protected]> To: Xu Xin <[email protected]> To: Chengming Zhou <[email protected]> To: Arnd Bergmann <[email protected]> To: Greg Kroah-Hartman <[email protected]> Cc: [email protected] Cc: [email protected] Cc: [email protected] Cc: [email protected] Cc: [email protected] --- Lorenzo Stoakes (ARM) (15): mm/vma: introduce VMA anon page offset field and add helpers mm: introduce linear_anon_page_index() mm: abstract vma_address() and introduce vma_anon_address() mm: update print_bad_page_map() to show anonymous page index mm: introduce and use vma_filebacked_address() mm: propagate VMA anonymous page offset on map, remap, split + merge mm/rmap: track whether the page VMA mapped pgoff is anonymous mm: introduce and use linear_folio_page_index() mm/rmap: use anon pgoff to track MAP_PRIVATE file-backed anon folios tools/testing/vma: expand VMA merge tests to assert anon pgoff tools/testing/selftests/mm: test anonymous page offset merge behaviour mm/vma: only permit MAP_PRIVATE /dev/zero to be mapped anonymous mm/vma: make MAP_PRIVATE-mapped /dev/zero mappings truly anonymous tools/testing/vma: add test to assert MAP_PRIVATE-/dev/zero is anon tools/testing/selftests/mm: add MAP_PRIVATE-/dev/zero merge tests drivers/char/mem.c | 8 +- include/linux/mm.h | 74 ++++++++-- include/linux/mm_types.h | 12 ++ include/linux/pagemap.h | 66 +++++++++ include/linux/rmap.h | 4 +- mm/huge_memory.c | 3 +- mm/internal.h | 89 ++++++++---- mm/interval_tree.c | 4 +- mm/ksm.c | 7 +- mm/memory-failure.c | 4 +- mm/memory.c | 8 +- mm/migrate.c | 6 +- mm/mremap.c | 6 +- mm/page_vma_mapped.c | 6 +- mm/rmap.c | 22 +-- mm/userfaultfd.c | 6 +- mm/vma.c | 127 +++++++++++++--- mm/vma.h | 92 +++++++----- mm/vma_exec.c | 2 +- mm/vma_init.c | 1 + mm/vma_internal.h | 1 + tools/testing/selftests/mm/merge.c | 161 +++++++++++++++++++++ .../selftests/proc/proc-self-map-files-001.c | 2 +- .../selftests/proc/proc-self-map-files-002.c | 2 +- tools/testing/vma/include/dup.h | 87 ++++++++++- tools/testing/vma/shared.c | 3 +- tools/testing/vma/tests/merge.c | 49 +++++-- tools/testing/vma/tests/mmap.c | 50 +++++++ tools/testing/vma/tests/vma.c | 4 +- tools/testing/vma/vma_internal.h | 1 + 30 files changed, 766 insertions(+), 141 deletions(-) --- base-commit: 5093dba1014c1d7f7e247fd118f0fa8f22136046 change-id: 20260711-b4-scalable-cow-virt-pgoff-a0cc0eb14bc6 Cheers, -- Lorenzo Stoakes (ARM) <[email protected]>
