copy_huge_pmd() reads the source PMD without the page table lock and,
for a special PMD, installs that value in the child without reading it
again under the lock. The comment says the PMD cannot change while the
mmap lock is held for write, but unmap_mapping_range() does not take
the mmap lock: it zaps under i_mmap_rwsem and the PMD lock.
dup_mmap() links the child VMA into the file's i_mmap tree right after
the parent's VMA before copying the page tables, so a zap visits the
parent first and the child second. When the zap clears the parent PMD
after fork has read it, and passes the child before fork installs the
copy, the child keeps a mapping the zap was meant to remove.
Drivers that map special PMDs with vmf_insert_pfn_pmd() rely on that
zap. vfio-pci zaps its BARs when the device memory is disabled or the
device is reset, so that userspace gets SIGBUS instead of reaching
disabled MMIO; a stale child mapping still reaches it. panthor zaps
the CPU mappings of a buffer object it evicts and then releases its
pages; a stale child mapping points to the freed pages.
Read the PMD again under the lock. The zap clears the parent PMD under
the same lock, so either the recheck finds it empty, or the copy is in
place before the zap reaches the child, which then removes it. If the
PMD is no longer a present special entry, return -EAGAIN: the source
mmap and VMA write locks keep faults out, so only a zap can have
cleared it, and copy_pmd_range() then skips the empty PMD. Otherwise
copy the value read under the lock. pmd_same() would be the wrong
test: threads of the parent can still set the accessed and dirty bits,
and -EAGAIN on a present huge PMD makes copy_pmd_range() clear it as a
bad PMD.
Fixes: bc02afbd4d73 ("mm/fork: accept huge pfnmap entries")
Cc: [email protected]
Assisted-by: LLM
Signed-off-by: Konstantin Khorenko <[email protected]>
---
mm/huge_memory.c | 23 ++++++++++++++++-------
1 file changed, 16 insertions(+), 7 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 1e5d68acf62a5..fcea98f0d8865 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1965,13 +1965,22 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct
mm_struct *src_mm,
src_ptl = pmd_lockptr(src_mm, src_pmd);
spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);
/*
- * No need to recheck the pmd, it can't change with write
- * mmap lock held here.
- *
- * Meanwhile, making sure it's not a CoW VMA with writable
- * mapping, otherwise it means either the anon page wrongly
- * applied special bit, or we made the PRIVATE mapping be
- * able to wrongly write to the backend MMIO.
+ * The mmap and VMA write locks of the source keep faults
+ * out, but unmap_mapping_range() does not take the mmap
+ * lock and may have cleared the pmd since the lockless
+ * read. Let copy_pmd_range() skip it then.
+ */
+ pmd = pmdp_get(src_pmd);
+ if (unlikely(!pmd_present(pmd) || !pmd_special(pmd))) {
+ spin_unlock(src_ptl);
+ spin_unlock(dst_ptl);
+ return -EAGAIN;
+ }
+ /*
+ * Make sure it's not a CoW VMA with writable mapping,
+ * otherwise it means either the anon page wrongly applied
+ * special bit, or we made the PRIVATE mapping be able to
+ * wrongly write to the backend MMIO.
*/
VM_WARN_ON_ONCE(vma_is_cow_mapping(src_vma) && pmd_write(pmd));
goto set_pmd;
--
2.43.0
