Device migration replaces a PTE that maps the zero page with a new PTE
that is built only from the VMA flags, so its userfaultfd WP or RWP
protection is lost. The same applies to the huge zero PMD. Writes and
accesses to the page are then no longer reported to userfaultfd.
To fix it, do not migrate a uffd-protected zero page. Skip it during
collection, and check again under the page table lock before inserting,
as another thread can protect the page after it was collected.
The alternative would be to carry the uffd state over to the new entry,
but a uffd-protected zero page in a range that a driver migrates should
be rare.
Fixes: 292924b26024 ("userfaultfd: wp: apply _PAGE_UFFD_WP bit")
Fixes: a30b48bf1b24 ("mm/migrate_device: implement THP migration of zone device
pages")
Cc: Thomas Hellström <[email protected]>
Cc: Lyude Paul <[email protected]>
Cc: Danilo Krummrich <[email protected]>
Cc: Felix Kuehling <[email protected]>
Cc: Jason Gunthorpe <[email protected]>
Cc: Leon Romanovsky <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Reviewed-by: Alistair Popple <[email protected]>
Signed-off-by: Tal Zussman <[email protected]>
---
mm/migrate_device.c | 23 +++++++++++++++++++++--
1 file changed, 21 insertions(+), 2 deletions(-)
diff --git a/mm/migrate_device.c b/mm/migrate_device.c
index 9e19fadfa65c..140b99f72508 100644
--- a/mm/migrate_device.c
+++ b/mm/migrate_device.c
@@ -159,7 +159,16 @@ static int migrate_vma_collect_huge_pmd(pmd_t *pmdp,
unsigned long start,
folio = pmd_folio(*pmdp);
if (is_huge_zero_folio(folio)) {
+ /*
+ * Nothing carries the uffd state of the zero page
+ * over, so skip it if its uffd bit is set.
+ */
+ bool protected = userfaultfd_protected(walk->vma) &&
+ pmd_uffd(*pmdp);
+
spin_unlock(ptl);
+ if (protected)
+ return migrate_vma_collect_skip(start, end,
walk);
return migrate_vma_collect_hole(start, end, -1, walk);
}
if (pmd_write(*pmdp))
@@ -337,6 +346,9 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
pfn = pte_pfn(pte);
if (is_zero_pfn(pfn) &&
(migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) {
+ /* See migrate_vma_collect_huge_pmd(). */
+ if (userfaultfd_protected(vma) && pte_uffd(pte))
+ goto next;
mpfn = MIGRATE_PFN_MIGRATE;
migrate->cpages++;
goto next;
@@ -871,10 +883,14 @@ static int migrate_vma_insert_huge_pmd_page(struct
migrate_vma *migrate,
if (userfaultfd_missing(vma))
goto unlock_abort;
- if (is_huge_zero_pmd(*pmdp))
+ if (is_huge_zero_pmd(*pmdp)) {
+ /* The uffd bit can have been set since the collection. */
+ if (userfaultfd_protected(vma) && pmd_uffd(*pmdp))
+ goto unlock_abort;
flush = true;
- else if (!pmd_none(*pmdp))
+ } else if (!pmd_none(*pmdp)) {
goto unlock_abort;
+ }
add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR);
folio_add_new_anon_rmap(folio, vma, addr, RMAP_EXCLUSIVE);
@@ -1074,6 +1090,9 @@ static void migrate_vma_insert_page(struct migrate_vma
*migrate,
if (!is_zero_pfn(pfn))
goto unlock_abort;
+ /* The uffd bit can have been set since the collection. */
+ if (userfaultfd_protected(vma) && pte_uffd(orig_pte))
+ goto unlock_abort;
flush = true;
} else if (!pte_none(orig_pte))
goto unlock_abort;
--
2.39.5