hmm_range_fault() requires the caller to hold the mmap read lock for the duration of the call. This is incompatible with mappings whose fault handler may release the mmap lock, notably userfaultfd-managed regions, where handle_mm_fault() can return VM_FAULT_RETRY or VM_FAULT_COMPLETED after dropping the lock. Drivers that need to populate device page tables for such mappings have no way to do so today.
Add hmm_range_fault_unlocked_timeout() for callers that do not need to hold mmap_lock across any work outside the HMM fault itself. The helper takes mmap_read_lock_killable() internally, calls the common HMM fault implementation, and releases the lock before returning if it is still held. The timeout is specified in jiffies; passing 0 retries indefinitely, while a non-zero timeout makes the helper return -EBUSY when the retry budget expires. When handle_mm_fault() drops mmap_lock, or when the range is invalidated, hmm_range_fault_unlocked_timeout() refreshes range->notifier_seq and retries the walk internally. If the lock was dropped, the retry deadline is also restarted because a lock-dropping fault handler made progress. Ordinary -EBUSY retries keep the existing deadline, preserving the caller's timeout policy for repeated mmu-notifier invalidations. The caller only needs to perform the usual post-success mmu_interval_read_retry() check while holding its update lock before consuming the pfns. If mmap_lock acquisition is interrupted or a fatal signal is pending during retry handling, -EINTR is returned instead. The common implementation conditionally sets FAULT_FLAG_ALLOW_RETRY and FAULT_FLAG_KILLABLE only for hmm_range_fault_unlocked_timeout(). The existing hmm_range_fault() path still passes no locked state, does not allow handle_mm_fault() to drop mmap_lock, and remains a thin wrapper preserving the existing API contract for current callers. The previous refactor that moved page fault handling out of the page-table walk callbacks is what makes this change small. Faults now run after walk_page_range() has unwound, with only mmap_lock held, so dropping it does not interact with the walker's pte spinlock or hugetlb_vma_lock. Hugetlb regions therefore participate in the unlocked path uniformly with PTE- and PMD-level mappings; no special case is required. Documentation/mm/hmm.rst is updated with a description of the new API and the recommended caller pattern. Signed-off-by: Stanislav Kinsburskii <[email protected]> --- Documentation/mm/hmm.rst | 76 +++++++++++++++------ include/linux/hmm.h | 2 + mm/hmm.c | 165 ++++++++++++++++++++++++++++++++++++++-------- 3 files changed, 192 insertions(+), 51 deletions(-) diff --git a/Documentation/mm/hmm.rst b/Documentation/mm/hmm.rst index 7d61b7a8b65b..4e5a750748ae 100644 --- a/Documentation/mm/hmm.rst +++ b/Documentation/mm/hmm.rst @@ -156,42 +156,57 @@ During the ops->invalidate() callback the device driver must perform the update action to the range (mark range read only, or fully unmap, etc.). The device must complete the update before the driver callback returns. -When the device driver wants to populate a range of virtual addresses, it can -use:: +When the device driver wants to populate a range of virtual addresses, the +normal interface is:: - int hmm_range_fault(struct hmm_range *range); + int hmm_range_fault_unlocked_timeout(struct hmm_range *range, + unsigned long timeout); It will trigger a page fault on missing or read-only entries if write access is requested (see below). Page faults use the generic mm page fault code path just -like a CPU page fault. The usage pattern is:: +like a CPU page fault. + +The caller must not hold ``mmap_read_lock`` before the call. +``hmm_range_fault_unlocked_timeout()`` takes the mmap read lock internally and +allows ``handle_mm_fault()`` to drop it during fault handling. This is required +for VMAs whose fault handlers may release the mmap lock, for example regions +managed by ``userfaultfd``. + +If the mmap lock is dropped or the range is invalidated, the function refreshes +``range->notifier_seq`` and restarts the walk internally. ``-EINTR`` is returned +if mmap lock acquisition is interrupted or a fatal signal is pending during +retry handling. + +The timeout is specified in jiffies; passing ``0`` means retry indefinitely. The +timeout exists to preserve caller policy for repeated mmu-notifier invalidation +and is checked between retry attempts. HMM does not interrupt page fault +handling when the timeout expires, but returns ``-EBUSY`` if the retry budget is +exhausted before a stable range is obtained. + +The usage pattern is:: int driver_populate_range(...) { struct hmm_range range; + unsigned long timeout; ... + timeout = msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); range.notifier = &interval_sub; range.start = ...; range.end = ...; range.hmm_pfns = ...; - if (!mmget_not_zero(interval_sub->notifier.mm)) + if (!mmget_not_zero(interval_sub.mm)) return -EFAULT; again: - range.notifier_seq = mmu_interval_read_begin(&interval_sub); - mmap_read_lock(mm); - ret = hmm_range_fault(&range); - if (ret) { - mmap_read_unlock(mm); - if (ret == -EBUSY) - goto again; - return ret; - } - mmap_read_unlock(mm); + ret = hmm_range_fault_unlocked_timeout(&range, timeout); + if (ret) + goto out_put; take_lock(driver->update); - if (mmu_interval_read_retry(&ni, range.notifier_seq) { + if (mmu_interval_read_retry(&interval_sub, range.notifier_seq)) { release_lock(driver->update); goto again; } @@ -200,7 +215,11 @@ like a CPU page fault. The usage pattern is:: * under the update lock */ release_lock(driver->update); - return 0; + ret = 0; + + out_put: + mmput(interval_sub.mm); + return ret; } The driver->update lock is the same lock that the driver takes inside its @@ -208,6 +227,19 @@ invalidate() callback. That lock must be held before calling mmu_interval_read_retry() to avoid any race with a concurrent CPU page table update. +Holding the mmap lock across HMM faults +======================================= + +Most callers should use ``hmm_range_fault_unlocked_timeout()``. If a driver +really needs to hold the mmap lock across work outside HMM, it can use:: + + int hmm_range_fault(struct hmm_range *range); + +The mmap lock must be held by the caller and will remain held on return. This +interface cannot support VMAs whose fault handlers need to drop the mmap lock. +New callers should prefer ``hmm_range_fault_unlocked_timeout()`` unless they +have a specific requirement to keep the mmap lock held across the call. + Leverage default_flags and pfn_flags_mask ========================================= @@ -221,8 +253,8 @@ permission, it sets:: range->default_flags = HMM_PFN_REQ_FAULT; range->pfn_flags_mask = 0; -and calls hmm_range_fault() as described above. This will fill fault all pages -in the range with at least read permission. +and calls the HMM range fault helper as described above. This will fault +all pages in the range with at least read permission. Now let's say the driver wants to do the same except for one page in the range for which it wants to have write permission. Now driver set:: @@ -236,9 +268,9 @@ address == range->start + (index_of_write << PAGE_SHIFT) it will fault with write permission i.e., if the CPU pte does not have write permission set then HMM will call handle_mm_fault(). -After hmm_range_fault completes the flag bits are set to the current state of -the page tables, ie HMM_PFN_VALID | HMM_PFN_WRITE will be set if the page is -writable. +After the HMM range fault helper completes the flag bits are set to the +current state of the page tables, ie HMM_PFN_VALID | HMM_PFN_WRITE will be +set if the page is writable. Represent and manage device memory from core kernel point of view diff --git a/include/linux/hmm.h b/include/linux/hmm.h index db75ffc949a7..6f04e3932f5b 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -123,6 +123,8 @@ struct hmm_range { * Please see Documentation/mm/hmm.rst for how to use the range API. */ int hmm_range_fault(struct hmm_range *range); +int hmm_range_fault_unlocked_timeout(struct hmm_range *range, + unsigned long timeout); /* * HMM_RANGE_DEFAULT_TIMEOUT - default timeout (ms) when waiting for a range diff --git a/mm/hmm.c b/mm/hmm.c index bc9361a715fa..fc2e1cd0cb22 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -32,6 +32,7 @@ struct hmm_vma_walk { struct hmm_range *range; + bool *locked; unsigned long last; unsigned long end; unsigned int required_fault; @@ -44,6 +45,14 @@ struct hmm_vma_walk { */ #define HMM_FAULT_PENDING -EAGAIN +/* + * Internal sentinel returned by hmm_do_fault() when handle_mm_fault() + * completes a page fault with the mmap lock dropped. hmm_do_fault() sets + * *locked = false; the outer loop consumes the sentinel and never propagates + * it to the caller. + */ +#define HMM_FAULT_UNLOCKED -ENOLCK + enum { HMM_NEED_FAULT = 1 << 0, HMM_NEED_WRITE_FAULT = 1 << 1, @@ -73,9 +82,9 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end, * * Called by the walk callbacks when they discover that part of the range * needs a page fault. The callback records what to fault and returns - * HMM_FAULT_PENDING; the outer loop in hmm_range_fault() drops back out of - * walk_page_range() and invokes handle_mm_fault() from a context where no - * page-table or hugetlb_vma_lock is held. + * HMM_FAULT_PENDING; the outer loop in hmm_range_fault_locked() drops + * back out of walk_page_range() and invokes handle_mm_fault() from a context + * where no page-table or hugetlb_vma_lock is held. */ static int hmm_record_fault(unsigned long addr, unsigned long end, unsigned int required_fault, @@ -624,7 +633,7 @@ static const struct mm_walk_ops hmm_walk_ops = { /* * hmm_do_fault - fault in a range recorded by a walk callback * - * Called from the outer loop in hmm_range_fault() after a callback + * Called from the outer loop in hmm_range_fault_locked() after a callback * returned HMM_FAULT_PENDING. At this point we hold only mmap_lock; * the page-table spinlock and any hugetlb_vma_lock acquired by the walk * framework have already been released by the unwind. @@ -641,6 +650,9 @@ static int hmm_do_fault(struct mm_struct *mm, unsigned int fault_flags = FAULT_FLAG_REMOTE; struct vm_area_struct *vma; + if (hmm_vma_walk->locked) + fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; + vma = vma_lookup(mm, addr); if (!vma) return -EFAULT; @@ -651,37 +663,34 @@ static int hmm_do_fault(struct mm_struct *mm, fault_flags |= FAULT_FLAG_WRITE; } - for (; addr < end; addr += PAGE_SIZE) - if (handle_mm_fault(vma, addr, fault_flags, NULL) & - VM_FAULT_ERROR) - return -EFAULT; + for (; addr < end; addr += PAGE_SIZE) { + vm_fault_t ret; + + ret = handle_mm_fault(vma, addr, fault_flags, NULL); + + if (ret & (VM_FAULT_COMPLETED | VM_FAULT_RETRY)) { + *hmm_vma_walk->locked = false; + return HMM_FAULT_UNLOCKED; + } + + if (ret & VM_FAULT_ERROR) { + int err = vm_fault_to_errno(ret, 0); + + if (WARN_ON(!err)) + err = -EINVAL; + + return err; + } + } return -EBUSY; } -/** - * hmm_range_fault - try to fault some address in a virtual address range - * @range: argument structure - * - * Returns 0 on success or one of the following error codes: - * - * -EINVAL: Invalid arguments or mm or virtual address is in an invalid vma - * (e.g., device file vma). - * -ENOMEM: Out of memory. - * -EPERM: Invalid permission (e.g., asking for write and range is read - * only). - * -EBUSY: The range has been invalidated and the caller needs to wait for - * the invalidation to finish. - * -EFAULT: A page was requested to be valid and could not be made valid - * ie it has no backing VMA or it is illegal to access - * - * This is similar to get_user_pages(), except that it can read the page tables - * without mutating them (ie causing faults). - */ -int hmm_range_fault(struct hmm_range *range) +static int hmm_range_fault_locked(struct hmm_range *range, bool *locked) { struct hmm_vma_walk hmm_vma_walk = { .range = range, + .locked = locked, .last = range->start, }; struct mm_struct *mm = range->notifier->mm; @@ -704,8 +713,14 @@ int hmm_range_fault(struct hmm_range *range) * returns -EBUSY so the loop re-walks and picks up the * now-present entries. */ - if (ret == HMM_FAULT_PENDING) + if (ret == HMM_FAULT_PENDING) { ret = hmm_do_fault(mm, &hmm_vma_walk); + if (ret == HMM_FAULT_UNLOCKED) { + if (fatal_signal_pending(current)) + return -EINTR; + return -EBUSY; + } + } /* * When -EBUSY is returned the loop restarts with * hmm_vma_walk.last set to an address that has not been stored @@ -715,8 +730,100 @@ int hmm_range_fault(struct hmm_range *range) } while (ret == -EBUSY); return ret; } + +/** + * hmm_range_fault - try to fault some address in a virtual address range + * @range: argument structure + * + * Returns 0 on success or one of the following error codes: + * + * -EINVAL: Invalid arguments or mm or virtual address is in an invalid vma + * (e.g., device file vma). + * -ENOMEM: Out of memory. + * -EPERM: Invalid permission (e.g., asking for write and range is read + * only). + * -EBUSY: The range has been invalidated and the caller needs to wait for + * the invalidation to finish. + * -EFAULT: A page was requested to be valid and could not be made valid + * ie it has no backing VMA or it is illegal to access + * + * This is similar to get_user_pages(), except that it can read the page tables + * without mutating them (ie causing faults). + * + * The mmap lock must be held by the caller and will remain held on return. + * New users should prefer hmm_range_fault_unlocked_timeout() unless they + * specifically need to keep the mmap lock held across the call. This helper + * cannot support VMAs whose fault handlers need to drop the mmap lock. + */ +int hmm_range_fault(struct hmm_range *range) +{ + return hmm_range_fault_locked(range, NULL); +} EXPORT_SYMBOL(hmm_range_fault); +/** + * hmm_range_fault_unlocked_timeout - fault in a range with a retry timeout + * @range: argument structure + * @timeout: timeout in jiffies for internal -EBUSY retries, or 0 to retry + * indefinitely + * + * The caller must not hold the mmap lock. The function takes the mmap read + * lock internally and allows handle_mm_fault() to drop it during faults. If + * the mmap lock is dropped or the range is invalidated, the function refreshes + * range->notifier_seq and restarts the walk internally. + * + * Passing 0 for @timeout retries indefinitely. A non-zero @timeout is a caller + * policy limit for repeated mmu-notifier invalidation retries. HMM does not + * interrupt page fault handling when the timeout expires, but returns -EBUSY + * if the retry budget is exhausted before a stable range is obtained. + * + * Returns 0 on success or one of the error codes documented for + * hmm_range_fault(). -EINTR is returned if mmap_lock acquisition is + * interrupted or a fatal signal is pending during retry handling. + */ +int hmm_range_fault_unlocked_timeout(struct hmm_range *range, + unsigned long timeout) +{ + struct mm_struct *mm = range->notifier->mm; + unsigned long deadline = 0; + bool locked = false; + int ret; + + do { + if (fatal_signal_pending(current)) + return -EINTR; + + if (timeout) { + /* + * If the previous fault dropped mmap_lock, then the fault + * handler made progress. Restart the retry timeout in that + * case, but keep the existing deadline for ordinary -EBUSY + * retries. + */ + if (!locked) + deadline = jiffies + timeout; + + if (time_after(jiffies, deadline)) + return -EBUSY; + } + + range->notifier_seq = + mmu_interval_read_begin(range->notifier); + + ret = mmap_read_lock_killable(mm); + if (ret) + return ret; + + locked = true; + ret = hmm_range_fault_locked(range, &locked); + if (locked) + mmap_read_unlock(mm); + } while (ret == -EBUSY); + + return ret; +} +EXPORT_SYMBOL(hmm_range_fault_unlocked_timeout); + /** * hmm_dma_map_alloc - Allocate HMM map structure * @dev: device to allocate structure for
