Re: [RFC PATCH RESEND 00/28] per-VMA locks proposal

On Sun, Sep 11, 2022 at 2:35 AM Vlastimil Babka wrote: > > On 9/2/22 01:26, Suren Baghdasaryan wrote: > > On Thu, Sep 1, 2022 at 1:58 PM Kent Overstreet > > wrote: > >> > >> On Thu, Sep 01, 2022 at 10:34:48AM -0700, Suren Baghdasaryan wrote: > >> &

Re: [PATCH 41/41] mm: replace rw_semaphore with atomic_t in vma_lock

On Tue, Jan 10, 2023 at 12:04 AM Vlastimil Babka wrote: > > On 1/9/23 21:53, Suren Baghdasaryan wrote: > > rw_semaphore is a sizable structure of 40 bytes and consumes > > considerable space for each vm_area_struct. However vma_lock has > > two important specifics whic

Re: [PATCH 08/41] mm: introduce CONFIG_PER_VMA_LOCK

On Tue, Jan 10, 2023 at 4:39 PM Davidlohr Bueso wrote: > > On Mon, 09 Jan 2023, Suren Baghdasaryan wrote: > > >This configuration variable will be used to build the support for VMA > >locking during page fault handling. > > > >This is enabled by default on support

[PATCH 03/41] maple_tree: Fix freeing of nodes in rcu mode

Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 73 1 file changed, 62 insertions(+), 11 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index a748938ad2e9..a1

[PATCH 06/41] maple_tree: Add smp_rmb() to dead node detection

re") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 8066fb1e8ec9..80ca28b656d3 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -535

[PATCH 13/41] mm: introduce vma->vm_flags modifier functions

To keep vma locking correctness when vm_flags are modified, add modifier functions to be used whenever flags are updated. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 38 ++ include/linux/mm_types.h | 8 +++- 2 files changed, 45

[PATCH 14/41] mm: replace VM_LOCKED_CLEAR_MASK with VM_LOCKED_MASK

To simplify the usage of VM_LOCKED_CLEAR_MASK in clear_vm_flags(), replace it with VM_LOCKED_MASK bitmask and convert all users. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 4 ++-- kernel/fork.c | 2 +- mm/hugetlb.c | 4 ++-- mm/mlock.c | 6 +++--- mm/mmap.c

[PATCH 18/41] mm/khugepaged: write-lock VMA while collapsing a huge page

by a page fault handler without proper locking. Signed-off-by: Suren Baghdasaryan --- mm/khugepaged.c | 5 + 1 file changed, 5 insertions(+) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 5376246a3052..d8d0647f0c2c 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1032,6 +1032,7

[PATCH 00/41] Per-VMA locks

ple_tree: Fix write memory barrier of nodes once dead for RCU mode maple_tree: Add smp_rmb() to dead node detection mm: Enable maple tree RCU mode by default. Michel Lespinasse (1): mm: rcu safe VMA freeing Suren Baghdasaryan (32): mm: introduce CONFIG_PER_VMA_LOCK mm: move mmap_lock a

[PATCH 01/41] maple_tree: Be more cautious about dead nodes

am Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 53 +++- 1 file changed, 43 insertions(+), 10 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 26e2045d3cda..ff9f04e0150d 100644 --- a/lib/maple_tree.c +++ b/lib/ma

[PATCH 02/41] maple_tree: Detect dead nodes in mas_start()

t;) Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 4 1 file changed, 4 insertions(+) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index ff9f04e0150d..a748938ad2e9 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1359,11 +1359,15 @@ sta

[PATCH 05/41] maple_tree: Fix write memory barrier of nodes once dead for RCU mode

e RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 7 +-- tools/testing/radix-tree/maple.c | 16 2 files changed, 21 insert

[PATCH 04/41] maple_tree: remove extra smp_wmb() from mas_dead_leaves()

From: Liam Howlett The call to mte_set_dead_node() before the smp_wmb() already calls smp_wmb() so this is not needed. This is an optimization for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off

[PATCH 07/41] mm: Enable maple tree RCU mode by default.

to ensure the nodes remain valid for readers. Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 3 ++- kernel/fork.c| 3 +++ mm/mmap.c| 4 +++- 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/include/linux/mm

[PATCH 11/41] mm: export dump_mm()

. Signed-off-by: Suren Baghdasaryan --- mm/debug.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/debug.c b/mm/debug.c index 7f8e5f744e42..b6e9e53469d1 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -215,6 +215,7 @@ void dump_mm(const struct mm_struct *mm) mm->def_fl

[PATCH 10/41] mm: move mmap_lock assert function definitions

Move mmap_lock assert function definitions up so that they can be used by other mmap_lock routines. Signed-off-by: Suren Baghdasaryan --- include/linux/mmap_lock.h | 24 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/mmap_lock.h b/include

[PATCH 17/41] mm/mmap: move VMA locking before anon_vma_lock_write call

Move VMA flag modification (which now implies VMA locking) before anon_vma_lock_write to match the locking order of page fault handler. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index fa994ae903d9

[PATCH 19/41] mm/mmap: write-lock VMAs before merging, splitting or expanding them

during COW operation. Write-lock all VMAs which might be affected by a merge or split operation before making decision how such operations should be performed. Not sure if expansion really needs this, just being paranoid. Otherwise mmap_region and vm_brk_flags might not locking. Signed-off-by:

[PATCH 20/41] mm/mmap: write-lock VMAs in vma_adjust

vma_adjust modifies a VMA and possibly its neighbors. Write-lock them before making the modifications. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 17 - 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index f6ca4a87f9e2..1e2154137631

[PATCH 24/41] mm: conditionally write-lock VMA in free_pgtables

to free_pgtables and unmap_region for such scenario. Signed-off-by: Suren Baghdasaryan --- mm/internal.h | 2 +- mm/memory.c | 6 +- mm/mmap.c | 18 -- 3 files changed, 18 insertions(+), 8 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index bcf75a8b032d..5ea4ff1a70e7

[PATCH 25/41] mm/mmap: write-lock adjacent VMAs if they can grow into unmapped area

While unmapping VMAs, adjacent VMAs might be able to grow into the area being unmapped. In such cases write-lock adjacent VMAs to prevent this growth. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 8 +--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/mmap.c b/mm

[PATCH 08/41] mm: introduce CONFIG_PER_VMA_LOCK

code which needs modifications to handle faults under VMA lock. Signed-off-by: Suren Baghdasaryan --- mm/Kconfig | 13 + 1 file changed, 13 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index ff7b209dec05..0aeca3794972 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1183,6 +1183,19

[PATCH 09/41] mm: rcu safe VMA freeing

From: Michel Lespinasse This prepares for page faults handling under VMA lock, looking up VMAs under protection of an rcu read lock, instead of the usual mmap read lock. Signed-off-by: Michel Lespinasse Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 13 ++--- kernel

[PATCH 12/41] mm: add per-VMA lock and helper functions to control it

ine while waiting on the lock. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h| 80 +++ include/linux/mm_types.h | 8 include/linux/mmap_lock.h | 13 +++ kernel/fork.c | 4 ++ mm/init-mm.c | 3 ++ 5 file

[PATCH 15/41] mm: replace vma->vm_flags direct modifications with modifier calls

Replace direct modifications to vma->vm_flags with calls to modifier functions to be able to track flag changes and to keep vma locking correctness. Signed-off-by: Suren Baghdasaryan --- arch/arm/kernel/process.c | 2 +- arch/ia64/mm/ini

[PATCH 16/41] mm: replace vma->vm_flags indirect modification in ksm_madvise

Replace indirect modifications to vma->vm_flags with calls to modifier functions to be able to track flag changes and to keep vma locking correctness. Add a BUG_ON check in ksm_madvise() to catch indirect vm_flags modification attempts. Signed-off-by: Suren Baghdasaryan --- arch/powerpc/

[PATCH 21/41] mm/mmap: write-lock VMAs affected by VMA expansion

vma_expand changes VMA boundaries and might result in freeing an adjacent VMA. Write-lock affected VMAs to prevent concurrent page faults. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 1e2154137631..ff02cb51e7e7

[PATCH 22/41] mm/mremap: write-lock VMA while remapping it to a new address range

Write-lock VMA as locked before copying it and when copy_vma produces a new VMA. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/mmap.c | 1 + mm/mremap.c | 1 + 2 files changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index ff02cb51e7e7..da1908730828 100644

[PATCH 23/41] mm: write-lock VMAs before removing them from VMA tree

Write-locking VMAs before isolating them ensures that page fault handlers don't operate on isolated VMAs. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 2 ++ mm/nommu.c | 5 + 2 files changed, 7 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index da1908730828..be289e0b693b 100644

[PATCH 26/41] kernel/fork: assert no VMA readers during its destruction

Assert there are no holders of VMA lock for reading when it is about to be destroyed. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 8 kernel/fork.c | 2 ++ 2 files changed, 10 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 594e835bad9c

[PATCH 27/41] mm/mmap: prevent pagefault handler from racing with mmu_notifier registration

as in page fault handlers. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 30c7d1c5206e..a256deca0bc0 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3566,6 +3566,7 @@ static void vm_lock_mapping(struct mm_struct *mm

[PATCH 28/41] mm: introduce lock_vma_under_rcu to be used from arch-specific code

is acquired. VMA lock statistics are updated according to the results. For now only anonymous VMAs can be searched this way. In other cases the function returns NULL. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c| 51

[PATCH 30/41] mm: add FAULT_FLAG_VMA_LOCK flag

Add a new flag to distinguish page faults handled under protection of per-vma lock. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- include/linux/mm.h | 3 ++- include/linux/mm_types.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux

[PATCH 31/41] mm: prevent do_swap_page from handling page faults under VMA lock

Due to the possibility of do_swap_page dropping mmap_lock, abort fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/memory.c | 5 + 1 file changed, 5 insertions

[PATCH 32/41] mm: prevent userfaults to be handled under per-vma lock

Due to the possibility of handle_userfault dropping mmap_lock, avoid fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Suggested-by: Peter Xu --- mm/memory.c | 7 +++ 1 file changed, 7 insertions

[PATCH 33/41] mm: introduce per-VMA lock statistics

Add a new CONFIG_PER_VMA_LOCK_STATS config option to dump extra statistics about handling page fault under VMA lock. Signed-off-by: Suren Baghdasaryan --- include/linux/vm_event_item.h | 6 ++ include/linux/vmstat.h| 6 ++ mm/Kconfig.debug | 8 mm/vmstat.c

[PATCH 36/41] powerc/mm: try VMA lock-based page fault handling first

From: Laurent Dufour Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Copied from "x86/mm: try VMA lock-based page fault handling first" Signed-off-by: Laurent Dufour Signed-off-by: Suren Baghdasaryan --- arch/

[PATCH 29/41] mm: fall back to mmap_lock if vma->anon_vma is not yet set

ed-off-by: Suren Baghdasaryan --- mm/memory.c | 4 1 file changed, 4 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index a658e26d965d..2560524ad7f4 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5264,6 +5264,10 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_stru

[PATCH 35/41] arm64/mm: try VMA lock-based page fault handling first

Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/arm64/Kconfig| 1 + arch/arm64/mm/fault.c | 36 2 files changed, 37 insertions(+) diff

[PATCH 34/41] x86/mm: try VMA lock-based page fault handling first

Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/x86/Kconfig| 1 + arch/x86/mm/fault.c | 36 2 files changed, 37 insertions(+) diff --git

[PATCH 37/41] mm: introduce mod_vm_flags_nolock

. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 8 1 file changed, 8 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 2e3be1d45371..7d436a5027cc 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -743,6 +743,14 @@ void clear_vm_flags(struct

[PATCH 39/41] kernel/fork: throttle call_rcu() calls in vm_area_free

-by: Suren Baghdasaryan --- include/linux/mm.h | 1 + include/linux/mm_types.h | 19 +-- kernel/fork.c| 68 +++- mm/init-mm.c | 3 ++ mm/mmap.c| 1 + 5 files changed, 82 insertions(+), 10 deletions(-) diff

[PATCH 38/41] mm: avoid assertion in untrack_pfn

untrack_pfn can be called after VMA was isolated and mmap_lock downgraded. An attempt to lock affected VMA would cause an assertion, therefore use mod_vm_flags_nolock in such situations. Signed-off-by: Suren Baghdasaryan --- arch/x86/mm/pat/memtype.c | 10 +++--- include/linux/mm.h

[PATCH 41/41] mm: replace rw_semaphore with atomic_t in vma_lock

() implements an overflow check and if that occurs it restors the previous value and exits with a failure to lock. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 37 + include/linux/mm_types.h | 10 -- kernel/fork.c| 6 +++--- mm

[PATCH 40/41] mm: separate vma->lock from vm_area_struct

A number of drivers allocate a pseudo VMA on the stack but they never use the VMA's lock, therefore it does not need to be allocated. The future drivers which might need the VMA lock should use vm_area_alloc()/vm_area_free() to allocate it. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h

Re: [PATCH 13/41] mm: introduce vma->vm_flags modifier functions

On Wed, Jan 11, 2023 at 12:19 PM Davidlohr Bueso wrote: > > On Wed, 11 Jan 2023, Suren Baghdasaryan wrote: > > >On Wed, Jan 11, 2023 at 8:13 AM Davidlohr Bueso wrote: > >> > >> On Mon, 09 Jan 2023, Suren Baghdasaryan wrote: > >> > >> >To k

Re: [PATCH 08/41] mm: introduce CONFIG_PER_VMA_LOCK

On Wed, Jan 11, 2023 at 9:37 AM Michal Hocko wrote: > > On Wed 11-01-23 09:04:41, Suren Baghdasaryan wrote: > > On Wed, Jan 11, 2023 at 8:44 AM Michal Hocko wrote: > > > > > > On Wed 11-01-23 08:28:49, Suren Baghdasaryan wrote: > > > [...] > > > &

Re: [PATCH 08/41] mm: introduce CONFIG_PER_VMA_LOCK

On Wed, Jan 11, 2023 at 8:44 AM Michal Hocko wrote: > > On Wed 11-01-23 08:28:49, Suren Baghdasaryan wrote: > [...] > > Anyhow. Sounds like the overhead of the current design is small enough > > to remove CONFIG_PER_VMA_LOCK and let it depend only on architecture > >

Re: [PATCH 08/41] mm: introduce CONFIG_PER_VMA_LOCK

On Wed, Jan 11, 2023 at 10:03 AM Michal Hocko wrote: > > On Wed 11-01-23 09:49:08, Suren Baghdasaryan wrote: > > On Wed, Jan 11, 2023 at 9:37 AM Michal Hocko wrote: > > > > > > On Wed 11-01-23 09:04:41, Suren Baghdasaryan wrote: > > > > On Wed, J

Re: [PATCH 13/41] mm: introduce vma->vm_flags modifier functions

On Wed, Jan 11, 2023 at 8:13 AM Davidlohr Bueso wrote: > > On Mon, 09 Jan 2023, Suren Baghdasaryan wrote: > > >To keep vma locking correctness when vm_flags are modified, add modifier > >functions to be used whenever flags are updated. > > How about moving this patch a

Re: [PATCH 08/41] mm: introduce CONFIG_PER_VMA_LOCK

On Wed, Jan 11, 2023 at 2:03 AM David Laight wrote: > > From: Ingo Molnar > > Sent: 11 January 2023 09:54 > > > > * Michal Hocko wrote: > > > > > On Tue 10-01-23 16:44:42, Suren Baghdasaryan wrote: > > > > On Tue, Jan

Re: [PATCH 41/41] mm: replace rw_semaphore with atomic_t in vma_lock

On Mon, Jan 16, 2023 at 3:15 AM Hyeonggon Yoo <42.hye...@gmail.com> wrote: > > On Mon, Jan 09, 2023 at 12:53:36PM -0800, Suren Baghdasaryan wrote: > > diff --git a/include/linux/mm.h b/include/linux/mm.h > > index d40bf8a5e19e..294dd44b2198 100644 > > --- a/includ

Re: [PATCH 41/41] mm: replace rw_semaphore with atomic_t in vma_lock

On Mon, Jan 16, 2023 at 8:14 PM Matthew Wilcox wrote: > > On Mon, Jan 16, 2023 at 11:14:38AM +, Hyeonggon Yoo wrote: > > > @@ -643,20 +647,28 @@ static inline void vma_write_lock(struct > > > vm_area_struct *vma) > > > static inline bool vma_read_trylock(struct vm_area_struct *vma) > > > {

Re: [PATCH 41/41] mm: replace rw_semaphore with atomic_t in vma_lock

On Mon, Jan 16, 2023 at 9:46 PM Matthew Wilcox wrote: > > On Mon, Jan 16, 2023 at 08:34:36PM -0800, Suren Baghdasaryan wrote: > > On Mon, Jan 16, 2023 at 8:14 PM Matthew Wilcox wrote: > > > > > > On Mon, Jan 16, 2023 at 11:14:38AM +, Hyeonggon Yoo wrote

Re: [PATCH 41/41] mm: replace rw_semaphore with atomic_t in vma_lock

On Tue, Jan 17, 2023 at 10:12 AM Jann Horn wrote: > > On Mon, Jan 9, 2023 at 9:55 PM Suren Baghdasaryan wrote: > > rw_semaphore is a sizable structure of 40 bytes and consumes > > considerable space for each vm_area_struct. However vma_lock has > > two important spe

Re: [PATCH 18/41] mm/khugepaged: write-lock VMA while collapsing a huge page

On Tue, Jan 17, 2023 at 12:28 PM Jann Horn wrote: > > On Tue, Jan 17, 2023 at 4:25 PM Michal Hocko wrote: > > On Mon 09-01-23 12:53:13, Suren Baghdasaryan wrote: > > > Protect VMA from concurrent page fault handler while collapsing a huge > > > page. Page fault han

Re: [PATCH 12/41] mm: add per-VMA lock and helper functions to control it

On Tue, Jan 17, 2023 at 7:12 AM Michal Hocko wrote: > > On Tue 17-01-23 16:04:26, Michal Hocko wrote: > > On Mon 09-01-23 12:53:07, Suren Baghdasaryan wrote: > > > Introduce a per-VMA rw_semaphore to be used during page fault handling > > > instead of mmap_lock

Re: [PATCH 28/41] mm: introduce lock_vma_under_rcu to be used from arch-specific code

On Tue, Jan 17, 2023 at 7:47 AM Michal Hocko wrote: > > On Mon 09-01-23 12:53:23, Suren Baghdasaryan wrote: > > Introduce lock_vma_under_rcu function to lookup and lock a VMA during > > page fault handling. When VMA is not found, can't be locked or changes > > after be

Re: [PATCH 41/41] mm: replace rw_semaphore with atomic_t in vma_lock

On Tue, Jan 17, 2023 at 10:31 AM Matthew Wilcox wrote: > > On Tue, Jan 17, 2023 at 10:26:32AM -0800, Suren Baghdasaryan wrote: > > On Tue, Jan 17, 2023 at 10:12 AM Jann Horn wrote: > > > > > > On Mon, Jan 9, 2023 at 9:55 PM Suren Baghdasaryan > > > w

Re: [PATCH 41/41] mm: replace rw_semaphore with atomic_t in vma_lock

On Tue, Jan 17, 2023 at 10:47 AM Matthew Wilcox wrote: > > On Tue, Jan 17, 2023 at 10:36:42AM -0800, Suren Baghdasaryan wrote: > > On Tue, Jan 17, 2023 at 10:31 AM Matthew Wilcox wrote: > > > > > > On Tue, Jan 17, 2023 at 10:26:32AM -0800, Suren Baghdasaryan wrote

Re: [PATCH 12/41] mm: add per-VMA lock and helper functions to control it

On Tue, Jan 17, 2023 at 1:46 PM Jann Horn wrote: > > On Tue, Jan 17, 2023 at 10:28 PM Suren Baghdasaryan wrote: > > On Tue, Jan 17, 2023 at 10:03 AM Jann Horn wrote: > > > > > > +locking maintainers > > > > Thanks! I'll CC the locking maintainers in the

Re: [PATCH 41/41] mm: replace rw_semaphore with atomic_t in vma_lock

On Tue, Jan 17, 2023 at 10:23 AM Matthew Wilcox wrote: > > On Mon, Jan 16, 2023 at 09:58:35PM -0800, Suren Baghdasaryan wrote: > > On Mon, Jan 16, 2023 at 9:46 PM Matthew Wilcox wrote: > > > > > > On Mon, Jan 16, 2023 at 08:34:36PM -0800, Suren Baghdasaryan wrote

Re: [PATCH 41/41] mm: replace rw_semaphore with atomic_t in vma_lock

On Tue, Jan 17, 2023 at 11:00 AM Jann Horn wrote: > > On Tue, Jan 17, 2023 at 7:55 PM Suren Baghdasaryan wrote: > > On Tue, Jan 17, 2023 at 10:47 AM Matthew Wilcox wrote: > > > > > > On Tue, Jan 17, 2023 at 10:36:42AM -0800, Suren Baghdasaryan wrote: > >

Re: [PATCH 41/41] mm: replace rw_semaphore with atomic_t in vma_lock

On Tue, Jan 17, 2023 at 12:31 PM Michal Hocko wrote: > > On Tue 17-01-23 10:28:40, Suren Baghdasaryan wrote: > [...] > > > Then yes, that's a starvable lock. Preventing starvation on the mmap > > > sem was the original motivation for making rwsems non-starvable, so &g

Re: [PATCH 12/41] mm: add per-VMA lock and helper functions to control it

On Tue, Jan 17, 2023 at 7:04 AM Michal Hocko wrote: > > On Mon 09-01-23 12:53:07, Suren Baghdasaryan wrote: > > Introduce a per-VMA rw_semaphore to be used during page fault handling > > instead of mmap_lock. Because there are cases when multiple VMAs need > > to be exclu

Re: [PATCH 40/41] mm: separate vma->lock from vm_area_struct

On Tue, Jan 17, 2023 at 10:34 AM Jann Horn wrote: > > On Mon, Jan 9, 2023 at 9:55 PM Suren Baghdasaryan wrote: > > vma->lock being part of the vm_area_struct causes performance regression > > during page faults because during contention its count and owner fields >

Re: [PATCH 39/41] kernel/fork: throttle call_rcu() calls in vm_area_free

On Tue, Jan 17, 2023 at 7:57 AM Michal Hocko wrote: > > On Mon 09-01-23 12:53:34, Suren Baghdasaryan wrote: > > call_rcu() can take a long time when callback offloading is enabled. > > Its use in the vm_area_free can cause regressions in the exit path when > > multi

Re: [PATCH 12/41] mm: add per-VMA lock and helper functions to control it

On Tue, Jan 17, 2023 at 7:07 AM Michal Hocko wrote: > > On Mon 09-01-23 12:53:07, Suren Baghdasaryan wrote: > > diff --git a/kernel/fork.c b/kernel/fork.c > > index 5986817f393c..c026d75108b3 100644 > > --- a/kernel/fork.c > > +++ b/kernel/fork.c > > @@ -4

Re: [PATCH 41/41] mm: replace rw_semaphore with atomic_t in vma_lock

On Tue, Jan 17, 2023 at 10:36 AM Jann Horn wrote: > > On Tue, Jan 17, 2023 at 7:31 PM Matthew Wilcox wrote: > > > > On Tue, Jan 17, 2023 at 10:26:32AM -0800, Suren Baghdasaryan wrote: > > > On Tue, Jan 17, 2023 at 10:12 AM Jann Horn wrote: > > > > >

Re: [PATCH 12/41] mm: add per-VMA lock and helper functions to control it

On Tue, Jan 17, 2023 at 1:54 PM Matthew Wilcox wrote: > > On Tue, Jan 17, 2023 at 01:21:47PM -0800, Suren Baghdasaryan wrote: > > On Tue, Jan 17, 2023 at 7:12 AM Michal Hocko wrote: > > > > > > On Tue 17-01-23 16:04:26, Michal Hocko wrote: > > > > On

Re: [PATCH 32/41] mm: prevent userfaults to be handled under per-vma lock

On Tue, Jan 17, 2023 at 12:36 PM Jann Horn wrote: > > On Tue, Jan 17, 2023 at 8:51 PM Jann Horn wrote: > > On Mon, Jan 9, 2023 at 9:55 PM Suren Baghdasaryan wrote: > > > Due to the possibility of handle_userfault dropping mmap_lock, avoid fault > > > handling un

Re: [PATCH 12/41] mm: add per-VMA lock and helper functions to control it

On Tue, Jan 17, 2023 at 10:03 AM Jann Horn wrote: > > +locking maintainers Thanks! I'll CC the locking maintainers in the next posting. > > On Mon, Jan 9, 2023 at 9:54 PM Suren Baghdasaryan wrote: > > Introduce a per-VMA rw_semaphore to be used during page fault ha

Re: [PATCH 17/41] mm/mmap: move VMA locking before anon_vma_lock_write call

On Tue, Jan 17, 2023 at 7:16 AM Michal Hocko wrote: > > On Mon 09-01-23 12:53:12, Suren Baghdasaryan wrote: > > Move VMA flag modification (which now implies VMA locking) before > > anon_vma_lock_write to match the locking order of page fault handler. > > Does this c

Re: [PATCH 26/41] kernel/fork: assert no VMA readers during its destruction

On Tue, Jan 17, 2023 at 7:42 AM 'Michal Hocko' via kernel-team wrote: > > On Mon 09-01-23 12:53:21, Suren Baghdasaryan wrote: > > Assert there are no holders of VMA lock for reading when it is about to be > > destroyed. > > > > Signed-off-by: Suren Baghdasaryan &

Re: [PATCH 13/41] mm: introduce vma->vm_flags modifier functions

On Tue, Jan 17, 2023 at 7:15 AM 'Michal Hocko' via kernel-team wrote: > > On Tue 17-01-23 16:09:03, Michal Hocko wrote: > > On Mon 09-01-23 12:53:08, Suren Baghdasaryan wrote: > > > To keep vma locking correctness when vm_flags are modified, add modifier > > > funct

Re: [PATCH 09/41] mm: rcu safe VMA freeing

On Tue, Jan 17, 2023 at 6:25 AM Michal Hocko wrote: > > On Mon 09-01-23 12:53:04, Suren Baghdasaryan wrote: > [...] > > void vm_area_free(struct vm_area_struct *vma) > > { > > free_anon_vma_name(vma); > > +#ifdef CONFIG_PER_VMA_LOCK > > + call_r

Re: [PATCH 39/41] kernel/fork: throttle call_rcu() calls in vm_area_free

On Thu, Jan 19, 2023 at 4:59 AM Michal Hocko wrote: > > On Mon 09-01-23 12:53:34, Suren Baghdasaryan wrote: > > call_rcu() can take a long time when callback offloading is enabled. > > Its use in the vm_area_free can cause regressions in the exit path when > > multi

Re: [PATCH 17/41] mm/mmap: move VMA locking before anon_vma_lock_write call

On Thu, Jan 19, 2023 at 1:31 AM Michal Hocko wrote: > > On Wed 18-01-23 13:48:13, Suren Baghdasaryan wrote: > > On Wed, Jan 18, 2023 at 1:33 PM Michal Hocko wrote: > [...] > > > So it will become: > > > Move VMA flag modification (which now implies VMA locking) b

Re: [PATCH 39/41] kernel/fork: throttle call_rcu() calls in vm_area_free

On Thu, Jan 19, 2023 at 11:20 AM Paul E. McKenney wrote: > > On Thu, Jan 19, 2023 at 10:52:03AM -0800, Suren Baghdasaryan wrote: > > On Thu, Jan 19, 2023 at 4:59 AM Michal Hocko wrote: > > > > > > On Mon 09-01-23 12:53:34, Suren Baghdasaryan wrote: > > >

[PATCH v2 4/6] mm: replace vma->vm_flags indirect modification in ksm_madvise

Replace indirect modifications to vma->vm_flags with calls to modifier functions to be able to track flag changes and to keep vma locking correctness. Add a BUG_ON check in ksm_madvise() to catch indirect vm_flags modification attempts. Signed-off-by: Suren Baghdasaryan --- arch/powerpc/

[PATCH v2 1/6] mm: introduce vma->vm_flags modifier functions

are updated. This way we can better check and control correct locking behavior during these updates. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 37 + include/linux/mm_types.h | 8 +++- 2 files changed, 44 insertions(+), 1 deletion

[PATCH v2 3/6] mm: replace vma->vm_flags direct modifications with modifier calls

Replace direct modifications to vma->vm_flags with calls to modifier functions to be able to track flag changes and to keep vma locking correctness. Signed-off-by: Suren Baghdasaryan --- arch/arm/kernel/process.c | 2 +- arch/ia64/mm/ini

[PATCH v2 2/6] mm: replace VM_LOCKED_CLEAR_MASK with VM_LOCKED_MASK

To simplify the usage of VM_LOCKED_CLEAR_MASK in clear_vm_flags(), replace it with VM_LOCKED_MASK bitmask and convert all users. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 4 ++-- kernel/fork.c | 2 +- mm/hugetlb.c | 4 ++-- mm/mlock.c | 6 +++--- mm/mmap.c

[PATCH v2 0/6] introduce vm_flags modifier functions

://lore.kernel.org/all/20230109205336.3665937-1-sur...@google.com/ The patchset applies cleanly over mm-unstable branch of mm tree. My apologies for an extremely large distribution list. The patch touches lots of files and many are in arch/ and drivers/. Suren Baghdasaryan (6): mm: introduce

[PATCH v2 6/6] mm: export dump_mm()

mmap_assert_write_locked() is used in vm_flags modifiers. Because mmap_assert_write_locked() uses dump_mm() and vm_flags are sometimes modified from from inside a module, it's necessary to export dump_mm() function. Signed-off-by: Suren Baghdasaryan --- mm/debug.c | 1 + 1 file changed, 1

[PATCH v2 5/6] mm: introduce mod_vm_flags_nolock and use it in untrack_pfn

mod_vm_flags_nolock for flags modification and to avoid assertion. Signed-off-by: Suren Baghdasaryan --- arch/x86/mm/pat/memtype.c | 10 +++--- include/linux/mm.h| 12 +--- include/linux/pgtable.h | 5 +++-- mm/memory.c | 13 +++-- mm/memremap.c | 4

Re: [PATCH 39/41] kernel/fork: throttle call_rcu() calls in vm_area_free

On Fri, Jan 20, 2023 at 9:32 AM Matthew Wilcox wrote: > > On Fri, Jan 20, 2023 at 09:17:46AM -0800, Suren Baghdasaryan wrote: > > On Fri, Jan 20, 2023 at 9:08 AM Liam R. Howlett > > wrote: > > > > > > * Matthew Wilcox [230120 11:50]: > > > >

Re: [PATCH 39/41] kernel/fork: throttle call_rcu() calls in vm_area_free

On Fri, Jan 20, 2023 at 9:21 AM Paul E. McKenney wrote: > > On Fri, Jan 20, 2023 at 04:49:42PM +, Matthew Wilcox wrote: > > On Fri, Jan 20, 2023 at 08:45:21AM -0800, Suren Baghdasaryan wrote: > > > On Fri, Jan 20, 2023 at 8:20 AM Suren Baghdasaryan > > > wrot

Re: [PATCH v2 1/6] mm: introduce vma->vm_flags modifier functions

On Wed, Jan 25, 2023 at 10:33 AM Matthew Wilcox wrote: > > On Wed, Jan 25, 2023 at 12:38:46AM -0800, Suren Baghdasaryan wrote: > > +/* Use when VMA is not part of the VMA tree and needs no locking */ > > +static inline void init_vm_flags(struct v

Re: [PATCH v3 2/7] mm: introduce vma->vm_flags wrapper functions

On Wed, Jan 25, 2023 at 4:24 PM Andrew Morton wrote: > > On Wed, 25 Jan 2023 15:35:49 -0800 Suren Baghdasaryan > wrote: > > > vm_flags are among VMA attributes which affect decisions like VMA merging > > and splitting. Therefore all vm_flags modifications are pe

[PATCH v3 0/7] introduce vm_flags modifier functions

/all/20230109205336.3665937-1-sur...@google.com/ [2] https://lore.kernel.org/lkml/20230125083851.27759-1-sur...@google.com/ Suren Baghdasaryan (7): kernel/fork: convert vma assignment to a memcpy mm: introduce vma->vm_flags wrapper functions mm: replace VM_LOCKED_CLEAR_MASK with VM_LOCKED_M

Re: [PATCH v3 1/7] kernel/fork: convert vma assignment to a memcpy

On Wed, Jan 25, 2023 at 4:22 PM Andrew Morton wrote: > > On Wed, 25 Jan 2023 15:35:48 -0800 Suren Baghdasaryan > wrote: > > > Convert vma assignment in vm_area_dup() to a memcpy() to prevent compiler > > errors when we add a const modifier to vma->vm_flags. >

[PATCH v3 4/7] mm: replace vma->vm_flags direct modifications with modifier calls

Replace direct modifications to vma->vm_flags with calls to modifier functions to be able to track flag changes and to keep vma locking correctness. Signed-off-by: Suren Baghdasaryan Acked-by: Michal Hocko --- arch/arm/kernel/process.c | 2 +- arch/ia64/mm/ini

[PATCH v3 3/7] mm: replace VM_LOCKED_CLEAR_MASK with VM_LOCKED_MASK

To simplify the usage of VM_LOCKED_CLEAR_MASK in clear_vm_flags(), replace it with VM_LOCKED_MASK bitmask and convert all users. Signed-off-by: Suren Baghdasaryan Acked-by: Michal Hocko --- include/linux/mm.h | 4 ++-- kernel/fork.c | 2 +- mm/hugetlb.c | 4 ++-- mm/mlock.c

Re: [PATCH v2 3/6] mm: replace vma->vm_flags direct modifications with modifier calls

On Wed, Jan 25, 2023 at 1:30 AM 'Michal Hocko' via kernel-team wrote: > > On Wed 25-01-23 00:38:48, Suren Baghdasaryan wrote: > > Replace direct modifications to vma->vm_flags with calls to modifier > > functions to be able to track flag changes and to keep vma lo

Re: [PATCH v2 1/6] mm: introduce vma->vm_flags modifier functions

On Wed, Jan 25, 2023 at 1:10 AM Peter Zijlstra wrote: > > On Wed, Jan 25, 2023 at 12:38:46AM -0800, Suren Baghdasaryan wrote: > > > diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h > > index 2d6d790d9bed..6c7c70bf50dd 100644 > > --- a/include/linux/m

Re: [PATCH v2 1/6] mm: introduce vma->vm_flags modifier functions

On Wed, Jan 25, 2023 at 10:37 AM Matthew Wilcox wrote: > > On Wed, Jan 25, 2023 at 08:49:50AM -0800, Suren Baghdasaryan wrote: > > On Wed, Jan 25, 2023 at 1:10 AM Peter Zijlstra wrote: > > > > + /* > > > > + * Flags, see mm.h. > > &g

[PATCH v3 7/7] mm: export dump_mm()

mmap_assert_write_locked() is used in vm_flags modifiers. Because mmap_assert_write_locked() uses dump_mm() and vm_flags are sometimes modified from inside a module, it's necessary to export dump_mm() function. Signed-off-by: Suren Baghdasaryan Acked-by: Michal Hocko --- mm/debug.c | 1 + 1

Re: [PATCH v3 2/7] mm: introduce vma->vm_flags wrapper functions

On Wed, Jan 25, 2023 at 4:28 PM Andrew Morton wrote: > > On Wed, 25 Jan 2023 15:35:49 -0800 Suren Baghdasaryan > wrote: > > > --- a/include/linux/mm_types.h > > +++ b/include/linux/mm_types.h > > @@ -491,7 +491,15 @@ struct vm_area_struct { > >

Re: [PATCH v2 5/6] mm: introduce mod_vm_flags_nolock and use it in untrack_pfn

On Wed, Jan 25, 2023 at 1:42 AM Michal Hocko wrote: > > On Wed 25-01-23 00:38:50, Suren Baghdasaryan wrote: > > In cases when VMA flags are modified after VMA was isolated and mmap_lock > > was downgraded, flags modifications would result in an assertion because > > mma

  1   2   3   4   >