[PATCH v3 07/35] maple_tree: Add RCU lock checking to rcu callback functions

and parent pointers. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Reported-by: Suren Baghdasaryan Signed-off-by: Liam R. Howlett --- lib/maple_tree.c | 188 --- 1 file changed, 96 insertions(+), 92 deletions(-) diff --git a/li

[PATCH v3 06/35] maple_tree: Add smp_rmb() to dead node detection

re") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 6b6eddadd9d2..8ad2d1669fad 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -539

[PATCH v3 15/35] mm/khugepaged: write-lock VMA while collapsing a huge page

a fourth lock under which page tables can be traversed, and so khugepaged must also lock out that one. Signed-off-by: Suren Baghdasaryan --- mm/khugepaged.c | 5 + mm/rmap.c | 31 --- 2 files changed, 21 insertions(+), 15 deletions(-) diff --git a/mm

[PATCH v3 16/35] mm/mmap: write-lock VMAs before merging, splitting or expanding them

during COW operation. Write-lock all VMAs which might be affected by a merge or split operation before making decision how such operations should be performed. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 23 --- 1 file changed, 20 insertions(+), 3 deletions(-) diff --gi

[PATCH v3 28/35] mm: prevent do_swap_page from handling page faults under VMA lock

Due to the possibility of do_swap_page dropping mmap_lock, abort fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/memory.c | 5 + 1 file changed, 5 insertions

[PATCH v3 29/35] mm: prevent userfaults to be handled under per-vma lock

Due to the possibility of handle_userfault dropping mmap_lock, avoid fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Suggested-by: Peter Xu --- mm/memory.c | 9 + 1 file changed, 9

[PATCH v3 11/35] mm: move mmap_lock assert function definitions

Move mmap_lock assert function definitions up so that they can be used by other mmap_lock routines. Signed-off-by: Suren Baghdasaryan --- include/linux/mmap_lock.h | 24 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/mmap_lock.h b/include

[PATCH v3 10/35] mm: rcu safe VMA freeing

From: Michel Lespinasse This prepares for page faults handling under VMA lock, looking up VMAs under protection of an rcu read lock, instead of the usual mmap read lock. Signed-off-by: Michel Lespinasse Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 13 ++--- kernel

[PATCH v3 22/35] kernel/fork: assert no VMA readers during its destruction

Assert there are no holders of VMA lock for reading when it is about to be destroyed. Signed-off-by: Suren Baghdasaryan --- kernel/fork.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/fork.c b/kernel/fork.c index 9141427a98b2..a08cc0e2bfde 100644 --- a/kernel/fork.c +++ b/kernel

[PATCH v3 32/35] arm64/mm: try VMA lock-based page fault handling first

Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/arm64/Kconfig| 1 + arch/arm64/mm/fault.c | 36 2 files changed, 37 insertions(+) diff

[PATCH v3 33/35] powerc/mm: try VMA lock-based page fault handling first

From: Laurent Dufour Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Copied from "x86/mm: try VMA lock-based page fault handling first" Signed-off-by: Laurent Dufour Signed-off-by: Suren Baghdasaryan --- arch/

[PATCH v3 19/35] mm: write-lock VMAs before removing them from VMA tree

Write-locking VMAs before isolating them ensures that page fault handlers don't operate on isolated VMAs. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 1 + mm/nommu.c | 5 + 2 files changed, 6 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 0eaa3d1a6cd1..aaa359a147b2 100644

[PATCH v3 34/35] mm/mmap: free vm_area_struct without call_rcu in exit_mmap

caused by it. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 2 ++ kernel/fork.c | 2 +- mm/mmap.c | 11 +++ 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 3c9167529417..cedef02dfd2b 100644

[PATCH v3 03/35] maple_tree: Fix freeing of nodes in rcu mode

Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 73 1 file changed, 62 insertions(+), 11 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 089cd76ec379..44

[PATCH v3 02/35] maple_tree: Detect dead nodes in mas_start()

t;) Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 4 1 file changed, 4 insertions(+) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index cc356b8369ad..089cd76ec379 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1360,12 +1360,16 @@ sta

[PATCH v3 13/35] mm: mark VMA as being written when changing vm_flags

Updates to vm_flags have to be done with VMA marked as being written for preventing concurrent page faults or other modifications. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include

[PATCH v3 14/35] mm/mmap: move VMA locking before vma_adjust_trans_huge call

vma_adjust_trans_huge() modifies the VMA and such modifications should be done after VMA is marked as being written. Therefore move VMA flag modifications before vma_adjust_trans_huge() so that VMA is marked before all these modifications. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 3

[PATCH v3 24/35] mm: introduce vma detached flag

Per-vma locking mechanism will search for VMA under RCU protection and then after locking it, has to ensure it was not removed from the VMA tree after we found it. To make this check efficient, introduce a vma->detached flag to mark VMAs which were removed from the VMA tree. Signed-off-by: Su

[PATCH v3 23/35] mm/mmap: prevent pagefault handler from racing with mmu_notifier registration

order as in page fault handlers. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 9 + 1 file changed, 9 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 00f8c5798936..801608726be8 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3501,6 +3501,7 @@ static void vm_lock_mapping(struct

[PATCH v3 00/35] Per-VMA locks

aple_tree: Add smp_rmb() to dead node detection maple_tree: Add RCU lock checking to rcu callback functions mm: Enable maple tree RCU mode by default. Michel Lespinasse (1): mm: rcu safe VMA freeing Suren Baghdasaryan (25): mm: introduce CONFIG_PER_VMA_LOCK mm: move mmap_lock assert function

[PATCH v3 01/35] maple_tree: Be more cautious about dead nodes

am Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 52 +++- 1 file changed, 43 insertions(+), 9 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 646297cae5d1..cc356b8369ad 100644 --- a/lib/maple_tree.c +++ b/lib/ma

[PATCH v3 12/35] mm: add per-VMA lock and helper functions to control it

ive mmap_lock is released by incrementing mm sequence counter (mm_lock_seq). - write downgrade - if the mmap_lock is downgraded to the read lock, all vma write locks are released as well (effectivelly same as write unlock). Signed-off-by: Suren Baghdasaryan --- include/linux/m

[PATCH v3 20/35] mm: conditionally write-lock VMA in free_pgtables

to free_pgtables for such scenario. Signed-off-by: Suren Baghdasaryan --- mm/internal.h | 2 +- mm/memory.c | 6 +- mm/mmap.c | 5 +++-- 3 files changed, 9 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index fc01fd092ea5..400c302fbf47 100644 --- a/mm/internal.h +++ b/mm

[PATCH v3 21/35] mm/mmap: write-lock adjacent VMAs if they can grow into unmapped area

While unmapping VMAs, adjacent VMAs might be able to grow into the area being unmapped. In such cases write-lock adjacent VMAs to prevent this growth. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 8 +--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/mmap.c b/mm

Re: [PATCH v3 21/35] mm/mmap: write-lock adjacent VMAs if they can grow into unmapped area

On Fri, Feb 17, 2023 at 6:51 AM Liam R. Howlett wrote: > > * Suren Baghdasaryan [230216 14:36]: > > On Thu, Feb 16, 2023 at 7:34 AM Liam R. Howlett > > wrote: > > > > > > > > > First, sorry I didn't see this before v3.. >

Re: [PATCH v3 26/35] mm: fall back to mmap_lock if vma->anon_vma is not yet set

On Fri, Feb 17, 2023 at 8:05 AM Matthew Wilcox wrote: > > On Thu, Feb 16, 2023 at 06:14:59PM -0800, Suren Baghdasaryan wrote: > > On Thu, Feb 16, 2023 at 11:43 AM Suren Baghdasaryan > > wrote: > > > > > > On Thu, Feb 16, 2023 at 7:44 AM Matthew Wilcox >

Re: [PATCH v3 26/35] mm: fall back to mmap_lock if vma->anon_vma is not yet set

On Fri, Feb 17, 2023 at 2:21 AM Hyeonggon Yoo <42.hye...@gmail.com> wrote: > > On Fri, Feb 17, 2023 at 11:15 AM Suren Baghdasaryan wrote: > > > > On Thu, Feb 16, 2023 at 11:43 AM Suren Baghdasaryan > > wrote: > > > > > > On Thu, Feb

Re: [PATCH v3 26/35] mm: fall back to mmap_lock if vma->anon_vma is not yet set

On Thu, Feb 16, 2023 at 7:44 AM Matthew Wilcox wrote: > > On Wed, Feb 15, 2023 at 09:17:41PM -0800, Suren Baghdasaryan wrote: > > When vma->anon_vma is not set, page fault handler will set it by either > > reusing anon_vma of an adjacent VMA if VMAs are compatible or by >

Re: [PATCH v3 21/35] mm/mmap: write-lock adjacent VMAs if they can grow into unmapped area

On Thu, Feb 16, 2023 at 7:34 AM Liam R. Howlett wrote: > > > First, sorry I didn't see this before v3.. Feedback at any time is highly appreciated! > > * Suren Baghdasaryan [230216 00:18]: > > While unmapping VMAs, adjacent VMAs might be able to grow into the ar

Re: [PATCH v3 26/35] mm: fall back to mmap_lock if vma->anon_vma is not yet set

On Thu, Feb 16, 2023 at 11:43 AM Suren Baghdasaryan wrote: > > On Thu, Feb 16, 2023 at 7:44 AM Matthew Wilcox wrote: > > > > On Wed, Feb 15, 2023 at 09:17:41PM -0800, Suren Baghdasaryan wrote: > > > When vma->anon_vma is not set, page fault handler will set it by

Re: [PATCH v3 17/35] mm/mmap: write-lock VMA before shrinking or expanding it

On Thu, Feb 23, 2023 at 5:46 PM Liam R. Howlett wrote: > > * Suren Baghdasaryan [230223 16:16]: > > On Thu, Feb 23, 2023 at 12:28 PM Liam R. Howlett > > wrote: > > > > > > > > > Wait, I figured a better place to do this. > > > > >

Re: [PATCH v3 17/35] mm/mmap: write-lock VMA before shrinking or expanding it

On Fri, Feb 24, 2023 at 8:14 AM Liam R. Howlett wrote: > > * Suren Baghdasaryan [230223 21:06]: > > On Thu, Feb 23, 2023 at 5:46 PM Liam R. Howlett > > wrote: > > > > > > * Suren Baghdasaryan [230223 16:16]: > > > > On Thu, Feb 23,

Re: [PATCH v4 18/33] mm: write-lock VMAs before removing them from VMA tree

On Wed, Mar 1, 2023 at 10:34 AM Suren Baghdasaryan wrote: > > On Tue, Feb 28, 2023 at 11:57 PM Hyeonggon Yoo <42.hye...@gmail.com> wrote: > > > > On Wed, Mar 01, 2023 at 07:43:33AM +, Hyeonggon Yoo wrote: > > > On Mon, Feb 27, 2023 at 09:36:17AM -0800, Suren

Re: [PATCH v4 18/33] mm: write-lock VMAs before removing them from VMA tree

On Wed, Mar 1, 2023 at 10:34 AM Suren Baghdasaryan wrote: > > On Tue, Feb 28, 2023 at 11:57 PM Hyeonggon Yoo <42.hye...@gmail.com> wrote: > > > > On Wed, Mar 01, 2023 at 07:43:33AM +, Hyeonggon Yoo wrote: > > > On Mon, Feb 27, 2023 at 09:36:17AM -0800, Suren

Re: [PATCH v4 18/33] mm: write-lock VMAs before removing them from VMA tree

On Tue, Feb 28, 2023 at 11:57 PM Hyeonggon Yoo <42.hye...@gmail.com> wrote: > > On Wed, Mar 01, 2023 at 07:43:33AM +, Hyeonggon Yoo wrote: > > On Mon, Feb 27, 2023 at 09:36:17AM -0800, Suren Baghdasaryan wrote: > > > Write-locking VMAs before isolating t

[PATCH 1/1] mm/nommu: remove unnecessary VMA locking

ocalhost/ Signed-off-by: Suren Baghdasaryan --- Fix cleanly applies over mm-unstable, SHA in "Fixes" is from that tree. mm/nommu.c | 5 - 1 file changed, 5 deletions(-) diff --git a/mm/nommu.c b/mm/nommu.c index 2ab162d773e2..57ba243c6a37 100644 --- a/mm/nommu.c +++ b/mm/nommu.c

Re: [PATCH v2 00/33] Per-VMA locks

On Tue, Feb 28, 2023 at 4:06 AM Punit Agrawal wrote: > > Punit Agrawal writes: > > > Suren Baghdasaryan writes: > > > >> Previous version: > >> v1: https://lore.kernel.org/all/20230109205336.3665937-1-sur...@google.com/ > >> RFC: https:/

Re: [PATCH v4 18/33] mm: write-lock VMAs before removing them from VMA tree

On Wed, Mar 1, 2023 at 4:54 PM Hyeonggon Yoo <42.hye...@gmail.com> wrote: > > On Wed, Mar 01, 2023 at 10:42:48AM -0800, Suren Baghdasaryan wrote: > > On Wed, Mar 1, 2023 at 10:34 AM Suren Baghdasaryan > > wrote: > > > > > > On Tue, Feb 28, 2023 at 1

Re: [PATCH v3 17/35] mm/mmap: write-lock VMA before shrinking or expanding it

On Fri, Feb 24, 2023 at 8:19 AM Suren Baghdasaryan wrote: > > On Fri, Feb 24, 2023 at 8:14 AM Liam R. Howlett > wrote: > > > > * Suren Baghdasaryan [230223 21:06]: > > > On Thu, Feb 23, 2023 at 5:46 PM Liam R. Howlett > > > wrote: > > &g

[PATCH v4 00/33] Per-VMA locks

maple_tree: Fix write memory barrier of nodes once dead for RCU mode maple_tree: Add smp_rmb() to dead node detection maple_tree: Add RCU lock checking to rcu callback functions mm: Enable maple tree RCU mode by default. Michel Lespinasse (1): mm: rcu safe VMA freeing Suren Baghdasaryan (23):

[PATCH v4 02/33] maple_tree: Detect dead nodes in mas_start()

t;) Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 4 1 file changed, 4 insertions(+) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index cc356b8369ad..089cd76ec379 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1360,12 +1360,16 @@ sta

[PATCH v4 01/33] maple_tree: Be more cautious about dead nodes

am Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 52 +++- 1 file changed, 43 insertions(+), 9 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 646297cae5d1..cc356b8369ad 100644 --- a/lib/maple_tree.c +++ b/lib/ma

[PATCH v4 03/33] maple_tree: Fix freeing of nodes in rcu mode

Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 73 1 file changed, 62 insertions(+), 11 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 089cd76ec379..44

[PATCH v4 04/33] maple_tree: remove extra smp_wmb() from mas_dead_leaves()

From: Liam Howlett The call to mte_set_dead_node() before the smp_wmb() already calls smp_wmb() so this is not needed. This is an optimization for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off

[PATCH v4 05/33] maple_tree: Fix write memory barrier of nodes once dead for RCU mode

e RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 7 +-- tools/testing/radix-tree/maple.c | 16 2 files changed, 21 insert

[PATCH v4 06/33] maple_tree: Add smp_rmb() to dead node detection

re") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 6b6eddadd9d2..8ad2d1669fad 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -539

[PATCH v4 07/33] maple_tree: Add RCU lock checking to rcu callback functions

and parent pointers. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Reported-by: Suren Baghdasaryan Signed-off-by: Liam R. Howlett --- lib/maple_tree.c | 188 --- 1 file changed, 96 insertions(+), 92 deletions(-) diff --git a/li

[PATCH v4 08/33] mm: Enable maple tree RCU mode by default.

to ensure the nodes remain valid for readers. Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 3 ++- kernel/fork.c| 3 +++ mm/mmap.c| 4 +++- 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/include/linux/mm

[PATCH v4 09/33] mm: introduce CONFIG_PER_VMA_LOCK

needs modifications to handle faults under VMA lock. Signed-off-by: Suren Baghdasaryan --- mm/Kconfig | 12 1 file changed, 12 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index ca98b2072df5..2e4a7e61768a 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1211,6 +1211,18 @@ config

[PATCH v4 10/33] mm: rcu safe VMA freeing

From: Michel Lespinasse This prepares for page faults handling under VMA lock, looking up VMAs under protection of an rcu read lock, instead of the usual mmap read lock. Signed-off-by: Michel Lespinasse Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 13 ++--- kernel

[PATCH v4 12/33] mm: add per-VMA lock and helper functions to control it

ive mmap_lock is released by incrementing mm sequence counter (mm_lock_seq). - write downgrade - if the mmap_lock is downgraded to the read lock, all vma write locks are released as well (effectivelly same as write unlock). Signed-off-by: Suren Baghdasaryan --- include/linux/m

[PATCH v4 11/33] mm: move mmap_lock assert function definitions

Move mmap_lock assert function definitions up so that they can be used by other mmap_lock routines. Signed-off-by: Suren Baghdasaryan --- include/linux/mmap_lock.h | 24 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/mmap_lock.h b/include

[PATCH v4 13/33] mm: mark VMA as being written when changing vm_flags

Updates to vm_flags have to be done with VMA marked as being written for preventing concurrent page faults or other modifications. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include

[PATCH v4 14/33] mm/mmap: move vma_prepare before vma_adjust_trans_huge

vma_prepare() acquires all locks required before VMA modifications. Move vma_prepare() before vma_adjust_trans_huge() so that VMA is locked before any modification. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm

[PATCH v4 15/33] mm/khugepaged: write-lock VMA while collapsing a huge page

a fourth lock under which page tables can be traversed, and so khugepaged must also lock out that one. Signed-off-by: Suren Baghdasaryan --- mm/khugepaged.c | 5 + mm/rmap.c | 31 --- 2 files changed, 21 insertions(+), 15 deletions(-) diff --git a/mm

[PATCH v4 16/33] mm/mmap: write-lock VMAs in vma_prepare before modifying them

Write-lock all VMAs which might be affected by a merge, split, expand or shrink operations. All these operations use vma_prepare() before making the modifications, therefore it provides a centralized place to perform VMA locking. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 10

[PATCH v4 17/33] mm/mremap: write-lock VMA while remapping it to a new address range

Write-lock VMA as locked before copying it and when copy_vma produces a new VMA. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/mmap.c | 1 + mm/mremap.c | 1 + 2 files changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index e73fbb84ce12..1f42b9a52b9b 100644

[PATCH v4 18/33] mm: write-lock VMAs before removing them from VMA tree

Write-locking VMAs before isolating them ensures that page fault handlers don't operate on isolated VMAs. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 1 + mm/nommu.c | 5 + 2 files changed, 6 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 1f42b9a52b9b..f7ed357056c4 100644

[PATCH v4 19/33] mm: conditionally write-lock VMA in free_pgtables

to free_pgtables for such scenario. Signed-off-by: Suren Baghdasaryan --- mm/internal.h | 2 +- mm/memory.c | 6 +- mm/mmap.c | 5 +++-- 3 files changed, 9 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 08ce56dbb1d9..fce94775819c 100644 --- a/mm/internal.h +++ b/mm

[PATCH v4 20/33] kernel/fork: assert no VMA readers during its destruction

Assert there are no holders of VMA lock for reading when it is about to be destroyed. Signed-off-by: Suren Baghdasaryan --- kernel/fork.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/fork.c b/kernel/fork.c index e1dd79c7738c..bdb55f25895d 100644 --- a/kernel/fork.c +++ b/kernel

[PATCH v4 21/33] mm/mmap: prevent pagefault handler from racing with mmu_notifier registration

order as in page fault handlers. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 9 + 1 file changed, 9 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index ec745586785c..b947d82e8522 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3486,6 +3486,7 @@ static void vm_lock_mapping(struct

[PATCH v4 22/33] mm: introduce vma detached flag

Per-vma locking mechanism will search for VMA under RCU protection and then after locking it, has to ensure it was not removed from the VMA tree after we found it. To make this check efficient, introduce a vma->detached flag to mark VMAs which were removed from the VMA tree. Signed-off-by: Su

[PATCH v4 23/33] mm: introduce lock_vma_under_rcu to be used from arch-specific code

is acquired. VMA lock statistics are updated according to the results. For now only anonymous VMAs can be searched this way. In other cases the function returns NULL. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c| 46

[PATCH v4 25/33] mm: add FAULT_FLAG_VMA_LOCK flag

Add a new flag to distinguish page faults handled under protection of per-vma lock. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- include/linux/mm.h | 3 ++- include/linux/mm_types.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux

[PATCH v4 24/33] mm: fall back to mmap_lock if vma->anon_vma is not yet set

ed-off-by: Suren Baghdasaryan --- mm/memory.c | 4 1 file changed, 4 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index bda4c1a991f0..8855846a361b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5243,6 +5243,10 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_stru

[PATCH v4 26/33] mm: prevent do_swap_page from handling page faults under VMA lock

Due to the possibility of do_swap_page dropping mmap_lock, abort fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/memory.c | 5 + 1 file changed, 5 insertions

[PATCH v4 27/33] mm: prevent userfaults to be handled under per-vma lock

Due to the possibility of handle_userfault dropping mmap_lock, avoid fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Suggested-by: Peter Xu --- mm/memory.c | 9 + 1 file changed, 9

[PATCH v4 28/33] mm: introduce per-VMA lock statistics

Add a new CONFIG_PER_VMA_LOCK_STATS config option to dump extra statistics about handling page fault under VMA lock. Signed-off-by: Suren Baghdasaryan --- include/linux/vm_event_item.h | 6 ++ include/linux/vmstat.h| 6 ++ mm/Kconfig.debug | 6 ++ mm/memory.c

[PATCH v4 29/33] x86/mm: try VMA lock-based page fault handling first

Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/x86/Kconfig| 1 + arch/x86/mm/fault.c | 36 2 files changed, 37 insertions(+) diff --git

[PATCH v4 31/33] powerc/mm: try VMA lock-based page fault handling first

From: Laurent Dufour Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Copied from "x86/mm: try VMA lock-based page fault handling first" Signed-off-by: Laurent Dufour Signed-off-by: Suren Baghdasaryan --- arch/

[PATCH v4 30/33] arm64/mm: try VMA lock-based page fault handling first

Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/arm64/Kconfig| 1 + arch/arm64/mm/fault.c | 36 2 files changed, 37 insertions(+) diff

[PATCH v4 32/33] mm/mmap: free vm_area_struct without call_rcu in exit_mmap

caused by it. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 2 ++ kernel/fork.c | 2 +- mm/mmap.c | 11 +++ 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d07ac92f..5e142bfe7a58 100644

[PATCH v4 33/33] mm: separate vma->lock from vm_area_struct

k anymore. A number of drivers allocate a pseudo VMA on the stack but they never use the VMA's lock, therefore it does not need to be allocated. The future drivers which might need the VMA lock should use vm_area_alloc()/vm_area_free() to allocate the VMA. Signed-off-by: Suren Baghdasaryan --- incl

Re: [PATCH v3 00/35] Per-VMA locks

On Mon, Feb 27, 2023 at 9:19 AM Davidlohr Bueso wrote: > > On Fri, 24 Feb 2023, freak07 wrote: > > >Here are some measurements from a Pixel 7 Pro that´s running a kernel either > >with the Per-VMA locks patchset or without. > >If there´s interest I can provide results of other specific apps as

Re: [PATCH v4 4/7] mm: replace vma->vm_flags direct modifications with modifier calls

On Tue, Jan 31, 2023 at 12:54 PM Andrew Morton wrote: > > On Tue, 31 Jan 2023 10:54:22 -0800 Suren Baghdasaryan > wrote: > > > > > - vma->vm_flags &= ~VM_MAYWRITE; > > > > + vm_flags_clear(vma, VM_MAYSHARE); &g

[PATCH 1/1] mm: introduce vm_flags_reset_once to replace WRITE_ONCE vm_flags updates

Provide vm_flags_reset_once() and replace the vm_flags updates which used WRITE_ONCE() to prevent compiler optimizations. Fixes: 0cce31a0aa0e ("mm: replace vma->vm_flags direct modifications with modifier calls") Reported-by: Hyeonggon Yoo <42.hye...@gmail.com> Signed-off-by:

Re: [PATCH v4 4/7] mm: replace vma->vm_flags direct modifications with modifier calls

On Tue, Jan 31, 2023 at 3:12 PM Andrew Morton wrote: > > On Tue, 31 Jan 2023 13:08:19 -0800 Suren Baghdasaryan > wrote: > > > On Tue, Jan 31, 2023 at 12:54 PM Andrew Morton > > wrote: > > > > > > On Tue, 31 Jan 2023 10:54

Re: [PATCH v3 1/7] kernel/fork: convert vma assignment to a memcpy

On Tue, Feb 7, 2023 at 9:16 AM Marco Elver wrote: > > On Thu, Jan 26, 2023 at 09:27AM -0800, Paul E. McKenney wrote: > > On Wed, Jan 25, 2023 at 05:34:49PM -0800, Andrew Morton wrote: > > > On Wed, 25 Jan 2023 16:50:01 -0800 Suren Baghdasaryan > > > wrote: >

Re: [External] [PATCH v2 00/33] Per-VMA locks

On Wed, Feb 15, 2023 at 9:33 AM Punit Agrawal wrote: > > Suren Baghdasaryan writes: > > > Previous version: > > v1: https://lore.kernel.org/all/20230109205336.3665937-1-sur...@google.com/ > > RFC: https://lore.kernel.org/all/20220901173516.702122-1-sur...@goog

Re: [PATCH v2 00/33] Per-VMA locks

On Fri, Jan 27, 2023 at 4:00 PM Suren Baghdasaryan wrote: > > On Fri, Jan 27, 2023 at 3:26 PM Matthew Wilcox wrote: > > > > On Fri, Jan 27, 2023 at 02:51:38PM -0800, Andrew Morton wrote: > > > On Fri, 27 Jan 2023 11:40:37 -0800 Suren Baghdasaryan > > > wro

Re: Fwd: Memory corruption in multithreaded user space program while calling fork

On Tue, Jul 4, 2023 at 3:04 PM Suren Baghdasaryan wrote: > > On Tue, Jul 4, 2023 at 2:28 PM Andrew Morton > wrote: > > > > On Tue, 4 Jul 2023 13:22:54 -0700 Suren Baghdasaryan > > wrote: > > > > > On Tue, Jul 4, 2023 at 9:18 AM Andrew Morton >

Re: Fwd: Memory corruption in multithreaded user space program while calling fork

On Tue, Jul 4, 2023 at 2:28 PM Andrew Morton wrote: > > On Tue, 4 Jul 2023 13:22:54 -0700 Suren Baghdasaryan > wrote: > > > On Tue, Jul 4, 2023 at 9:18 AM Andrew Morton > > wrote: > > > > > > On Tue, 4 Jul 2023 09:00:19 +0100 Greg KH >

Re: Fwd: Memory corruption in multithreaded user space program while calling fork

On Tue, Jul 4, 2023 at 9:18 AM Andrew Morton wrote: > > On Tue, 4 Jul 2023 09:00:19 +0100 Greg KH wrote: > > > > > > > Thanks! I'll investigate this later today. After discussing with > > > > > > Andrew, we would like to disable CONFIG_PER_VMA_LOCK by default > > > > > > until > > > > > > the

Re: Fwd: Memory corruption in multithreaded user space program while calling fork

y memory corruption with either my > reproducer or the original program. > ``` > > FWIW: 0bff0aaea03 ("x86/mm: try VMA lock-based page fault handling > first") [merged for v6.4-rc1, authored by Suren Baghdasaryan [already CCed]] > > That's the same commit that

Re: Fwd: Memory corruption in multithreaded user space program while calling fork

On Mon, Jul 3, 2023 at 11:08 AM Suren Baghdasaryan wrote: > > On Mon, Jul 3, 2023 at 2:53 AM Linux regression tracking (Thorsten > Leemhuis) wrote: > > > > On 02.07.23 14:27, Bagas Sanjaya wrote: > > > I notice a regression report on Bugzilla [1]. Quoting from it

Re: [PATCH v2 3/3] fork: lock VMAs of the parent process when forking

On Sat, Jul 8, 2023 at 2:18 PM Linus Torvalds wrote: > > On Sat, 8 Jul 2023 at 12:12, Suren Baghdasaryan wrote: > > > > kernel/fork.c | 1 + > > 1 file changed, 1 insertion(+) > > I ended up editing your explanation a lot. > > I'm not convinced that the bug

[PATCH v2 1/3] mm: lock a vma before stack expansion

With recent changes necessitating mmap_lock to be held for write while expanding a stack, per-VMA locks should follow the same rules and be write-locked to prevent page faults into the VMA being expanded. Add the necessary locking. Cc: sta...@vger.kernel.org Signed-off-by: Suren Baghdasaryan

[PATCH v2 2/3] mm: lock newly mapped VMA which can be modified after it becomes visible

it into the VMA tree. Other places where a new VMA is added into VMA tree do not modify it after the insertion, so do not need the same locking. Cc: sta...@vger.kernel.org Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index

[PATCH v2 3/3] fork: lock VMAs of the parent process when forking

?id=217624 Fixes: 0bff0aaea03e ("x86/mm: try VMA lock-based page fault handling first") Cc: sta...@vger.kernel.org Signed-off-by: Suren Baghdasaryan --- kernel/fork.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/fork.c b/kernel/fork.c index b85814e614a5..d2e12b6d2

Re: [PATCH] mm: lock newly mapped VMA with corrected ordering

On Sat, Jul 8, 2023 at 4:04 PM Hugh Dickins wrote: > > Lockdep is certainly right to complain about > (>vm_lock->lock){}-{3:3}, at: vma_start_write+0x2d/0x3f >but task is already holding lock: > (>i_mmap_rwsem){+.+.}-{3:3}, at: mmap_region+0x4dc/0x6db > Invert those to the

Re: [PATCH v2 3/3] fork: lock VMAs of the parent process when forking

On Sat, Jul 8, 2023 at 12:12 PM Suren Baghdasaryan wrote: > > When forking a child process, parent write-protects an anonymous page > and COW-shares it with the child being forked using copy_present_pte(). > Parent's TLB is flushed right before we drop the parent's mmap_lock i

Re: Fwd: Memory corruption in multithreaded user space program while calling fork

On Sat, Jul 8, 2023 at 11:05 AM Linus Torvalds wrote: > > On Sat, 8 Jul 2023 at 10:39, Andrew Morton wrote: > > > > That was the v1 fix, but after some discussion > > (https://lkml.kernel.org/r/20230705063711.2670599-1-sur...@google.com) > > it was decided to take the "excessive" approach. > >

Re: Fwd: Memory corruption in multithreaded user space program while calling fork

On Sat, Jul 8, 2023 at 12:23 PM Linus Torvalds wrote: > > On Sat, 8 Jul 2023 at 12:17, Suren Baghdasaryan wrote: > > > > Do you want me to disable per-VMA locks by default as well? > > No. I seriously believe that if the per-vma locking is so broken that &

Re: [PATCH] mm: lock newly mapped VMA with corrected ordering

On Sat, Jul 8, 2023 at 4:10 PM Suren Baghdasaryan wrote: > > On Sat, Jul 8, 2023 at 4:04 PM Hugh Dickins wrote: > > > > Lockdep is certainly right to complain about > > (>vm_lock->lock){}-{3:3}, at: vma_start_write+0x2d/0x3f > >

Re: [PATCH v2 3/3] fork: lock VMAs of the parent process when forking

On Sat, Jul 8, 2023 at 3:54 PM Linus Torvalds wrote: > > On Sat, 8 Jul 2023 at 15:36, Suren Baghdasaryan wrote: > > > > On Sat, Jul 8, 2023 at 2:18 PM Linus Torvalds > > > > > > Again - maybe I messed up, but it really feels like the missing > >

Re: [PATCH v4 00/33] Per-VMA locks

> On Mon, Feb 27, 2023 at 09:35:59AM -0800, Suren Baghdasaryan wrote: > > > > > > > > <...> > > > > > > > >> Laurent Dufour (1): > > > >> powerc/mm: try VMA lock-based page fault handling first > > > > > >

Re: Fwd: Memory corruption in multithreaded user space program while calling fork

On Mon, Jul 3, 2023 at 11:44 AM Greg KH wrote: > > On Mon, Jul 03, 2023 at 11:27:19AM -0700, Suren Baghdasaryan wrote: > > On Mon, Jul 3, 2023 at 11:08 AM Suren Baghdasaryan > > wrote: > > > > > > On Mon, Jul 3, 2023 at 2:53 AM Linux regression track

Re: Fwd: Memory corruption in multithreaded user space program while calling fork

On Wed, Jul 5, 2023 at 9:14 AM Suren Baghdasaryan wrote: > > On Wed, Jul 5, 2023 at 8:49 AM Andrew Morton > wrote: > > > > On Wed, 5 Jul 2023 10:51:57 +0200 "Linux regression tracking (Thorsten > > Leemhuis)" wrote: > > > > > >>

Re: Fwd: Memory corruption in multithreaded user space program while calling fork

On Wed, Jul 5, 2023 at 8:49 AM Andrew Morton wrote: > > On Wed, 5 Jul 2023 10:51:57 +0200 "Linux regression tracking (Thorsten > Leemhuis)" wrote: > > > >>> I'm in wait-a-few-days-mode on this. To see if we have a backportable > > >>> fix rather than disabling the feature in -stable. > > > >

Re: [PATCH rfc -next 01/10] mm: add a generic VMA lock-based page fault handler

On Thu, Jul 13, 2023 at 9:15 AM Matthew Wilcox wrote: > > > +int try_vma_locked_page_fault(struct vm_locked_fault *vmlf, vm_fault_t > > *ret) > > +{ > > + struct vm_area_struct *vma; > > + vm_fault_t fault; > > > On Thu, Jul 13, 2023 at 05:53:29PM +0800, Kefeng Wang wrote: > > +#define

<    1   2   3   4   >