The code in on_each_cpu_cond sets CPUs in a locally allocated bitmask,
which should never be used by other CPUs simultaneously. There is no
need to use locked memory accesses to set the bits in this bitmap.
Switch to __cpumask_set_cpu.
Suggested-by: Peter Zijlstra
Signed-off-by: Rik van Riel
Move some code that will be needed for the lazy -> !lazy state
transition when a lazy TLB CPU has gotten out of date.
No functional changes, since the if (real_prev == next) branch
always returns.
Suggested-by: Andy Lutomirski
Signed-off-by: Rik van Riel
Acked-by: Dave Hansen
Cc: Li
Now that CPUs in lazy TLB mode no longer receive TLB shootdown IPIs, except
at page table freeing time, and idle CPUs will no longer get shootdown IPIs
for things like mprotect and madvise, we can always use lazy TLB mode.
Tested-by: Song Liu
Signed-off-by: Rik van Riel
Acked-by: Dave Hansen
The code in on_each_cpu_cond sets CPUs in a locally allocated bitmask,
which should never be used by other CPUs simultaneously. There is no
need to use locked memory accesses to set the bits in this bitmap.
Switch to __cpumask_set_cpu.
Suggested-by: Peter Zijlstra
Signed-off-by: Rik van Riel
Move some code that will be needed for the lazy -> !lazy state
transition when a lazy TLB CPU has gotten out of date.
No functional changes, since the if (real_prev == next) branch
always returns.
Suggested-by: Andy Lutomirski
Signed-off-by: Rik van Riel
Acked-by: Dave Hansen
Cc: Li
Now that CPUs in lazy TLB mode no longer receive TLB shootdown IPIs, except
at page table freeing time, and idle CPUs will no longer get shootdown IPIs
for things like mprotect and madvise, we can always use lazy TLB mode.
Tested-by: Song Liu
Signed-off-by: Rik van Riel
Acked-by: Dave Hansen
Pass the information on to native_flush_tlb_others.
No functional changes.
Signed-off-by: Rik van Riel
---
arch/x86/include/asm/tlbflush.h | 1 +
arch/x86/mm/tlb.c | 1 +
2 files changed, 2 insertions(+)
diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm
Pass the information on to native_flush_tlb_others.
No functional changes.
Signed-off-by: Rik van Riel
---
arch/x86/include/asm/tlbflush.h | 1 +
arch/x86/mm/tlb.c | 1 +
2 files changed, 2 insertions(+)
diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm
Linus asked me to come up with a smaller patch set to get the benefits
of lazy TLB mode, so I spent some time trying out various permutations
of the code, with a few workloads that do lots of context switches, and
also happen to have a fair number of TLB flushes a second.
Both of the workloads
Linus asked me to come up with a smaller patch set to get the benefits
of lazy TLB mode, so I spent some time trying out various permutations
of the code, with a few workloads that do lots of context switches, and
also happen to have a fair number of TLB flushes a second.
Both of the workloads
On Mon, 2018-09-24 at 14:37 -0400, Rik van Riel wrote:
> Linus asked me to come up with a smaller patch set to get the
> benefits
> of lazy TLB mode, so I spent some time trying out various
> permutations
> of the code, with a few workloads that do lots of context switches,
>
On Mon, 2018-09-24 at 14:37 -0400, Rik van Riel wrote:
> Linus asked me to come up with a smaller patch set to get the
> benefits
> of lazy TLB mode, so I spent some time trying out various
> permutations
> of the code, with a few workloads that do lots of context switches,
>
Add an argument to flush_tlb_mm_range to indicate whether page tables
are about to be freed after this TLB flush. This allows for an
optimization of flush_tlb_mm_range to skip CPUs in lazy TLB mode.
No functional changes.
Signed-off-by: Rik van Riel
---
arch/x86/include/asm/tlb.h | 6
Introduce a variant of on_each_cpu_cond that iterates only over the
CPUs in a cpumask, in order to avoid making callbacks for every single
CPU in the system when we only need to test a subset.
Signed-off-by: Rik van Riel
---
include/linux/smp.h | 4
kernel/smp.c| 17
The code in on_each_cpu_cond sets CPUs in a locally allocated bitmask,
which should never be used by other CPUs simultaneously. There is no
need to use locked memory accesses to set the bits in this bitmap.
Switch to __cpumask_set_cpu.
Suggested-by: Peter Zijlstra
Signed-off-by: Rik van Riel
The code in on_each_cpu_cond sets CPUs in a locally allocated bitmask,
which should never be used by other CPUs simultaneously. There is no
need to use locked memory accesses to set the bits in this bitmap.
Switch to __cpumask_set_cpu.
Suggested-by: Peter Zijlstra
Signed-off-by: Rik van Riel
Add an argument to flush_tlb_mm_range to indicate whether page tables
are about to be freed after this TLB flush. This allows for an
optimization of flush_tlb_mm_range to skip CPUs in lazy TLB mode.
No functional changes.
Signed-off-by: Rik van Riel
---
arch/x86/include/asm/tlb.h | 6
Introduce a variant of on_each_cpu_cond that iterates only over the
CPUs in a cpumask, in order to avoid making callbacks for every single
CPU in the system when we only need to test a subset.
Signed-off-by: Rik van Riel
---
include/linux/smp.h | 4
kernel/smp.c| 17
Move some code that will be needed for the lazy -> !lazy state
transition when a lazy TLB CPU has gotten out of date.
No functional changes, since the if (real_prev == next) branch
always returns.
Suggested-by: Andy Lutomirski
Signed-off-by: Rik van Riel
Acked-by: Dave Hansen
Cc: Li
Pass the information on to native_flush_tlb_others.
No functional changes.
Signed-off-by: Rik van Riel
---
arch/x86/include/asm/tlbflush.h | 1 +
arch/x86/mm/tlb.c | 1 +
2 files changed, 2 insertions(+)
diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm
Linus asked me to come up with a smaller patch set to get the benefits
of lazy TLB mode, so I spent some time trying out various permutations
of the code, with a few workloads that do lots of context switches, and
also happen to have a fair number of TLB flushes a second.
Both of the workloads
Now that CPUs in lazy TLB mode no longer receive TLB shootdown IPIs, except
at page table freeing time, and idle CPUs will no longer get shootdown IPIs
for things like mprotect and madvise, we can always use lazy TLB mode.
Tested-by: Song Liu
Signed-off-by: Rik van Riel
Acked-by: Dave Hansen
ode remain part of the mm_cpumask(mm), both because
that allows TLB flush IPIs to be sent at page table freeing time, and
because the cache line bouncing on the mm_cpumask(mm) was responsible
for about half the CPU use in switch_mm_irqs_off().
Tested-by: Song Liu
Signed-off-by: Rik van Riel
---
a
Move some code that will be needed for the lazy -> !lazy state
transition when a lazy TLB CPU has gotten out of date.
No functional changes, since the if (real_prev == next) branch
always returns.
Suggested-by: Andy Lutomirski
Signed-off-by: Rik van Riel
Acked-by: Dave Hansen
Cc: Li
Pass the information on to native_flush_tlb_others.
No functional changes.
Signed-off-by: Rik van Riel
---
arch/x86/include/asm/tlbflush.h | 1 +
arch/x86/mm/tlb.c | 1 +
2 files changed, 2 insertions(+)
diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm
Linus asked me to come up with a smaller patch set to get the benefits
of lazy TLB mode, so I spent some time trying out various permutations
of the code, with a few workloads that do lots of context switches, and
also happen to have a fair number of TLB flushes a second.
Both of the workloads
Now that CPUs in lazy TLB mode no longer receive TLB shootdown IPIs, except
at page table freeing time, and idle CPUs will no longer get shootdown IPIs
for things like mprotect and madvise, we can always use lazy TLB mode.
Tested-by: Song Liu
Signed-off-by: Rik van Riel
Acked-by: Dave Hansen
ode remain part of the mm_cpumask(mm), both because
that allows TLB flush IPIs to be sent at page table freeing time, and
because the cache line bouncing on the mm_cpumask(mm) was responsible
for about half the CPU use in switch_mm_irqs_off().
Tested-by: Song Liu
Signed-off-by: Rik van Riel
---
a
On Mon, 2018-09-24 at 17:23 +0200, Jan H. Schönherr wrote:
> On 09/18/2018 04:40 PM, Rik van Riel wrote:
> > On Fri, 2018-09-14 at 18:25 +0200, Jan H. Schönherr wrote:
> > > On 09/14/2018 01:12 PM, Peter Zijlstra wrote:
> > > > On Fri, Sep 07, 2018 at 11:3
On Mon, 2018-09-24 at 17:23 +0200, Jan H. Schönherr wrote:
> On 09/18/2018 04:40 PM, Rik van Riel wrote:
> > On Fri, 2018-09-14 at 18:25 +0200, Jan H. Schönherr wrote:
> > > On 09/14/2018 01:12 PM, Peter Zijlstra wrote:
> > > > On Fri, Sep 07, 2018 at 11:3
On Thu, 2018-09-20 at 03:14 +0100, Edward Cree wrote:
> I think there are important differences between code to be run by
> CPUs
> and a Code to be run by humans. And when the author goes on a
> victory
> lap on Twitter and declares the Code to be "a political document",
> is
> it any
On Thu, 2018-09-20 at 03:14 +0100, Edward Cree wrote:
> I think there are important differences between code to be run by
> CPUs
> and a Code to be run by humans. And when the author goes on a
> victory
> lap on Twitter and declares the Code to be "a political document",
> is
> it any
> On Sep 19, 2018, at 1:00 PM, Paolo Bonzini wrote:
>
> On 19/09/2018 18:57, Sebastian Andrzej Siewior wrote:
>> On 2018-09-19 07:55:51 [+0200], Paolo Bonzini wrote:
>>> A kthread can do use_mm/unuse_mm.
>>
>> indeed. The FPU struct for the kernel thread isn't valid / does not
>> contain the
> On Sep 19, 2018, at 1:00 PM, Paolo Bonzini wrote:
>
> On 19/09/2018 18:57, Sebastian Andrzej Siewior wrote:
>> On 2018-09-19 07:55:51 [+0200], Paolo Bonzini wrote:
>>> A kthread can do use_mm/unuse_mm.
>>
>> indeed. The FPU struct for the kernel thread isn't valid / does not
>> contain the
On Tue, 2018-09-18 at 18:04 +0200, Sebastian Andrzej Siewior wrote:
> On 2018-09-18 17:29:52 [+0200], Paolo Bonzini wrote:
> > > I don't think it matters what the PKRU state is
> > > for kernel threads, since kernel PTEs should not
> > > be using protection keys anyway.
> >
> > What about
On Tue, 2018-09-18 at 18:04 +0200, Sebastian Andrzej Siewior wrote:
> On 2018-09-18 17:29:52 [+0200], Paolo Bonzini wrote:
> > > I don't think it matters what the PKRU state is
> > > for kernel threads, since kernel PTEs should not
> > > be using protection keys anyway.
> >
> > What about
On Tue, 2018-09-18 at 17:07 +0200, Paolo Bonzini wrote:
> On 18/09/2018 16:27, Sebastian Andrzej Siewior wrote:
> > > Likewise, move this to fpu__clear and outside "if
> > > (static_cpu_has(X86_FEATURE_FPU))"?
> >
> > okay. But if there is no FPU we did not save/restore the pkru
> > value. Is
> >
On Tue, 2018-09-18 at 17:07 +0200, Paolo Bonzini wrote:
> On 18/09/2018 16:27, Sebastian Andrzej Siewior wrote:
> > > Likewise, move this to fpu__clear and outside "if
> > > (static_cpu_has(X86_FEATURE_FPU))"?
> >
> > okay. But if there is no FPU we did not save/restore the pkru
> > value. Is
> >
On Fri, 2018-09-14 at 18:25 +0200, Jan H. Schönherr wrote:
> On 09/14/2018 01:12 PM, Peter Zijlstra wrote:
> > On Fri, Sep 07, 2018 at 11:39:47PM +0200, Jan H. Schönherr wrote:
> > >
> > > B) Why would I want this?
> > >In the L1TF context, it prevents other applications from
> > > loading
>
On Fri, 2018-09-14 at 18:25 +0200, Jan H. Schönherr wrote:
> On 09/14/2018 01:12 PM, Peter Zijlstra wrote:
> > On Fri, Sep 07, 2018 at 11:39:47PM +0200, Jan H. Schönherr wrote:
> > >
> > > B) Why would I want this?
> > >In the L1TF context, it prevents other applications from
> > > loading
>
On Tue, 2018-09-18 at 15:22 +0200, Jan H. Schönherr wrote:
> On 09/17/2018 11:48 AM, Peter Zijlstra wrote:
> > On Sat, Sep 15, 2018 at 10:48:20AM +0200, Jan H. Schönherr wrote:
> >
> > >
> > > CFS bandwidth control would also need to change significantly as
> > > we would now
> > > have to
On Tue, 2018-09-18 at 15:22 +0200, Jan H. Schönherr wrote:
> On 09/17/2018 11:48 AM, Peter Zijlstra wrote:
> > On Sat, Sep 15, 2018 at 10:48:20AM +0200, Jan H. Schönherr wrote:
> >
> > >
> > > CFS bandwidth control would also need to change significantly as
> > > we would now
> > > have to
On Wed, 2018-09-12 at 08:20 -0700, Andy Lutomirski wrote:
> >
> > --- a/arch/x86/mm/pkeys.c
> > +++ b/arch/x86/mm/pkeys.c
> > @@ -18,6 +18,20 @@
> >
> > #include /* boot_cpu_has,
> > ...*/
> > #include /*
> > vma_pkey() */
> > +#include
> >
On Wed, 2018-09-12 at 08:20 -0700, Andy Lutomirski wrote:
> >
> > --- a/arch/x86/mm/pkeys.c
> > +++ b/arch/x86/mm/pkeys.c
> > @@ -18,6 +18,20 @@
> >
> > #include /* boot_cpu_has,
> > ...*/
> > #include /*
> > vma_pkey() */
> > +#include
> >
On Fri, 2018-09-07 at 14:44 +0100, Will Deacon wrote:
> On Thu, Sep 06, 2018 at 04:29:59PM -0400, Rik van Riel wrote:
> > On Thu, 2018-08-23 at 18:47 +1000, Nicholas Piggin wrote:
> > > There is no need to call this from tlb_flush_mmu_tlbonly, it
> > > logically belo
On Fri, 2018-09-07 at 14:44 +0100, Will Deacon wrote:
> On Thu, Sep 06, 2018 at 04:29:59PM -0400, Rik van Riel wrote:
> > On Thu, 2018-08-23 at 18:47 +1000, Nicholas Piggin wrote:
> > > There is no need to call this from tlb_flush_mmu_tlbonly, it
> > > logically belo
On Thu, 2018-08-23 at 18:47 +1000, Nicholas Piggin wrote:
> There is no need to call this from tlb_flush_mmu_tlbonly, it
> logically belongs with tlb_flush_mmu_free. This allows some
> code consolidation with a subsequent fix.
>
> Signed-off-by: Nicholas Piggin
Reviewed-b
On Thu, 2018-08-23 at 18:47 +1000, Nicholas Piggin wrote:
> There is no need to call this from tlb_flush_mmu_tlbonly, it
> logically belongs with tlb_flush_mmu_free. This allows some
> code consolidation with a subsequent fix.
>
> Signed-off-by: Nicholas Piggin
Reviewed-b
On Fri, 2018-08-31 at 14:31 -0700, Roman Gushchin wrote:
> On Fri, Aug 31, 2018 at 05:15:39PM -0400, Rik van Riel wrote:
> > On Fri, 2018-08-31 at 13:34 -0700, Roman Gushchin wrote:
> >
> > > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > > index fa2c150ab7b9..c
On Fri, 2018-08-31 at 14:31 -0700, Roman Gushchin wrote:
> On Fri, Aug 31, 2018 at 05:15:39PM -0400, Rik van Riel wrote:
> > On Fri, 2018-08-31 at 13:34 -0700, Roman Gushchin wrote:
> >
> > > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > > index fa2c150ab7b9..c
it?
With this patch, a slab with 5000 objects on it will
get 1 item scanned, while a slab with 4000 objects on
it will see shrinker->batch or SHRINK_BATCH objects
scanned every time.
I don't know if this would cause any issues, just
something to ponder.
If nobody things this is a problem, you ca
it?
With this patch, a slab with 5000 objects on it will
get 1 item scanned, while a slab with 4000 objects on
it will see shrinker->batch or SHRINK_BATCH objects
scanned every time.
I don't know if this would cause any issues, just
something to ponder.
If nobody things this is a problem, you ca
he wrong memory.
>
> Fix it by adding a new nmi_uaccess_okay() helper and checking it in
> copy_from_user_nmi() and in __copy_from_user_nmi()'s callers.
>
> Cc: sta...@vger.kernel.org
> Cc: Peter Zijlstra
> Cc: Nadav Amit
> Signed-off-by: Andy Lutomirski
Reviewed-by:
he wrong memory.
>
> Fix it by adding a new nmi_uaccess_okay() helper and checking it in
> copy_from_user_nmi() and in __copy_from_user_nmi()'s callers.
>
> Cc: sta...@vger.kernel.org
> Cc: Peter Zijlstra
> Cc: Nadav Amit
> Signed-off-by: Andy Lutomirski
Reviewed-by:
On Wed, 2018-08-29 at 08:36 -0700, Andy Lutomirski wrote:
> On Wed, Aug 29, 2018 at 8:17 AM, Rik van Riel
> wrote:
> > On Tue, 2018-08-28 at 20:46 -0700, Andy Lutomirski wrote:
> > > On Tue, Aug 28, 2018 at 10:56 AM, Rik van Riel
> > > wrote:
> > &g
On Wed, 2018-08-29 at 08:36 -0700, Andy Lutomirski wrote:
> On Wed, Aug 29, 2018 at 8:17 AM, Rik van Riel
> wrote:
> > On Tue, 2018-08-28 at 20:46 -0700, Andy Lutomirski wrote:
> > > On Tue, Aug 28, 2018 at 10:56 AM, Rik van Riel
> > > wrote:
> > &g
On Tue, 2018-08-28 at 20:46 -0700, Andy Lutomirski wrote:
> On Tue, Aug 28, 2018 at 10:56 AM, Rik van Riel
> wrote:
> > On Mon, 27 Aug 2018 16:04:16 -0700
> > Andy Lutomirski wrote:
> >
> > > The 0day bot is still chewing on this, but I've tested it a bit
>
On Tue, 2018-08-28 at 20:46 -0700, Andy Lutomirski wrote:
> On Tue, Aug 28, 2018 at 10:56 AM, Rik van Riel
> wrote:
> > On Mon, 27 Aug 2018 16:04:16 -0700
> > Andy Lutomirski wrote:
> >
> > > The 0day bot is still chewing on this, but I've tested it a bit
>
r.kernel.org
Cc: Peter Zijlstra
Cc: Nadav Amit
Signed-off-by: Rik van Riel
Signed-off-by: Andy Lutomirski
---
arch/x86/events/core.c | 2 +-
arch/x86/include/asm/tlbflush.h | 15 +++
arch/x86/lib/usercopy.c | 5 +
arch/x86/mm/tlb.c | 9 +++
r.kernel.org
Cc: Peter Zijlstra
Cc: Nadav Amit
Signed-off-by: Rik van Riel
Signed-off-by: Andy Lutomirski
---
arch/x86/events/core.c | 2 +-
arch/x86/include/asm/tlbflush.h | 15 +++
arch/x86/lib/usercopy.c | 5 +
arch/x86/mm/tlb.c | 9 +++
On Mon, 2018-08-27 at 19:10 -0700, Andy Lutomirski wrote:
> On Mon, Aug 27, 2018 at 6:31 PM, Rik van Riel
> wrote:
>
> > What is special about this path wrt nmi_uaccess_ok that is
> > not also true for the need_flush branch right above it?
> >
> > What am I
On Mon, 2018-08-27 at 19:10 -0700, Andy Lutomirski wrote:
> On Mon, Aug 27, 2018 at 6:31 PM, Rik van Riel
> wrote:
>
> > What is special about this path wrt nmi_uaccess_ok that is
> > not also true for the need_flush branch right above it?
> >
> > What am I
On Mon, 2018-08-27 at 16:04 -0700, Andy Lutomirski wrote:
> +++ b/arch/x86/mm/tlb.c
> @@ -345,6 +345,9 @@ void switch_mm_irqs_off(struct mm_struct *prev,
> struct mm_struct *next,
>*/
> trace_tlb_flush_rcuidle(TLB_FLUSH_ON_TASK_SWITCH,
> TLB_FLUSH_ALL);
> }
On Mon, 2018-08-27 at 16:04 -0700, Andy Lutomirski wrote:
> +++ b/arch/x86/mm/tlb.c
> @@ -345,6 +345,9 @@ void switch_mm_irqs_off(struct mm_struct *prev,
> struct mm_struct *next,
>*/
> trace_tlb_flush_rcuidle(TLB_FLUSH_ON_TASK_SWITCH,
> TLB_FLUSH_ALL);
> }
On Mon, 2018-08-27 at 18:04 +1000, Nicholas Piggin wrote:
> It could do that. It requires a tlbie that matches the page size,
> so it means 3 sizes. I think possibly even that would be better
> than current code, but we could do better if we had a few specific
> fields in there.
Would it cause a
On Mon, 2018-08-27 at 18:04 +1000, Nicholas Piggin wrote:
> It could do that. It requires a tlbie that matches the page size,
> so it means 3 sizes. I think possibly even that would be better
> than current code, but we could do better if we had a few specific
> fields in there.
Would it cause a
On Wed, 2018-08-22 at 14:37 -0700, Linus Torvalds wrote:
> On Wed, Aug 22, 2018 at 8:46 AM Peter Zijlstra
> wrote:
> >
> > Revert [..] in order to simplify the TLB invalidate fixes for x86.
> > We'll try again later.
>
> Rik, I assume I should take your earlier "yeah, I can try later" as
> an
>
On Wed, 2018-08-22 at 14:37 -0700, Linus Torvalds wrote:
> On Wed, Aug 22, 2018 at 8:46 AM Peter Zijlstra
> wrote:
> >
> > Revert [..] in order to simplify the TLB invalidate fixes for x86.
> > We'll try again later.
>
> Rik, I assume I should take your earlier "yeah, I can try later" as
> an
>
On Wed, 2018-08-15 at 18:54 -0700, Andy Lutomirski wrote:
> On Mon, Jul 16, 2018 at 12:03 PM, Rik van Riel
> wrote:
> Hi Rik-
>
> I was looking through this, and I see:
>
> > -static void tlb_remove_table_one(void *table)
> > +static void tlb_remove_table_one(
On Wed, 2018-08-15 at 18:54 -0700, Andy Lutomirski wrote:
> On Mon, Jul 16, 2018 at 12:03 PM, Rik van Riel
> wrote:
> Hi Rik-
>
> I was looking through this, and I see:
>
> > -static void tlb_remove_table_one(void *table)
> > +static void tlb_remove_table_one(
On Fri, 2018-08-03 at 19:25 +0200, Peter Zijlstra wrote:
> On Fri, Aug 03, 2018 at 12:40:48PM -0400, Rik van Riel wrote:
> > On Fri, 2018-08-03 at 17:56 +0200, Peter Zijlstra wrote:
> > >
> > > Why can't we skip the ->active_mm swizzle and keep ->active_mm
On Fri, 2018-08-03 at 19:25 +0200, Peter Zijlstra wrote:
> On Fri, Aug 03, 2018 at 12:40:48PM -0400, Rik van Riel wrote:
> > On Fri, 2018-08-03 at 17:56 +0200, Peter Zijlstra wrote:
> > >
> > > Why can't we skip the ->active_mm swizzle and keep ->active_mm
On Fri, 2018-08-03 at 17:56 +0200, Peter Zijlstra wrote:
> On Wed, Aug 01, 2018 at 06:02:55AM -0400, Rik van Riel wrote:
> > Conditionally skip lazy TLB mm refcounting. When an architecture
> > has
> > CONFIG_ARCH_NO_ACTIVE_MM_REFCOUNTING enabled, an mm that is used in
>
On Fri, 2018-08-03 at 17:56 +0200, Peter Zijlstra wrote:
> On Wed, Aug 01, 2018 at 06:02:55AM -0400, Rik van Riel wrote:
> > Conditionally skip lazy TLB mm refcounting. When an architecture
> > has
> > CONFIG_ARCH_NO_ACTIVE_MM_REFCOUNTING enabled, an mm that is used in
>
The function leave_mm does not use its cpu argument, but always works on
the CPU where it is called. Change the argument to a void *, so leave_mm
can be called directly from smp_call_function_mask, and stop looking up the
CPU number in current leave_mm callers.
Signed-off-by: Rik van Riel
The function leave_mm does not use its cpu argument, but always works on
the CPU where it is called. Change the argument to a void *, so leave_mm
can be called directly from smp_call_function_mask, and stop looking up the
CPU number in current leave_mm callers.
Signed-off-by: Rik van Riel
Clarify exactly what the memory barrier synchronizes with.
Suggested-by: Peter Zijlstra
Signed-off-by: Rik van Riel
Reviewed-by: Andy Lutomirski
---
arch/x86/mm/tlb.c | 7 +--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index
Clarify exactly what the memory barrier synchronizes with.
Suggested-by: Peter Zijlstra
Signed-off-by: Rik van Riel
Reviewed-by: Andy Lutomirski
---
arch/x86/mm/tlb.c | 7 +--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index
is about to start using.
Signed-off-by: Rik van Riel
---
arch/x86/mm/tlb.c| 5 +++--
fs/exec.c| 2 +-
include/linux/sched/mm.h | 25 +
kernel/sched/core.c | 29 +
mm/mmu_context.c | 21
is about to start using.
Signed-off-by: Rik van Riel
---
arch/x86/mm/tlb.c| 5 +++--
fs/exec.c| 2 +-
include/linux/sched/mm.h | 25 +
kernel/sched/core.c | 29 +
mm/mmu_context.c | 21
in lazy TLB mode.
Signed-off-by: Rik van Riel
---
arch/x86/mm/tlb.c | 4
1 file changed, 4 insertions(+)
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index 7b1add904396..425cb9fa2640 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -140,6 +140,8 @@ void leave_mm(void *dummy
Instead of open coding bitmap magic, use on_each_cpu_cond
to determine which CPUs to send TLB flush IPIs to.
This might be a little bit slower than examining the bitmaps,
but it should be a lot easier to maintain in the long run.
Suggested-by: Peter Zijlstra
Signed-off-by: Rik van Riel
in lazy TLB mode.
Signed-off-by: Rik van Riel
---
arch/x86/mm/tlb.c | 4
1 file changed, 4 insertions(+)
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index 7b1add904396..425cb9fa2640 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -140,6 +140,8 @@ void leave_mm(void *dummy
Instead of open coding bitmap magic, use on_each_cpu_cond
to determine which CPUs to send TLB flush IPIs to.
This might be a little bit slower than examining the bitmaps,
but it should be a lot easier to maintain in the long run.
Suggested-by: Peter Zijlstra
Signed-off-by: Rik van Riel
to the same mm after a flush.
Suggested-by: Andy Lutomirski
Signed-off-by: Rik van Riel
Acked-by: Andy Lutomirski
---
arch/x86/mm/tlb.c | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index 671cc66df801..149fb64e4bf4 100644
--- a/arch/x86/mm
Shooting down lazy TLB references to an mm at exit_mmap time ensures
that no users of the mm_struct will be left anywhere in the system,
allowing it to be torn down and freed immediately.
Signed-off-by: Rik van Riel
Suggested-by: Andy Lutomirski
Suggested-by: Peter Zijlstra
---
arch/x86
This patch series implements the cleanups suggested by Peter and Andy,
removes lazy TLB mm refcounting on x86, and shows how other architectures
could implement that same optimization.
The previous patch series already seems to have removed most of the
cache line contention I was seeing at
Add a config variable indicating that this architecture does not require
lazy TLB mm refcounting, because lazy TLB mms get shot down instantly
at exit_mmap time.
Signed-off-by: Rik van Riel
---
arch/Kconfig | 4
1 file changed, 4 insertions(+)
diff --git a/arch/Kconfig b/arch/Kconfig
to the same mm after a flush.
Suggested-by: Andy Lutomirski
Signed-off-by: Rik van Riel
Acked-by: Andy Lutomirski
---
arch/x86/mm/tlb.c | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index 671cc66df801..149fb64e4bf4 100644
--- a/arch/x86/mm
Shooting down lazy TLB references to an mm at exit_mmap time ensures
that no users of the mm_struct will be left anywhere in the system,
allowing it to be torn down and freed immediately.
Signed-off-by: Rik van Riel
Suggested-by: Andy Lutomirski
Suggested-by: Peter Zijlstra
---
arch/x86
This patch series implements the cleanups suggested by Peter and Andy,
removes lazy TLB mm refcounting on x86, and shows how other architectures
could implement that same optimization.
The previous patch series already seems to have removed most of the
cache line contention I was seeing at
Add a config variable indicating that this architecture does not require
lazy TLB mm refcounting, because lazy TLB mms get shot down instantly
at exit_mmap time.
Signed-off-by: Rik van Riel
---
arch/Kconfig | 4
1 file changed, 4 insertions(+)
diff --git a/arch/Kconfig b/arch/Kconfig
Introduce a variant of on_each_cpu_cond that iterates only over the
CPUs in a cpumask, in order to avoid making callbacks for every single
CPU in the system when we only need to test a subset.
Signed-off-by: Rik van Riel
---
include/linux/smp.h | 4
kernel/smp.c| 17
Introduce a variant of on_each_cpu_cond that iterates only over the
CPUs in a cpumask, in order to avoid making callbacks for every single
CPU in the system when we only need to test a subset.
Signed-off-by: Rik van Riel
---
include/linux/smp.h | 4
kernel/smp.c| 17
The code in on_each_cpu_cond sets CPUs in a locally allocated bitmask,
which should never be used by other CPUs simultaneously. There is no
need to use locked memory accesses to set the bits in this bitmap.
Switch to __cpumask_set_cpu.
Suggested-by: Peter Zijlstra
Signed-off-by: Rik van Riel
The code in on_each_cpu_cond sets CPUs in a locally allocated bitmask,
which should never be used by other CPUs simultaneously. There is no
need to use locked memory accesses to set the bits in this bitmap.
Switch to __cpumask_set_cpu.
Suggested-by: Peter Zijlstra
Signed-off-by: Rik van Riel
Turn the dummy tlb_flush_remove_tables* defines into inline functions,
in order to get compiler type checking, etc.
Suggested-by: Peter Zijlstra
Signed-off-by: Rik van Riel
---
include/asm-generic/tlb.h | 9 +++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/include/asm
Turn the dummy tlb_flush_remove_tables* defines into inline functions,
in order to get compiler type checking, etc.
Suggested-by: Peter Zijlstra
Signed-off-by: Rik van Riel
---
include/asm-generic/tlb.h | 9 +++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/include/asm
On Tue, 2018-07-31 at 07:29 -0700, Andy Lutomirski wrote:
> > On Jul 31, 2018, at 2:12 AM, Peter Zijlstra
> > wrote:
> >
> > > On Mon, Jul 30, 2018 at 09:05:55PM -0400, Rik van Riel wrote:
> > > > On Mon, 2018-07-30 at 18:26 +0200, Peter Zijlstra wrote:
&g
On Tue, 2018-07-31 at 07:29 -0700, Andy Lutomirski wrote:
> > On Jul 31, 2018, at 2:12 AM, Peter Zijlstra
> > wrote:
> >
> > > On Mon, Jul 30, 2018 at 09:05:55PM -0400, Rik van Riel wrote:
> > > > On Mon, 2018-07-30 at 18:26 +0200, Peter Zijlstra wrote:
&g
301 - 400 of 7046 matches
Mail list logo
