Re: [PATCH 4/4] powerpc/64s/radix: combine final TLB flush and lazy tlb mm shootdown IPIs

2023-07-20 Thread Nicholas Piggin 
On Tue Jul 18, 2023 at 12:54 PM AEST, Michael Ellerman wrote:
> Nicholas Piggin  writes:
> > This performs lazy tlb mm shootdown when doing the exit TLB flush when
> > all mm users go away and user mappings are removed, which avoids having
> > to do the lazy tlb mm shootdown IPIs on the final mmput when all kernel
> > references disappear.
> >
> > powerpc/64s uses a broadcast TLBIE for the exit TLB flush if remote CPUs
> > need to be invalidated (unless TLBIE is disabled), so this doesn't
> > necessarily save IPIs but it does avoid a broadcast TLBIE which is quite
> > expensive.
> >
> > Signed-off-by: Nicholas Piggin 
> > ---
> >  arch/powerpc/mm/book3s64/radix_tlb.c | 26 +-
> >  1 file changed, 25 insertions(+), 1 deletion(-)
>
> This gives me:
>
> [1.438910][T1] Run /init as init process
> [1.442759][   T96] [ cut here ]
> [1.442836][   T96] WARNING: CPU: 0 PID: 96 at kernel/smp.c:748 
> smp_call_function_many_cond+0xe0/0xad0
> [1.442920][   T96] Modules linked in:
> [1.442960][   T96] CPU: 0 PID: 96 Comm: init Not tainted 
> 6.5.0-rc2-g1954d181ea09 #168
> [1.443028][   T96] Hardware name: IBM pSeries (emulated by qemu) POWER9 
> (raw) 0x4e1202 0xf05 of:SLOF,git-6b6c16 hv:linux,kvm pSeries
> [1.443126][   T96] NIP:  c02aab20 LR: c00a5fc4 CTR: 
> 
> [1.443199][   T96] REGS: cc36f5b0 TRAP: 0700   Not tainted  
> (6.5.0-rc2-g1954d181ea09)
> [1.443280][   T96] MSR:  80029033   CR: 
> 44008244  XER: 2004
> [1.443382][   T96] CFAR: c02ab524 IRQMASK: 0
> [1.443382][   T96] GPR00: c00a5fc4 cc36f850 
> c17f9000 c617c580
> [1.443382][   T96] GPR04: c00a55b0 c617bd00 
> 0001 0001
> [1.443382][   T96] GPR08: c29fc88c cc25aa00 
>  44008244
> [1.443382][   T96] GPR12: fd78 c36c 
>  c4042a00
> [1.443382][   T96] GPR16: 0001  
>  
> [1.443382][   T96] GPR20:  c00a5fc4 
>  c29f85d0
> [1.443382][   T96] GPR24: cc25b518  
> c617be60 c617bd00
> [1.443382][   T96] GPR28: c617c580 c00a55b0 
>  
> [1.443994][   T96] NIP [c02aab20] 
> smp_call_function_many_cond+0xe0/0xad0
> [1.444069][   T96] LR [c00a5fc4] radix__tlb_flush+0xf4/0x190
> [1.444133][   T96] Call Trace:
> [1.444172][   T96] [cc36f850] [] 
> 0x (unreliable)
> [1.444250][   T96] [cc36f920] [c29f7fe0] 
> __cpu_possible_mask+0x0/0x100
> [1.444326][   T96] [cc36f950] [c04f346c] 
> tlb_finish_mmu+0x16c/0x220
> [1.02][   T96] [cc36f980] [c04ee894] 
> exit_mmap+0x1b4/0x580
> [1.74][   T96] [cc36faa0] [c014c140] 
> __mmput+0x60/0x1c0
> [1.444546][   T96] [cc36fae0] [c05cf014] 
> begin_new_exec+0x5d4/0xec0
> [1.444622][   T96] [cc36fb60] [c066c6e8] 
> load_elf_binary+0x4a8/0x1cf0
> [1.444697][   T96] [cc36fc60] [c05cc410] 
> bprm_execve+0x3b0/0xa60
> [1.444773][   T96] [cc36fd30] [c05ce3a0] 
> do_execveat_common+0x1d0/0x300
> [1.444852][   T96] [cc36fde0] [c05ce524] 
> sys_execve+0x54/0x70
> [1.444928][   T96] [cc36fe10] [c0031c24] 
> system_call_exception+0x134/0x360
> [1.445000][   T96] [cc36fe50] [c000d6a0] 
> system_call_common+0x160/0x2c4
> [1.445070][   T96] --- interrupt: c00 at 0x7fffb664cc98
> [1.445119][   T96] NIP:  7fffb664cc98 LR: 1004bcb0 CTR: 
> 
> [1.445189][   T96] REGS: cc36fe80 TRAP: 0c00   Not tainted  
> (6.5.0-rc2-g1954d181ea09)
> [1.445271][   T96] MSR:  8280f033 
>   CR: 22004842  XER: 
> [1.445390][   T96] IRQMASK: 0
> [1.445390][   T96] GPR00: 000b 7fffd9d11ec0 
> 7fffb6767300 2b3f06e8
> [1.445390][   T96] GPR04: 2b3f0780 2b3f07b0 
>  
> [1.445390][   T96] GPR08: 2b3f06e8  
>  
> [1.445390][   T96] GPR12:  7fffb683a930 
> 100f0ff8 
> [1.445390][   T96] GPR16:  7fffd9d12020 
> 2b3f0780 
> [1.445390][   T96] GPR20: 2b3f0778 2b3f1330 
>  100c6cb0
> [1.445390][   T96] GPR24:   
>  
> [1.445390][   T96] GPR28: 100d34ae 100c6cf8 
> 2b3f0780 2b3f06e8
> [1.446042][   T96]

Re: [PATCH 4/4] powerpc/64s/radix: combine final TLB flush and lazy tlb mm shootdown IPIs

2023-07-17 Thread Michael Ellerman 
Nicholas Piggin  writes:
> This performs lazy tlb mm shootdown when doing the exit TLB flush when
> all mm users go away and user mappings are removed, which avoids having
> to do the lazy tlb mm shootdown IPIs on the final mmput when all kernel
> references disappear.
>
> powerpc/64s uses a broadcast TLBIE for the exit TLB flush if remote CPUs
> need to be invalidated (unless TLBIE is disabled), so this doesn't
> necessarily save IPIs but it does avoid a broadcast TLBIE which is quite
> expensive.
>
> Signed-off-by: Nicholas Piggin 
> ---
>  arch/powerpc/mm/book3s64/radix_tlb.c | 26 +-
>  1 file changed, 25 insertions(+), 1 deletion(-)

This gives me:

[1.438910][T1] Run /init as init process
[1.442759][   T96] [ cut here ]
[1.442836][   T96] WARNING: CPU: 0 PID: 96 at kernel/smp.c:748 
smp_call_function_many_cond+0xe0/0xad0
[1.442920][   T96] Modules linked in:
[1.442960][   T96] CPU: 0 PID: 96 Comm: init Not tainted 
6.5.0-rc2-g1954d181ea09 #168
[1.443028][   T96] Hardware name: IBM pSeries (emulated by qemu) POWER9 
(raw) 0x4e1202 0xf05 of:SLOF,git-6b6c16 hv:linux,kvm pSeries
[1.443126][   T96] NIP:  c02aab20 LR: c00a5fc4 CTR: 

[1.443199][   T96] REGS: cc36f5b0 TRAP: 0700   Not tainted  
(6.5.0-rc2-g1954d181ea09)
[1.443280][   T96] MSR:  80029033   CR: 
44008244  XER: 2004
[1.443382][   T96] CFAR: c02ab524 IRQMASK: 0
[1.443382][   T96] GPR00: c00a5fc4 cc36f850 
c17f9000 c617c580
[1.443382][   T96] GPR04: c00a55b0 c617bd00 
0001 0001
[1.443382][   T96] GPR08: c29fc88c cc25aa00 
 44008244
[1.443382][   T96] GPR12: fd78 c36c 
 c4042a00
[1.443382][   T96] GPR16: 0001  
 
[1.443382][   T96] GPR20:  c00a5fc4 
 c29f85d0
[1.443382][   T96] GPR24: cc25b518  
c617be60 c617bd00
[1.443382][   T96] GPR28: c617c580 c00a55b0 
 
[1.443994][   T96] NIP [c02aab20] 
smp_call_function_many_cond+0xe0/0xad0
[1.444069][   T96] LR [c00a5fc4] radix__tlb_flush+0xf4/0x190
[1.444133][   T96] Call Trace:
[1.444172][   T96] [cc36f850] [] 0x 
(unreliable)
[1.444250][   T96] [cc36f920] [c29f7fe0] 
__cpu_possible_mask+0x0/0x100
[1.444326][   T96] [cc36f950] [c04f346c] 
tlb_finish_mmu+0x16c/0x220
[1.02][   T96] [cc36f980] [c04ee894] 
exit_mmap+0x1b4/0x580
[1.74][   T96] [cc36faa0] [c014c140] __mmput+0x60/0x1c0
[1.444546][   T96] [cc36fae0] [c05cf014] 
begin_new_exec+0x5d4/0xec0
[1.444622][   T96] [cc36fb60] [c066c6e8] 
load_elf_binary+0x4a8/0x1cf0
[1.444697][   T96] [cc36fc60] [c05cc410] 
bprm_execve+0x3b0/0xa60
[1.444773][   T96] [cc36fd30] [c05ce3a0] 
do_execveat_common+0x1d0/0x300
[1.444852][   T96] [cc36fde0] [c05ce524] 
sys_execve+0x54/0x70
[1.444928][   T96] [cc36fe10] [c0031c24] 
system_call_exception+0x134/0x360
[1.445000][   T96] [cc36fe50] [c000d6a0] 
system_call_common+0x160/0x2c4
[1.445070][   T96] --- interrupt: c00 at 0x7fffb664cc98
[1.445119][   T96] NIP:  7fffb664cc98 LR: 1004bcb0 CTR: 

[1.445189][   T96] REGS: cc36fe80 TRAP: 0c00   Not tainted  
(6.5.0-rc2-g1954d181ea09)
[1.445271][   T96] MSR:  8280f033 
  CR: 22004842  XER: 
[1.445390][   T96] IRQMASK: 0
[1.445390][   T96] GPR00: 000b 7fffd9d11ec0 
7fffb6767300 2b3f06e8
[1.445390][   T96] GPR04: 2b3f0780 2b3f07b0 
 
[1.445390][   T96] GPR08: 2b3f06e8  
 
[1.445390][   T96] GPR12:  7fffb683a930 
100f0ff8 
[1.445390][   T96] GPR16:  7fffd9d12020 
2b3f0780 
[1.445390][   T96] GPR20: 2b3f0778 2b3f1330 
 100c6cb0
[1.445390][   T96] GPR24:   
 
[1.445390][   T96] GPR28: 100d34ae 100c6cf8 
2b3f0780 2b3f06e8
[1.446042][   T96] NIP [7fffb664cc98] 0x7fffb664cc98
[1.446095][   T96] LR [1004bcb0] 0x1004bcb0
[1.446147][   T96] --- interrupt: c00
[1.446186][   T96] Code: 8149 394a0001 9149 e8ed0030 3d420097 
394ae900 7cea382e 8149 2c07 394a 9149

[PATCH 4/4] powerpc/64s/radix: combine final TLB flush and lazy tlb mm shootdown IPIs

2023-05-24 Thread Nicholas Piggin 
This performs lazy tlb mm shootdown when doing the exit TLB flush when
all mm users go away and user mappings are removed, which avoids having
to do the lazy tlb mm shootdown IPIs on the final mmput when all kernel
references disappear.

powerpc/64s uses a broadcast TLBIE for the exit TLB flush if remote CPUs
need to be invalidated (unless TLBIE is disabled), so this doesn't
necessarily save IPIs but it does avoid a broadcast TLBIE which is quite
expensive.

Signed-off-by: Nicholas Piggin 
---
 arch/powerpc/mm/book3s64/radix_tlb.c | 26 +-
 1 file changed, 25 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c 
b/arch/powerpc/mm/book3s64/radix_tlb.c
index 8160c1630c3d..e2aaee6df1f6 100644
--- a/arch/powerpc/mm/book3s64/radix_tlb.c
+++ b/arch/powerpc/mm/book3s64/radix_tlb.c
@@ -1301,7 +1301,31 @@ void radix__tlb_flush(struct mmu_gather *tlb)
 * See the comment for radix in arch_exit_mmap().
 */
if (tlb->fullmm) {
-   __flush_all_mm(mm, true);
+   if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_SHOOTDOWN)) {
+   /*
+* Shootdown based lazy tlb mm refcounting means we
+* have to IPI everyone in the mm_cpumask anyway soon
+* when the mm goes away, so might as well do it as
+* part of the final flush now.
+*
+* If lazy shootdown was improved to reduce IPIs (e.g.,
+* by batching), then it may end up being better to use
+* tlbies here instead.
+*/
+   smp_mb(); /* see radix__flush_tlb_mm */
+   exit_flush_lazy_tlbs(mm);
+   _tlbiel_pid(mm->context.id, RIC_FLUSH_ALL);
+
+   /*
+* It should not be possible to have coprocessors still
+* attached here.
+*/
+   if (WARN_ON_ONCE(atomic_read(>context.copros) > 0))
+   __flush_all_mm(mm, true);
+   } else {
+   __flush_all_mm(mm, true);
+   }
+
} else if ( (psize = radix_get_mmu_psize(page_size)) == -1) {
if (!tlb->freed_tables)
radix__flush_tlb_mm(mm);
-- 
2.40.1