Re: [PATCH] powerpc/64: Fix kernel stack 16-byte alignment

[Michael Ellerman]

Sandipan Das  writes:

> On 15/11/18 8:04 AM, Nicholas Piggin wrote:
>> Commit 4c2de74cc869 ("powerpc/64: Interrupts save PPR on stack rather
>> than thread_struct") changed sizeof(struct pt_regs) % 16 from 0 to 8,
>> which causes the interrupt frame allocation on kernel entry to put the
>> kernel stack out of alignment.
>> 
>> Add a pad field to fix alignment, and add a BUILD_BUG_ON to catch this
>> in future.
>> 
>> Fixes: 4c2de74cc869 ("powerpc/64: Interrupts save PPR on stack rather
>> than thread_struct")
>> Signed-off-by: Nicholas Piggin 
>> [...]
>
> Thanks for fixing this. Commit 4c2de74cc869 ("powerpc/64: Interrupts save
> PPR on stack rather than thread_struct") was also leading to incorrect
> kernel stack traces.

OK thanks.

In future if you see something weird like that please report it to the
list.

cheers

> E.g.
> If you are using `perf record -g` and expect to see a stack trace like this:
>
> c02b19a0 bpf_check+0x1910 
> (/usr/lib/debug/lib/modules/4.18.17-200.fc28.ppc64le/vmlinux)
> c02a5554 bpf_prog_load+0x684 
> (/usr/lib/debug/lib/modules/4.18.17-200.fc28.ppc64le/vmlinux)
> c02a6938 sys_bpf+0xaf8 
> (/usr/lib/debug/lib/modules/4.18.17-200.fc28.ppc64le/vmlinux)
> c000b9e4 system_call+0x5c 
> (/usr/lib/debug/lib/modules/4.18.17-200.fc28.ppc64le/vmlinux)
> 7fff815bca90 syscall+0x50 (/usr/lib64/libc-2.27.so)
> 7fff7d96ee0c bpf_prog_load+0x16c (/usr/lib64/libbcc.so.0.7.0)
> [...]
>
> you would instead see something like this:
>
> c02bdb88 bpf_check+0xb88 
> (/lib/modules/4.20.0-rc1+/build/vmlinux)
> c02bdb60 bpf_check+0xb60 
> (/lib/modules/4.20.0-rc1+/build/vmlinux)
> 3fff8f350a90 syscall+0x50 (/usr/lib64/libc-2.27.so)
> 3fff8b788e0c bpf_prog_load+0x16c (/usr/lib64/libbcc.so.0.7.0)
> [...]
>
> --
> With Regards,
> Sandipan

Re: [PATCH] powerpc/64: Fix kernel stack 16-byte alignment

[Sandipan Das]

On 15/11/18 8:04 AM, Nicholas Piggin wrote:
> Commit 4c2de74cc869 ("powerpc/64: Interrupts save PPR on stack rather
> than thread_struct") changed sizeof(struct pt_regs) % 16 from 0 to 8,
> which causes the interrupt frame allocation on kernel entry to put the
> kernel stack out of alignment.
> 
> Add a pad field to fix alignment, and add a BUILD_BUG_ON to catch this
> in future.
> 
> Fixes: 4c2de74cc869 ("powerpc/64: Interrupts save PPR on stack rather
> than thread_struct")
> Signed-off-by: Nicholas Piggin 
> [...]

Thanks for fixing this. Commit 4c2de74cc869 ("powerpc/64: Interrupts save
PPR on stack rather than thread_struct") was also leading to incorrect
kernel stack traces.

E.g.
If you are using `perf record -g` and expect to see a stack trace like this:

c02b19a0 bpf_check+0x1910 
(/usr/lib/debug/lib/modules/4.18.17-200.fc28.ppc64le/vmlinux)
c02a5554 bpf_prog_load+0x684 
(/usr/lib/debug/lib/modules/4.18.17-200.fc28.ppc64le/vmlinux)
c02a6938 sys_bpf+0xaf8 
(/usr/lib/debug/lib/modules/4.18.17-200.fc28.ppc64le/vmlinux)
c000b9e4 system_call+0x5c 
(/usr/lib/debug/lib/modules/4.18.17-200.fc28.ppc64le/vmlinux)
7fff815bca90 syscall+0x50 (/usr/lib64/libc-2.27.so)
7fff7d96ee0c bpf_prog_load+0x16c (/usr/lib64/libbcc.so.0.7.0)
[...]

you would instead see something like this:

c02bdb88 bpf_check+0xb88 
(/lib/modules/4.20.0-rc1+/build/vmlinux)
c02bdb60 bpf_check+0xb60 
(/lib/modules/4.20.0-rc1+/build/vmlinux)
3fff8f350a90 syscall+0x50 (/usr/lib64/libc-2.27.so)
3fff8b788e0c bpf_prog_load+0x16c (/usr/lib64/libbcc.so.0.7.0)
[...]

--
With Regards,
Sandipan

Re: [PATCH] powerpc/64: Fix kernel stack 16-byte alignment

[Abdul Haleem]

On Thu, 2018-11-15 at 12:34 +1000, Nicholas Piggin wrote:
> Commit 4c2de74cc869 ("powerpc/64: Interrupts save PPR on stack rather
> than thread_struct") changed sizeof(struct pt_regs) % 16 from 0 to 8,
> which causes the interrupt frame allocation on kernel entry to put the
> kernel stack out of alignment.
> 
> Add a pad field to fix alignment, and add a BUILD_BUG_ON to catch this
> in future.
> 
> Fixes: 4c2de74cc869 ("powerpc/64: Interrupts save PPR on stack rather
> than thread_struct")
> Signed-off-by: Nicholas Piggin 
> ---
>  arch/powerpc/include/asm/ptrace.h | 1 +
>  arch/powerpc/kernel/setup_64.c| 2 ++
>  2 files changed, 3 insertions(+)
> 
> diff --git a/arch/powerpc/include/asm/ptrace.h 
> b/arch/powerpc/include/asm/ptrace.h
> index f73886a1a7f5..1513292bf046 100644
> --- a/arch/powerpc/include/asm/ptrace.h
> +++ b/arch/powerpc/include/asm/ptrace.h
> @@ -54,6 +54,7 @@ struct pt_regs
> 
>  #ifdef CONFIG_PPC64
>   unsigned long ppr;
> + unsigned long pad;  /* Maintain 16 byte interrupt stack alignment */
>  #endif
>  };
>  #endif
> diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c
> index 2a51e4cc8246..236c1151a3a7 100644
> --- a/arch/powerpc/kernel/setup_64.c
> +++ b/arch/powerpc/kernel/setup_64.c
> @@ -636,6 +636,8 @@ static void *__init alloc_stack(unsigned long limit, int 
> cpu)
>  {
>   unsigned long pa;
> 
> + BUILD_BUG_ON(STACK_INT_FRAME_SIZE % 16);
> +
>   pa = memblock_alloc_base_nid(THREAD_SIZE, THREAD_SIZE, limit,
>   early_cpu_to_node(cpu), MEMBLOCK_NONE);
>   if (!pa) {

Tested-by: Abdul Haleem 

This patch fixes hard lockup warnings at opal_interrupt on our P8 system
seen when bnx2x module load unload.

watchdog: CPU 80 self-detected hard LOCKUP @ opal_interrupt+0x34/0x90
watchdog: CPU 80 TB:2355493195232, last heartbeat TB:2350368214474 (10009ms ago)
Modules linked in: iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4
nf_nat xt_conntrack nf_conntrack nf_defrag_ipv4 ipt_REJECT
nf_reject_ipv4 xt_tcpudp tun bridge stp llc iptable_filter dm_mirror
dm_region_hash dm_log dm_service_time vmx_crypto powernv_rng rng_core
dm_multipath kvm_hv kvm nfsd binfmt_misc ip_tables x_tables autofs4 xfs
lpfc bnx2x crc_t10dif crct10dif_generic nvme_fc mdio nvme libcrc32c
nvme_fabrics nvme_core crct10dif_common
CPU: 80 PID: 411 Comm: ksoftirqd/80 Not tainted 
4.20.0-rc2-next-20181114-autotest-autotest #1
NIP:  c009ffc4 LR: c0095d28 CTR: 30032a00
REGS: c03fff48fd78 TRAP: 0900   Not tainted  
(4.20.0-rc2-next-20181114-autotest-autotest)
MSR:  90009033   CR: 4822  XER: 
CFAR: c0095d14 IRQMASK: 1
GPR00: 30005128 c03fff70bd70 c10dcd00 
GPR04: 4822 c009ffc4 90009033 0090
GPR08:   c0095d3c 90001003
GPR12: c0095d14 c03fff7ff300 c03c95080c80 0001
GPR16:  003c c03c95080800 c03c95030e00
GPR20: 0001 0001 0002 0014
GPR24: c03ca4981000 c03fff70be84  
GPR28:  0014 c03ca4981000 c03ca4970200
NIP [c009ffc4] opal_interrupt+0x34/0x90
LR [c0095d28] opal_return+0x14/0x48
Call Trace:
[c03fff70bd70] [c009ffc4] opal_interrupt+0x34/0x90 (unreliable)
[c03fff70bda0] [c01756d0] __handle_irq_event_percpu+0x90/0x2d0
[c03fff70be60] [c0175948] handle_irq_event_percpu+0x38/0xb0
[c03fff70bea0] [c0175a20] handle_irq_event+0x60/0xc0
[c03fff70bed0] [c017aedc] handle_fasteoi_irq+0xbc/0x1f0
[c03fff70bf00] [c0173ec4] generic_handle_irq+0x44/0x70
[c03fff70bf20] [c0019eac] __do_irq+0x8c/0x200
[c03fff70bf90] [c002c744] call_do_irq+0x14/0x24
[c03ca96f7320] [c001a0b4] do_IRQ+0x94/0x110
[c03ca96f7370] [c0008d54] hardware_interrupt_common+0x154/0x160
--- interrupt: 501 at ip_route_input_rcu+0x510/0x950
LR = ip_route_input_rcu+0x220/0x950
[c03ca96f7768] [c092a69c] ip_route_input_noref+0x2c/0x60
[c03ca96f77b8] [c09735b0] arp_process.constprop.14+0x3f0/0x8c0
[c03ca96f78d8] [c08c7508] __netif_receive_skb_one_core+0x68/0xa0
[c03ca96f7918] [c08d09bc] netif_receive_skb_internal+0x3c/0x130
[c03ca96f7968] [c08d153c] napi_gro_receive+0x11c/0x1c0
[c03ca96f79a8] [c07232bc] tg3_poll_work+0x60c/0x10a0
[c03ca96f7ab8] [c0723da4] tg3_poll_msix+0x54/0x210
[c03ca96f7b08] [c08d1f0c] net_rx_action+0x35c/0x4c0
[c03ca96f7c28] [c0a2d8ac] __do_softirq+0x16c/0x3e0
[c03ca96f7d28] [c0102df4] run_ksoftirqd+0x54/0x80
[c03ca96f7d48] [c012d830] smpboot_thread_fn+0x2b0/0x2c0
[c03ca96f7db8] [c0127d9c] kthread+0x15c/0x1a0
[c03ca96f7e28]