Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
On Fri, Dec 20, 2019 at 12:32:06PM +0100, Jason A. Donenfeld wrote: > On Thu, Dec 19, 2019 at 2:08 PM Cédric Le Goater wrote:> > > There is a patch for msgsndp in my tree you could try : > > > > https://github.com/legoater/qemu/tree/powernv-5.0 > > > > Currently being reviewed. I have to address some remarks from David before > > it can be merged. > > Thanks. I've cherry-picked > https://github.com/legoater/qemu/commit/910c9ea5ecc and can confirm > that I no longer receive the crashes. QEMU 5.1 or 5.0.1 release, I > guess? Unless the revision takes much longer than I expect, I'd anticipate it in 5.0. -- David Gibson| I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson signature.asc Description: PGP signature
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
On 12/20/19 12:32 PM, Jason A. Donenfeld wrote: > On Thu, Dec 19, 2019 at 2:08 PM Cédric Le Goater wrote:> >> There is a patch for msgsndp in my tree you could try : >> >> https://github.com/legoater/qemu/tree/powernv-5.0 >> >> Currently being reviewed. I have to address some remarks from David before >> it can be merged. > > Thanks. I've cherry-picked > https://github.com/legoater/qemu/commit/910c9ea5ecc and can confirm > that I no longer receive the crashes. QEMU 5.1 or 5.0.1 release, I > guess? QEMU 5.1 I would say. That's enough time to address the comments. Michael's patch to drop self IPIs is also interesting for Linux. I wonder how we can reach that point. Thanks, C.
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
On Thu, Dec 19, 2019 at 2:08 PM Cédric Le Goater wrote:> > There is a patch for msgsndp in my tree you could try : > > https://github.com/legoater/qemu/tree/powernv-5.0 > > Currently being reviewed. I have to address some remarks from David before > it can be merged. Thanks. I've cherry-picked https://github.com/legoater/qemu/commit/910c9ea5ecc and can confirm that I no longer receive the crashes. QEMU 5.1 or 5.0.1 release, I guess? Jason
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
On Thu, Dec 19, 2019 at 1:52 PM Michael Ellerman wrote:
> diff --git a/arch/powerpc/kernel/dbell.c b/arch/powerpc/kernel/dbell.c
> index f17ff1200eaa..e45cb9bba193 100644
> --- a/arch/powerpc/kernel/dbell.c
> +++ b/arch/powerpc/kernel/dbell.c
> @@ -63,7 +63,7 @@ int doorbell_try_core_ipi(int cpu)
> int this_cpu = get_cpu();
> int ret = 0;
>
> - if (cpumask_test_cpu(cpu, cpu_sibling_mask(this_cpu))) {
> + if (cpu != this_cpu && cpumask_test_cpu(cpu,
> cpu_sibling_mask(this_cpu))) {
> doorbell_core_ipi(cpu);
> ret = 1;
> }
I realize the best solution is that nice powernv branch that will
eventually be merged for qemu/tcg. But, maybe the above would be a
decent idea to upstream? It seems like that's a case of a superfluous
doorbell?
Jason
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
On Thu, Dec 19, 2019 at 02:08:29PM +0100, Cédric Le Goater wrote: > On 19/12/2019 13:45, Michael Ellerman wrote: > > "Jason A. Donenfeld" writes: > >> Hi folks, > >> > >> I'm actually still experiencing this sporadically in the WireGuard test > >> suite, which you can see being run on https://build.wireguard.com/ . > > > > Fancy dashboard you got there :) > > > >> About 50% of the time the powerpc64 build will fail at a place like this: > >> > >> [ 65.147823] Oops: Exception in kernel mode, sig: 4 [#1] > >> [ 65.149198] LE PAGE_SIZE=4K MMU=Hash PREEMPT SMP NR_CPUS=4 NUMA pSeries > >> [ 65.149595] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.5.0-rc1+ #1 > >> [ 65.149745] NIP: c000 LR: c007eda0 CTR: > >> c007ed80 > >> [ 65.149934] REGS: c0a47970 TRAP: 0700 Not tainted > >> (5.5.0-rc1+) > >> [ 65.150032] MSR: 8288b033 > > >> CR: 48008288 XER: > >> [ 65.150352] CFAR: c00332bc IRQMASK: 1 > >> [ 65.150352] GPR00: c0036508 c0a47c00 c0a4c100 > >> 0001 > >> [ 65.150352] GPR04: c0a50998 c0a50908 > >> 0f509000 > >> [ 65.150352] GPR08: 2800 > >> cff24f00 > >> [ 65.150352] GPR12: c007ed80 c0ad9000 > >> > >> [ 65.150352] GPR16: 008c9190 008c94a8 008c92f8 > >> 008c98b0 > >> [ 65.150352] GPR20: 008f2f88 fffd 0014 > >> 00e6c100 > >> [ 65.150352] GPR24: 00e6c100 0001 > >> c0a50998 > >> [ 65.150352] GPR28: c0a9e280 c0a50aa4 0002 > >> > >> [ 65.151591] NIP [c000] doorbell_try_core_ipi+0xd0/0xf0 > >> [ 65.151750] LR [c007eda0] smp_pseries_cause_ipi+0x20/0x70 > >> [ 65.151913] Call Trace: > >> [ 65.152109] [c0a47c00] [c00c7c9c] > >> _nohz_idle_balance+0xbc/0x300 (unreliable) > >> [ 65.152370] [c0a47c30] [c0036508] > >> smp_send_reschedule+0x98/0xb0 > >> [ 65.152711] [c0a47c50] [c00c1634] kick_ilb+0x114/0x140 > >> [ 65.152962] [c0a47ca0] [c00c86d8] > >> newidle_balance+0x4e8/0x500 > >> [ 65.153213] [c0a47d20] [c00c8788] > >> pick_next_task_fair+0x48/0x3a0 > >> [ 65.153424] [c0a47d80] [c0466620] __schedule+0xf0/0x430 > >> [ 65.153612] [c0a47de0] [c0466b04] > >> schedule_idle+0x34/0x70 > >> [ 65.153786] [c0a47e10] [c00c0bc8] do_idle+0x1a8/0x220 > >> [ 65.154121] [c0a47e70] [c00c0e94] > >> cpu_startup_entry+0x34/0x40 > >> [ 65.154313] [c0a47ea0] [c000ef1c] rest_init+0x10c/0x124 > >> [ 65.154414] [c0a47ee0] [c054] > >> start_kernel+0x568/0x594 > >> [ 65.154585] [c0a47f90] [c000a7cc] > >> start_here_common+0x1c/0x330 > >> [ 65.154854] Instruction dump: > >> [ 65.155191] 38210030 e8010010 7c0803a6 4e800020 3d220004 39295228 > >> 8129 3929 > >> [ 65.155498] 7d284038 7c0004ac 5508017e 65082800 <7c00411c> e94d0178 > >> 812a 3929 > > ^ > > Again the faulting instruction there is "msgsndp r8" > > > >> [ 65.156155] ---[ end trace 6180d12e268ffdaf ]--- > >> [ 65.185452] > >> [ 66.187490] Kernel panic - not syncing: Fatal exception > >> > >> This is with "qemu-system-ppc64 -smp 4 -machine pseries" on QEMU 4.0.0. > >> > >> I'm not totally sure what's going on here. I'm emulating a pseries, and > >> using that with qemu's pseries model, and I see that selecting the > >> pseries forces the selection of 'config PPC_DOORBELL' (twice in the same > >> section, actually). > > > > Noted. > > > >> Then inside the kernel there appears to be some runtime CPU check for > >> doorbell support. > > > > Not really. The kernel looks at the CPU revision (PVR) and decides that > > it has doorbell support. > > > >> Is this a case in which QEMU is advertising doorbell support that TCG > >> doesn't have? Or is something else happening here? > > > > It's a gap in the emulation I guess. qemu doesn't emulate msgsndp, but > > it really should because that's a supported instruction since Power8. > > There is a patch for msgsndp in my tree you could try : > > https://github.com/legoater/qemu/tree/powernv-5.0 > > Currently being reviewed. I have to address some remarks from David before > it can be merged. Right. It needs some polish, but I expect we'll have this merged in the not too distant future. > > I suspect msgsndp wasn't implemented for TCG because TCG doesn't support > > more than one thread per core, and you can only send doorbells to other > > threads in the same core, and therefore there is no reason to ever use > > msgsndp. > > There is a need now with KVM
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
On 19/12/2019 13:45, Michael Ellerman wrote: > "Jason A. Donenfeld" writes: >> Hi folks, >> >> I'm actually still experiencing this sporadically in the WireGuard test >> suite, which you can see being run on https://build.wireguard.com/ . > > Fancy dashboard you got there :) > >> About 50% of the time the powerpc64 build will fail at a place like this: >> >> [ 65.147823] Oops: Exception in kernel mode, sig: 4 [#1] >> [ 65.149198] LE PAGE_SIZE=4K MMU=Hash PREEMPT SMP NR_CPUS=4 NUMA pSeries >> [ 65.149595] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.5.0-rc1+ #1 >> [ 65.149745] NIP: c000 LR: c007eda0 CTR: >> c007ed80 >> [ 65.149934] REGS: c0a47970 TRAP: 0700 Not tainted (5.5.0-rc1+) >> [ 65.150032] MSR: 8288b033 > >> CR: 48008288 XER: >> [ 65.150352] CFAR: c00332bc IRQMASK: 1 >> [ 65.150352] GPR00: c0036508 c0a47c00 c0a4c100 >> 0001 >> [ 65.150352] GPR04: c0a50998 c0a50908 >> 0f509000 >> [ 65.150352] GPR08: 2800 >> cff24f00 >> [ 65.150352] GPR12: c007ed80 c0ad9000 >> >> [ 65.150352] GPR16: 008c9190 008c94a8 008c92f8 >> 008c98b0 >> [ 65.150352] GPR20: 008f2f88 fffd 0014 >> 00e6c100 >> [ 65.150352] GPR24: 00e6c100 0001 >> c0a50998 >> [ 65.150352] GPR28: c0a9e280 c0a50aa4 0002 >> >> [ 65.151591] NIP [c000] doorbell_try_core_ipi+0xd0/0xf0 >> [ 65.151750] LR [c007eda0] smp_pseries_cause_ipi+0x20/0x70 >> [ 65.151913] Call Trace: >> [ 65.152109] [c0a47c00] [c00c7c9c] >> _nohz_idle_balance+0xbc/0x300 (unreliable) >> [ 65.152370] [c0a47c30] [c0036508] >> smp_send_reschedule+0x98/0xb0 >> [ 65.152711] [c0a47c50] [c00c1634] kick_ilb+0x114/0x140 >> [ 65.152962] [c0a47ca0] [c00c86d8] >> newidle_balance+0x4e8/0x500 >> [ 65.153213] [c0a47d20] [c00c8788] >> pick_next_task_fair+0x48/0x3a0 >> [ 65.153424] [c0a47d80] [c0466620] __schedule+0xf0/0x430 >> [ 65.153612] [c0a47de0] [c0466b04] schedule_idle+0x34/0x70 >> [ 65.153786] [c0a47e10] [c00c0bc8] do_idle+0x1a8/0x220 >> [ 65.154121] [c0a47e70] [c00c0e94] >> cpu_startup_entry+0x34/0x40 >> [ 65.154313] [c0a47ea0] [c000ef1c] rest_init+0x10c/0x124 >> [ 65.154414] [c0a47ee0] [c054] start_kernel+0x568/0x594 >> [ 65.154585] [c0a47f90] [c000a7cc] >> start_here_common+0x1c/0x330 >> [ 65.154854] Instruction dump: >> [ 65.155191] 38210030 e8010010 7c0803a6 4e800020 3d220004 39295228 >> 8129 3929 >> [ 65.155498] 7d284038 7c0004ac 5508017e 65082800 <7c00411c> e94d0178 >> 812a 3929 > ^ > Again the faulting instruction there is "msgsndp r8" > >> [ 65.156155] ---[ end trace 6180d12e268ffdaf ]--- >> [ 65.185452] >> [ 66.187490] Kernel panic - not syncing: Fatal exception >> >> This is with "qemu-system-ppc64 -smp 4 -machine pseries" on QEMU 4.0.0. >> >> I'm not totally sure what's going on here. I'm emulating a pseries, and >> using that with qemu's pseries model, and I see that selecting the >> pseries forces the selection of 'config PPC_DOORBELL' (twice in the same >> section, actually). > > Noted. > >> Then inside the kernel there appears to be some runtime CPU check for >> doorbell support. > > Not really. The kernel looks at the CPU revision (PVR) and decides that > it has doorbell support. > >> Is this a case in which QEMU is advertising doorbell support that TCG >> doesn't have? Or is something else happening here? > > It's a gap in the emulation I guess. qemu doesn't emulate msgsndp, but > it really should because that's a supported instruction since Power8. There is a patch for msgsndp in my tree you could try : https://github.com/legoater/qemu/tree/powernv-5.0 Currently being reviewed. I have to address some remarks from David before it can be merged. > I suspect msgsndp wasn't implemented for TCG because TCG doesn't support > more than one thread per core, and you can only send doorbells to other > threads in the same core, and therefore there is no reason to ever use > msgsndp. There is a need now with KVM emulation under TCG, but, yes, QEMU still lacks SMT support. > That's the message Suraj mentioned up thread, eg: > > $ qemu-system-ppc64 -nographic -vga none -M pseries -smp 2,threads=2 -cpu > POWER8 -kernel build~/vmlinux > qemu-system-ppc64: TCG cannot support more than 1 thread/core on a pseries > machine > > > But I guess we've hit another case of a CPU sending itself an
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
"Jason A. Donenfeld" writes:
> Hi folks,
>
> I'm actually still experiencing this sporadically in the WireGuard test
> suite, which you can see being run on https://build.wireguard.com/ .
Fancy dashboard you got there :)
> About 50% of the time the powerpc64 build will fail at a place like this:
>
> [ 65.147823] Oops: Exception in kernel mode, sig: 4 [#1]
> [ 65.149198] LE PAGE_SIZE=4K MMU=Hash PREEMPT SMP NR_CPUS=4 NUMA pSeries
> [ 65.149595] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.5.0-rc1+ #1
> [ 65.149745] NIP: c000 LR: c007eda0 CTR:
> c007ed80
> [ 65.149934] REGS: c0a47970 TRAP: 0700 Not tainted (5.5.0-rc1+)
> [ 65.150032] MSR: 8288b033 > CR:
> 48008288 XER:
> [ 65.150352] CFAR: c00332bc IRQMASK: 1
> [ 65.150352] GPR00: c0036508 c0a47c00 c0a4c100
> 0001
> [ 65.150352] GPR04: c0a50998 c0a50908
> 0f509000
> [ 65.150352] GPR08: 2800
> cff24f00
> [ 65.150352] GPR12: c007ed80 c0ad9000
>
> [ 65.150352] GPR16: 008c9190 008c94a8 008c92f8
> 008c98b0
> [ 65.150352] GPR20: 008f2f88 fffd 0014
> 00e6c100
> [ 65.150352] GPR24: 00e6c100 0001
> c0a50998
> [ 65.150352] GPR28: c0a9e280 c0a50aa4 0002
>
> [ 65.151591] NIP [c000] doorbell_try_core_ipi+0xd0/0xf0
> [ 65.151750] LR [c007eda0] smp_pseries_cause_ipi+0x20/0x70
> [ 65.151913] Call Trace:
> [ 65.152109] [c0a47c00] [c00c7c9c]
> _nohz_idle_balance+0xbc/0x300 (unreliable)
> [ 65.152370] [c0a47c30] [c0036508]
> smp_send_reschedule+0x98/0xb0
> [ 65.152711] [c0a47c50] [c00c1634] kick_ilb+0x114/0x140
> [ 65.152962] [c0a47ca0] [c00c86d8]
> newidle_balance+0x4e8/0x500
> [ 65.153213] [c0a47d20] [c00c8788]
> pick_next_task_fair+0x48/0x3a0
> [ 65.153424] [c0a47d80] [c0466620] __schedule+0xf0/0x430
> [ 65.153612] [c0a47de0] [c0466b04] schedule_idle+0x34/0x70
> [ 65.153786] [c0a47e10] [c00c0bc8] do_idle+0x1a8/0x220
> [ 65.154121] [c0a47e70] [c00c0e94]
> cpu_startup_entry+0x34/0x40
> [ 65.154313] [c0a47ea0] [c000ef1c] rest_init+0x10c/0x124
> [ 65.154414] [c0a47ee0] [c054] start_kernel+0x568/0x594
> [ 65.154585] [c0a47f90] [c000a7cc]
> start_here_common+0x1c/0x330
> [ 65.154854] Instruction dump:
> [ 65.155191] 38210030 e8010010 7c0803a6 4e800020 3d220004 39295228 8129
> 3929
> [ 65.155498] 7d284038 7c0004ac 5508017e 65082800 <7c00411c> e94d0178
> 812a 3929
^
Again the faulting instruction there is "msgsndp r8"
> [ 65.156155] ---[ end trace 6180d12e268ffdaf ]---
> [ 65.185452]
> [ 66.187490] Kernel panic - not syncing: Fatal exception
>
> This is with "qemu-system-ppc64 -smp 4 -machine pseries" on QEMU 4.0.0.
>
> I'm not totally sure what's going on here. I'm emulating a pseries, and
> using that with qemu's pseries model, and I see that selecting the
> pseries forces the selection of 'config PPC_DOORBELL' (twice in the same
> section, actually).
Noted.
> Then inside the kernel there appears to be some runtime CPU check for
> doorbell support.
Not really. The kernel looks at the CPU revision (PVR) and decides that
it has doorbell support.
> Is this a case in which QEMU is advertising doorbell support that TCG
> doesn't have? Or is something else happening here?
It's a gap in the emulation I guess. qemu doesn't emulate msgsndp, but
it really should because that's a supported instruction since Power8.
I suspect msgsndp wasn't implemented for TCG because TCG doesn't support
more than one thread per core, and you can only send doorbells to other
threads in the same core, and therefore there is no reason to ever use
msgsndp.
That's the message Suraj mentioned up thread, eg:
$ qemu-system-ppc64 -nographic -vga none -M pseries -smp 2,threads=2 -cpu
POWER8 -kernel build~/vmlinux
qemu-system-ppc64: TCG cannot support more than 1 thread/core on a pseries
machine
But I guess we've hit another case of a CPU sending itself an IPI, and
the way the sibling masks are done, CPUs are siblings of themselves, so
the sibling test passes, eg:
int doorbell_try_core_ipi(int cpu)
{
int this_cpu = get_cpu();
int ret = 0;
if (cpumask_test_cpu(cpu, cpu_sibling_mask(this_cpu))) {
doorbell_core_ipi(cpu);
In which case this patch should fix it.
diff --git a/arch/powerpc/kernel/dbell.c b/arch/powerpc/kernel/dbell.c
index f17ff1200eaa..e45cb9bba193 100644
---
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
On Thu, Dec 19, 2019 at 12:13 PM Sebastian Andrzej Siewior wrote: > Based on my understanding is that the doorbell feature is part of the > architecture. It can be used to signal other siblings on the same CPU. > qemu TCG doesn't support that and does not allow to announce multiple > siblings on the same CPU. However, the kernel uses this interface if it > tries to send an interrupt to itself (the same CPU) so everything > matches. > Last time I run into this, the interface was change so the kernel das > not send an IPI to itself. This changed now for another function… One way of fixing this is to just "not use the feature", as you seem to be suggesting. But actually shouldn't there be some CPU feature detection available? Something like -- QEMU advertises to the kernel that it supports or doesn't support doorbells, and the kernel then avoids those paths if the CPU feature flag isn't present? Jason
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
On 2019-12-19 11:41:21 [+0100], Jason A. Donenfeld wrote:
> Hi folks,
Hi,
so this should duct tape it:
diff --git a/arch/powerpc/kernel/dbell.c b/arch/powerpc/kernel/dbell.c
index f17ff1200eaae..ec044bdf362a1 100644
--- a/arch/powerpc/kernel/dbell.c
+++ b/arch/powerpc/kernel/dbell.c
@@ -60,16 +60,8 @@ void doorbell_core_ipi(int cpu)
*/
int doorbell_try_core_ipi(int cpu)
{
- int this_cpu = get_cpu();
int ret = 0;
- if (cpumask_test_cpu(cpu, cpu_sibling_mask(this_cpu))) {
- doorbell_core_ipi(cpu);
- ret = 1;
- }
-
- put_cpu();
-
return ret;
}
> This is with "qemu-system-ppc64 -smp 4 -machine pseries" on QEMU 4.0.0.
Interesting. I didn't get v5.4 to boot a while ago and didn't have the
time to look into it.
> I'm not totally sure what's going on here. I'm emulating a pseries, and
> using that with qemu's pseries model, and I see that selecting the pseries
> forces the selection of 'config PPC_DOORBELL' (twice in the same section,
> actually). Then inside the kernel there appears to be some runtime CPU check
> for doorbell support. Is this a case in which QEMU is advertising doorbell
> support that TCG doesn't have? Or is something else happening here?
Based on my understanding is that the doorbell feature is part of the
architecture. It can be used to signal other siblings on the same CPU.
qemu TCG doesn't support that and does not allow to announce multiple
siblings on the same CPU. However, the kernel uses this interface if it
tries to send an interrupt to itself (the same CPU) so everything
matches.
Last time I run into this, the interface was change so the kernel das
not send an IPI to itself. This changed now for another function…
> Thanks,
> Jason
Sebastian
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
Hi folks, I'm actually still experiencing this sporadically in the WireGuard test suite, which you can see being run on https://build.wireguard.com/ . About 50% of the time the powerpc64 build will fail at a place like this: [ 65.147823] Oops: Exception in kernel mode, sig: 4 [#1] [ 65.149198] LE PAGE_SIZE=4K MMU=Hash PREEMPT SMP NR_CPUS=4 NUMA pSeries [ 65.149595] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.5.0-rc1+ #1 [ 65.149745] NIP: c000 LR: c007eda0 CTR: c007ed80 [ 65.149934] REGS: c0a47970 TRAP: 0700 Not tainted (5.5.0-rc1+) [ 65.150032] MSR: 8288b033 CR: 48008288 XER: [ 65.150352] CFAR: c00332bc IRQMASK: 1 [ 65.150352] GPR00: c0036508 c0a47c00 c0a4c100 0001 [ 65.150352] GPR04: c0a50998 c0a50908 0f509000 [ 65.150352] GPR08: 2800 cff24f00 [ 65.150352] GPR12: c007ed80 c0ad9000 [ 65.150352] GPR16: 008c9190 008c94a8 008c92f8 008c98b0 [ 65.150352] GPR20: 008f2f88 fffd 0014 00e6c100 [ 65.150352] GPR24: 00e6c100 0001 c0a50998 [ 65.150352] GPR28: c0a9e280 c0a50aa4 0002 [ 65.151591] NIP [c000] doorbell_try_core_ipi+0xd0/0xf0 [ 65.151750] LR [c007eda0] smp_pseries_cause_ipi+0x20/0x70 [ 65.151913] Call Trace: [ 65.152109] [c0a47c00] [c00c7c9c] _nohz_idle_balance+0xbc/0x300 (unreliable) [ 65.152370] [c0a47c30] [c0036508] smp_send_reschedule+0x98/0xb0 [ 65.152711] [c0a47c50] [c00c1634] kick_ilb+0x114/0x140 [ 65.152962] [c0a47ca0] [c00c86d8] newidle_balance+0x4e8/0x500 [ 65.153213] [c0a47d20] [c00c8788] pick_next_task_fair+0x48/0x3a0 [ 65.153424] [c0a47d80] [c0466620] __schedule+0xf0/0x430 [ 65.153612] [c0a47de0] [c0466b04] schedule_idle+0x34/0x70 [ 65.153786] [c0a47e10] [c00c0bc8] do_idle+0x1a8/0x220 [ 65.154121] [c0a47e70] [c00c0e94] cpu_startup_entry+0x34/0x40 [ 65.154313] [c0a47ea0] [c000ef1c] rest_init+0x10c/0x124 [ 65.154414] [c0a47ee0] [c054] start_kernel+0x568/0x594 [ 65.154585] [c0a47f90] [c000a7cc] start_here_common+0x1c/0x330 [ 65.154854] Instruction dump: [ 65.155191] 38210030 e8010010 7c0803a6 4e800020 3d220004 39295228 8129 3929 [ 65.155498] 7d284038 7c0004ac 5508017e 65082800 <7c00411c> e94d0178 812a 3929 [ 65.156155] ---[ end trace 6180d12e268ffdaf ]--- [ 65.185452] [ 66.187490] Kernel panic - not syncing: Fatal exception This is with "qemu-system-ppc64 -smp 4 -machine pseries" on QEMU 4.0.0. I'm not totally sure what's going on here. I'm emulating a pseries, and using that with qemu's pseries model, and I see that selecting the pseries forces the selection of 'config PPC_DOORBELL' (twice in the same section, actually). Then inside the kernel there appears to be some runtime CPU check for doorbell support. Is this a case in which QEMU is advertising doorbell support that TCG doesn't have? Or is something else happening here? Thanks, Jason
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
Frederic Weisbecker's on April 6, 2019 10:06 am:
> On Mon, Apr 01, 2019 at 10:38:27AM +0200, Peter Zijlstra wrote:
>>
>> + fweisbec, who did the remote bits
>>
>> On Sat, Mar 30, 2019 at 01:10:28PM +1000, Nicholas Piggin wrote:
>> > diff --git a/kernel/irq_work.c b/kernel/irq_work.c
>> > index 6b7cdf17ccf8..f0e539d0f879 100644
>> > --- a/kernel/irq_work.c
>> > +++ b/kernel/irq_work.c
>> > -/* Enqueue the irq work @work on the current CPU */
>> > -bool irq_work_queue(struct irq_work *work)
>> > +/*
>> > + * Enqueue the irq_work @work on @cpu unless it's already pending
>> > + * somewhere.
>> > + *
>> > + * Can be re-enqueued while the callback is still in progress.
>> > + */
>> > +bool irq_work_queue_on(struct irq_work *work, int cpu)
>> > {
>> > +#ifndef CONFIG_SMP
>> > + return irq_work_queue(work);
>> > +
>
> I'd suggest to use "if (!IS_ENABLED(CONFIG_SMP))" here to avoid the large
> ifdeffery.
Sadly you can't do it because arch_send_call_function_single_ipi is
not defined for !SMP. I made your suggested name change though (with
the __ prefix because work needs to be claimed and preempt disabled).
Thanks,
Nick
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
On Mon, Apr 01, 2019 at 10:38:27AM +0200, Peter Zijlstra wrote:
>
> + fweisbec, who did the remote bits
>
> On Sat, Mar 30, 2019 at 01:10:28PM +1000, Nicholas Piggin wrote:
> > diff --git a/kernel/irq_work.c b/kernel/irq_work.c
> > index 6b7cdf17ccf8..f0e539d0f879 100644
> > --- a/kernel/irq_work.c
> > +++ b/kernel/irq_work.c
> > -/* Enqueue the irq work @work on the current CPU */
> > -bool irq_work_queue(struct irq_work *work)
> > +/*
> > + * Enqueue the irq_work @work on @cpu unless it's already pending
> > + * somewhere.
> > + *
> > + * Can be re-enqueued while the callback is still in progress.
> > + */
> > +bool irq_work_queue_on(struct irq_work *work, int cpu)
> > {
> > +#ifndef CONFIG_SMP
> > + return irq_work_queue(work);
> > +
I'd suggest to use "if (!IS_ENABLED(CONFIG_SMP))" here to avoid the large
ifdeffery.
> > +#else /* #ifndef CONFIG_SMP */
> > + /* All work should have been flushed before going offline */
> > + WARN_ON_ONCE(cpu_is_offline(cpu));
> > +
> > /* Only queue if not already pending */
> > if (!irq_work_claim(work))
> > return false;
> >
> > - /* Queue the entry and raise the IPI if needed. */
> > preempt_disable();
> > -
> > - /* If the work is "lazy", handle it from next tick if any */
> > - if (work->flags & IRQ_WORK_LAZY) {
> > - if (llist_add(>llnode, this_cpu_ptr(_list)) &&
> > - tick_nohz_tick_stopped())
> > - arch_irq_work_raise();
> > - } else {
> > - if (llist_add(>llnode, this_cpu_ptr(_list)))
> > - arch_irq_work_raise();
> > - }
> > -
> > + if (cpu != smp_processor_id()) {
> > + /* Arch remote IPI send/receive backend aren't NMI safe */
> > + WARN_ON_ONCE(in_nmi());
> > + if (llist_add(>llnode, _cpu(raised_list, cpu)))
> > + arch_send_call_function_single_ipi(cpu);
> > + } else
> > + __irq_work_queue(work);
Also perhaps rename __irq_work_queue() to irq_work_queue_local() to make it
instantly clearer to reviewers.
Other than those cosmetic changes,
Reviewed-by: Frederic Weisbecker
Thanks.
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
On 2019-04-05 02:25:44 [+1000], Nicholas Piggin wrote: > Sebastian, are you able to test if this patch solves your problem? yes, it does. Tested-by: Sebastian Andrzej Siewior > Thanks, > Nick Sebastian
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
Peter Zijlstra's on April 1, 2019 6:38 pm: > > + fweisbec, who did the remote bits > > On Sat, Mar 30, 2019 at 01:10:28PM +1000, Nicholas Piggin wrote: >> Something like this? >> >> kernel/irq_work: Do not raise an IPI when queueing work on the local CPU >> >> The QEMU powerpc/pseries machine model was not expecting a self-IPI, >> and it may be a bit surprising thing to do, so have irq_work_queue_on >> do local queueing when target is current CPU. > > This seems OK to me. > >> Suggested-by: Steven Rostedt >> Signed-off-by: Nicholas Piggin > > Acked-by: Peter Zijlstra (Intel) Thanks for taking a look. Sebastian, are you able to test if this patch solves your problem? Thanks, Nick
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
+ fweisbec, who did the remote bits
On Sat, Mar 30, 2019 at 01:10:28PM +1000, Nicholas Piggin wrote:
> Something like this?
>
> kernel/irq_work: Do not raise an IPI when queueing work on the local CPU
>
> The QEMU powerpc/pseries machine model was not expecting a self-IPI,
> and it may be a bit surprising thing to do, so have irq_work_queue_on
> do local queueing when target is current CPU.
This seems OK to me.
> Suggested-by: Steven Rostedt
> Signed-off-by: Nicholas Piggin
Acked-by: Peter Zijlstra (Intel)
> ---
> kernel/irq_work.c | 78 ++-
> 1 file changed, 43 insertions(+), 35 deletions(-)
>
> diff --git a/kernel/irq_work.c b/kernel/irq_work.c
> index 6b7cdf17ccf8..f0e539d0f879 100644
> --- a/kernel/irq_work.c
> +++ b/kernel/irq_work.c
> @@ -56,61 +56,69 @@ void __weak arch_irq_work_raise(void)
>*/
> }
>
> -/*
> - * Enqueue the irq_work @work on @cpu unless it's already pending
> - * somewhere.
> - *
> - * Can be re-enqueued while the callback is still in progress.
> - */
> -bool irq_work_queue_on(struct irq_work *work, int cpu)
> +/* Enqueue on current CPU, work must already be claimed and preempt disabled
> */
> +static void __irq_work_queue(struct irq_work *work)
> {
> - /* All work should have been flushed before going offline */
> - WARN_ON_ONCE(cpu_is_offline(cpu));
> -
> -#ifdef CONFIG_SMP
> -
> - /* Arch remote IPI send/receive backend aren't NMI safe */
> - WARN_ON_ONCE(in_nmi());
> + /* If the work is "lazy", handle it from next tick if any */
> + if (work->flags & IRQ_WORK_LAZY) {
> + if (llist_add(>llnode, this_cpu_ptr(_list)) &&
> + tick_nohz_tick_stopped())
> + arch_irq_work_raise();
> + } else {
> + if (llist_add(>llnode, this_cpu_ptr(_list)))
> + arch_irq_work_raise();
> + }
> +}
>
> +/* Enqueue the irq work @work on the current CPU */
> +bool irq_work_queue(struct irq_work *work)
> +{
> /* Only queue if not already pending */
> if (!irq_work_claim(work))
> return false;
>
> - if (llist_add(>llnode, _cpu(raised_list, cpu)))
> - arch_send_call_function_single_ipi(cpu);
> -
> -#else /* #ifdef CONFIG_SMP */
> - irq_work_queue(work);
> -#endif /* #else #ifdef CONFIG_SMP */
> + /* Queue the entry and raise the IPI if needed. */
> + preempt_disable();
> + __irq_work_queue(work);
> + preempt_enable();
>
> return true;
> }
> +EXPORT_SYMBOL_GPL(irq_work_queue);
>
> -/* Enqueue the irq work @work on the current CPU */
> -bool irq_work_queue(struct irq_work *work)
> +/*
> + * Enqueue the irq_work @work on @cpu unless it's already pending
> + * somewhere.
> + *
> + * Can be re-enqueued while the callback is still in progress.
> + */
> +bool irq_work_queue_on(struct irq_work *work, int cpu)
> {
> +#ifndef CONFIG_SMP
> + return irq_work_queue(work);
> +
> +#else /* #ifndef CONFIG_SMP */
> + /* All work should have been flushed before going offline */
> + WARN_ON_ONCE(cpu_is_offline(cpu));
> +
> /* Only queue if not already pending */
> if (!irq_work_claim(work))
> return false;
>
> - /* Queue the entry and raise the IPI if needed. */
> preempt_disable();
> -
> - /* If the work is "lazy", handle it from next tick if any */
> - if (work->flags & IRQ_WORK_LAZY) {
> - if (llist_add(>llnode, this_cpu_ptr(_list)) &&
> - tick_nohz_tick_stopped())
> - arch_irq_work_raise();
> - } else {
> - if (llist_add(>llnode, this_cpu_ptr(_list)))
> - arch_irq_work_raise();
> - }
> -
> + if (cpu != smp_processor_id()) {
> + /* Arch remote IPI send/receive backend aren't NMI safe */
> + WARN_ON_ONCE(in_nmi());
> + if (llist_add(>llnode, _cpu(raised_list, cpu)))
> + arch_send_call_function_single_ipi(cpu);
> + } else
> + __irq_work_queue(work);
> preempt_enable();
>
> return true;
> +#endif /* #else #ifndef CONFIG_SMP */
> }
> -EXPORT_SYMBOL_GPL(irq_work_queue);
> +
>
> bool irq_work_needs_cpu(void)
> {
> --
> 2.20.1
>
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
Steven Rostedt's on March 30, 2019 1:31 am:
> [ Added Peter ]
>
> On Fri, 29 Mar 2019 19:13:55 +1000
> Nicholas Piggin wrote:
>
>> Suraj Jitindar Singh's on March 29, 2019 3:20 pm:
>> > On Wed, 2019-03-27 at 17:51 +0100, Cédric Le Goater wrote:
>> >> On 3/27/19 5:37 PM, Cédric Le Goater wrote:
>> >> > On 3/27/19 1:36 PM, Sebastian Andrzej Siewior wrote:
>> >> > > With qemu-system-ppc64le -machine pseries -smp 4 I get:
>> >> > >
>> >> > > > # chrt 1 hackbench
>> >> > > > Running in process mode with 10 groups using 40 file
>> >> > > > descriptors each (== 400 tasks)
>> >> > > > Each sender will pass 100 messages of 100 bytes
>> >> > > > Oops: Exception in kernel mode, sig: 4 [#1]
>> >> > > > LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=2048 NUMA pSeries
>> >> > > > Modules linked in:
>> >> > > > CPU: 0 PID: 629 Comm: hackbench Not tainted 5.1.0-rc2 #71
>> >> > > > NIP: c0046978 LR: c0046a38 CTR:
>> >> > > > c00b0150
>> >> > > > REGS: c001fffeb8e0 TRAP: 0700 Not tainted (5.1.0-rc2)
>> >> > > > MSR: 80089033 CR:
>> >> > > > 42000874 XER:
>> >> > > > CFAR: c0046a34 IRQMASK: 1
>> >> > > > GPR00: c00b0170 c001fffebb70 c0a6ba00
>> >> > > > 2800
>> >> > >
>> >> > > …
>> >> > > > NIP [c0046978] doorbell_core_ipi+0x28/0x30
>> >> > > > LR [c0046a38] doorbell_try_core_ipi+0xb8/0xf0
>> >> > > > Call Trace:
>> >> > > > [c001fffebb70] [c001fffebba0] 0xc001fffebba0
>> >> > > > (unreliable)
>> >> > > > [c001fffebba0] [c00b0170]
>> >> > > > smp_pseries_cause_ipi+0x20/0x70
>> >> > > > [c001fffebbd0] [c004b02c]
>> >> > > > arch_send_call_function_single_ipi+0x8c/0xa0
>> >> > > > [c001fffebbf0] [c01de600]
>> >> > > > irq_work_queue_on+0xe0/0x130
>> >> > > > [c001fffebc30] [c01340c8]
>> >> > > > rto_push_irq_work_func+0xc8/0x120
>> >> > >
>> >> > > …
>> >> > > > Instruction dump:
>> >> > > > 6000 6000 3c4c00a2 384250b0 3d220009 392949c8 8129
>> >> > > > 3929
>> >> > > > 7d231838 7c0004ac 5463017e 64632800 <7c00191c> 4e800020
>> >> > > > 3c4c00a2 38425080
>> >> > > > ---[ end trace eb842b544538cbdf ]---
>>
>> This is unusual and causing powerpc code to crash because the rt
>> scheduler is telling irq_work_queue_on to queue work on this CPU.
>> Is that something allowed? There's no warnings in there but it must
>> be a rarely tested path, would it be better to ban it?
>>
>> Steven is this queue_work_on to self by design?
>
> I just figured that a irq_work_queue_on() would default to just
> irq_work_queue() if cpu == smp_processor_id().
Yeah it actually doesn't, it raises a CALL_FUNCTION IPI on the
current CPU. Maybe it should do that.
>
> I'm sure this case has been tested (on x86), as I stressed this quite a
> bit, and all it takes is to have an RT task queued on the CPU
> processing the current "push" logic when it's not holding the rq locks
> and read that the current CPU now has an overloaded RT task.
>
> If calling irq_work_queue_on() is really bad to do to the same CPU,
> then we can work on changing the logic to do something different if the
> CPU returned from rto_next_cpu() is the current CPU.
I don't actually know that it is bad. It does seem to violate the
principle of least surprise for arch code though.
>From this bug report, I'm guessing it's pretty rare case to hit so
I would worry about subtle bugs. And I just cc'ed you in case your
code was actually not intending to do this.
> Or would it be possible to just change irq_work_queue_on() to call
> irq_work_queue() if cpu == smp_processor_id()?
Something like this?
kernel/irq_work: Do not raise an IPI when queueing work on the local CPU
The QEMU powerpc/pseries machine model was not expecting a self-IPI,
and it may be a bit surprising thing to do, so have irq_work_queue_on
do local queueing when target is current CPU.
Suggested-by: Steven Rostedt
Signed-off-by: Nicholas Piggin
---
kernel/irq_work.c | 78 ++-
1 file changed, 43 insertions(+), 35 deletions(-)
diff --git a/kernel/irq_work.c b/kernel/irq_work.c
index 6b7cdf17ccf8..f0e539d0f879 100644
--- a/kernel/irq_work.c
+++ b/kernel/irq_work.c
@@ -56,61 +56,69 @@ void __weak arch_irq_work_raise(void)
*/
}
-/*
- * Enqueue the irq_work @work on @cpu unless it's already pending
- * somewhere.
- *
- * Can be re-enqueued while the callback is still in progress.
- */
-bool irq_work_queue_on(struct irq_work *work, int cpu)
+/* Enqueue on current CPU, work must already be claimed and preempt disabled */
+static void __irq_work_queue(struct irq_work *work)
{
- /* All work should have been flushed before going offline */
- WARN_ON_ONCE(cpu_is_offline(cpu));
-
-#ifdef CONFIG_SMP
-
- /* Arch remote IPI send/receive backend aren't NMI safe */
- WARN_ON_ONCE(in_nmi());
+ /* If the work is "lazy", handle it from next
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
[ Added Peter ] On Fri, 29 Mar 2019 19:13:55 +1000 Nicholas Piggin wrote: > Suraj Jitindar Singh's on March 29, 2019 3:20 pm: > > On Wed, 2019-03-27 at 17:51 +0100, Cédric Le Goater wrote: > >> On 3/27/19 5:37 PM, Cédric Le Goater wrote: > >> > On 3/27/19 1:36 PM, Sebastian Andrzej Siewior wrote: > >> > > With qemu-system-ppc64le -machine pseries -smp 4 I get: > >> > > > >> > > > # chrt 1 hackbench > >> > > > Running in process mode with 10 groups using 40 file > >> > > > descriptors each (== 400 tasks) > >> > > > Each sender will pass 100 messages of 100 bytes > >> > > > Oops: Exception in kernel mode, sig: 4 [#1] > >> > > > LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=2048 NUMA pSeries > >> > > > Modules linked in: > >> > > > CPU: 0 PID: 629 Comm: hackbench Not tainted 5.1.0-rc2 #71 > >> > > > NIP: c0046978 LR: c0046a38 CTR: > >> > > > c00b0150 > >> > > > REGS: c001fffeb8e0 TRAP: 0700 Not tainted (5.1.0-rc2) > >> > > > MSR: 80089033 CR: > >> > > > 42000874 XER: > >> > > > CFAR: c0046a34 IRQMASK: 1 > >> > > > GPR00: c00b0170 c001fffebb70 c0a6ba00 > >> > > > 2800 > >> > > > >> > > … > >> > > > NIP [c0046978] doorbell_core_ipi+0x28/0x30 > >> > > > LR [c0046a38] doorbell_try_core_ipi+0xb8/0xf0 > >> > > > Call Trace: > >> > > > [c001fffebb70] [c001fffebba0] 0xc001fffebba0 > >> > > > (unreliable) > >> > > > [c001fffebba0] [c00b0170] > >> > > > smp_pseries_cause_ipi+0x20/0x70 > >> > > > [c001fffebbd0] [c004b02c] > >> > > > arch_send_call_function_single_ipi+0x8c/0xa0 > >> > > > [c001fffebbf0] [c01de600] > >> > > > irq_work_queue_on+0xe0/0x130 > >> > > > [c001fffebc30] [c01340c8] > >> > > > rto_push_irq_work_func+0xc8/0x120 > >> > > > >> > > … > >> > > > Instruction dump: > >> > > > 6000 6000 3c4c00a2 384250b0 3d220009 392949c8 8129 > >> > > > 3929 > >> > > > 7d231838 7c0004ac 5463017e 64632800 <7c00191c> 4e800020 > >> > > > 3c4c00a2 38425080 > >> > > > ---[ end trace eb842b544538cbdf ]--- > > This is unusual and causing powerpc code to crash because the rt > scheduler is telling irq_work_queue_on to queue work on this CPU. > Is that something allowed? There's no warnings in there but it must > be a rarely tested path, would it be better to ban it? > > Steven is this queue_work_on to self by design? I just figured that a irq_work_queue_on() would default to just irq_work_queue() if cpu == smp_processor_id(). I'm sure this case has been tested (on x86), as I stressed this quite a bit, and all it takes is to have an RT task queued on the CPU processing the current "push" logic when it's not holding the rq locks and read that the current CPU now has an overloaded RT task. If calling irq_work_queue_on() is really bad to do to the same CPU, then we can work on changing the logic to do something different if the CPU returned from rto_next_cpu() is the current CPU. Or would it be possible to just change irq_work_queue_on() to call irq_work_queue() if cpu == smp_processor_id()? -- Steve > > >> > > > >> > > and I was wondering whether this is a qemu bug or the kernel is > >> > > using an > >> > > opcode it should rather not. If I skip doorbell_try_core_ipi() in > >> > > smp_pseries_cause_ipi() then there is no crash. The comment says > >> > > "POWER9 > >> > > should not use this handler" so… > >> > > >> > I would say Linux is using a msgsndp instruction which is not > >> > implemented > >> > in QEMU TCG. But why have we started using dbells in Linux ? > > > > Yeah the kernel must have used msgsndp which isn't implemented for TCG > > yet. We use doorbells in linux but only for threads which are on the > > same core. > > And when I try to construct a situation with more than 1 thread per > > core (e.g. -smp 4,threads=4), I get "TCG cannot support more than 1 > > thread/core on a pseries machine". > > > > So I wonder why the guest thinks it can use msgsndp... > > IPI to self evidently. Under TCG it really should implement the > instruction or remove the DBELL feature. > > Thanks, > Nick >
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
Suraj Jitindar Singh's on March 29, 2019 3:20 pm: > On Wed, 2019-03-27 at 17:51 +0100, Cédric Le Goater wrote: >> On 3/27/19 5:37 PM, Cédric Le Goater wrote: >> > On 3/27/19 1:36 PM, Sebastian Andrzej Siewior wrote: >> > > With qemu-system-ppc64le -machine pseries -smp 4 I get: >> > > >> > > > # chrt 1 hackbench >> > > > Running in process mode with 10 groups using 40 file >> > > > descriptors each (== 400 tasks) >> > > > Each sender will pass 100 messages of 100 bytes >> > > > Oops: Exception in kernel mode, sig: 4 [#1] >> > > > LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=2048 NUMA pSeries >> > > > Modules linked in: >> > > > CPU: 0 PID: 629 Comm: hackbench Not tainted 5.1.0-rc2 #71 >> > > > NIP: c0046978 LR: c0046a38 CTR: >> > > > c00b0150 >> > > > REGS: c001fffeb8e0 TRAP: 0700 Not tainted (5.1.0-rc2) >> > > > MSR: 80089033 CR: >> > > > 42000874 XER: >> > > > CFAR: c0046a34 IRQMASK: 1 >> > > > GPR00: c00b0170 c001fffebb70 c0a6ba00 >> > > > 2800 >> > > >> > > … >> > > > NIP [c0046978] doorbell_core_ipi+0x28/0x30 >> > > > LR [c0046a38] doorbell_try_core_ipi+0xb8/0xf0 >> > > > Call Trace: >> > > > [c001fffebb70] [c001fffebba0] 0xc001fffebba0 >> > > > (unreliable) >> > > > [c001fffebba0] [c00b0170] >> > > > smp_pseries_cause_ipi+0x20/0x70 >> > > > [c001fffebbd0] [c004b02c] >> > > > arch_send_call_function_single_ipi+0x8c/0xa0 >> > > > [c001fffebbf0] [c01de600] >> > > > irq_work_queue_on+0xe0/0x130 >> > > > [c001fffebc30] [c01340c8] >> > > > rto_push_irq_work_func+0xc8/0x120 >> > > >> > > … >> > > > Instruction dump: >> > > > 6000 6000 3c4c00a2 384250b0 3d220009 392949c8 8129 >> > > > 3929 >> > > > 7d231838 7c0004ac 5463017e 64632800 <7c00191c> 4e800020 >> > > > 3c4c00a2 38425080 >> > > > ---[ end trace eb842b544538cbdf ]--- This is unusual and causing powerpc code to crash because the rt scheduler is telling irq_work_queue_on to queue work on this CPU. Is that something allowed? There's no warnings in there but it must be a rarely tested path, would it be better to ban it? Steven is this queue_work_on to self by design? >> > > >> > > and I was wondering whether this is a qemu bug or the kernel is >> > > using an >> > > opcode it should rather not. If I skip doorbell_try_core_ipi() in >> > > smp_pseries_cause_ipi() then there is no crash. The comment says >> > > "POWER9 >> > > should not use this handler" so… >> > >> > I would say Linux is using a msgsndp instruction which is not >> > implemented >> > in QEMU TCG. But why have we started using dbells in Linux ? > > Yeah the kernel must have used msgsndp which isn't implemented for TCG > yet. We use doorbells in linux but only for threads which are on the > same core. > And when I try to construct a situation with more than 1 thread per > core (e.g. -smp 4,threads=4), I get "TCG cannot support more than 1 > thread/core on a pseries machine". > > So I wonder why the guest thinks it can use msgsndp... IPI to self evidently. Under TCG it really should implement the instruction or remove the DBELL feature. Thanks, Nick
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
On 2019-03-29 16:20:51 [+1100], Suraj Jitindar Singh wrote: > > Yeah the kernel must have used msgsndp which isn't implemented for TCG > yet. We use doorbells in linux but only for threads which are on the > same core. It is msgsndp as per instruction decode. > And when I try to construct a situation with more than 1 thread per > core (e.g. -smp 4,threads=4), I get "TCG cannot support more than 1 > thread/core on a pseries machine". > > So I wonder why the guest thinks it can use msgsndp... I see |# cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list |0 |1 |2 |3 so this works all well. The problem is when a CPU sends an IPI to itself then linux's doorbell_try_core_ipi() considers this as a valid sibling and here we have the msgsndp. Sebastian
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
On Wed, 2019-03-27 at 17:51 +0100, Cédric Le Goater wrote: > On 3/27/19 5:37 PM, Cédric Le Goater wrote: > > On 3/27/19 1:36 PM, Sebastian Andrzej Siewior wrote: > > > With qemu-system-ppc64le -machine pseries -smp 4 I get: > > > > > > > # chrt 1 hackbench > > > > Running in process mode with 10 groups using 40 file > > > > descriptors each (== 400 tasks) > > > > Each sender will pass 100 messages of 100 bytes > > > > Oops: Exception in kernel mode, sig: 4 [#1] > > > > LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=2048 NUMA pSeries > > > > Modules linked in: > > > > CPU: 0 PID: 629 Comm: hackbench Not tainted 5.1.0-rc2 #71 > > > > NIP: c0046978 LR: c0046a38 CTR: > > > > c00b0150 > > > > REGS: c001fffeb8e0 TRAP: 0700 Not tainted (5.1.0-rc2) > > > > MSR: 80089033 CR: > > > > 42000874 XER: > > > > CFAR: c0046a34 IRQMASK: 1 > > > > GPR00: c00b0170 c001fffebb70 c0a6ba00 > > > > 2800 > > > > > > … > > > > NIP [c0046978] doorbell_core_ipi+0x28/0x30 > > > > LR [c0046a38] doorbell_try_core_ipi+0xb8/0xf0 > > > > Call Trace: > > > > [c001fffebb70] [c001fffebba0] 0xc001fffebba0 > > > > (unreliable) > > > > [c001fffebba0] [c00b0170] > > > > smp_pseries_cause_ipi+0x20/0x70 > > > > [c001fffebbd0] [c004b02c] > > > > arch_send_call_function_single_ipi+0x8c/0xa0 > > > > [c001fffebbf0] [c01de600] > > > > irq_work_queue_on+0xe0/0x130 > > > > [c001fffebc30] [c01340c8] > > > > rto_push_irq_work_func+0xc8/0x120 > > > > > > … > > > > Instruction dump: > > > > 6000 6000 3c4c00a2 384250b0 3d220009 392949c8 8129 > > > > 3929 > > > > 7d231838 7c0004ac 5463017e 64632800 <7c00191c> 4e800020 > > > > 3c4c00a2 38425080 > > > > ---[ end trace eb842b544538cbdf ]--- > > > > > > and I was wondering whether this is a qemu bug or the kernel is > > > using an > > > opcode it should rather not. If I skip doorbell_try_core_ipi() in > > > smp_pseries_cause_ipi() then there is no crash. The comment says > > > "POWER9 > > > should not use this handler" so… > > > > I would say Linux is using a msgsndp instruction which is not > > implemented > > in QEMU TCG. But why have we started using dbells in Linux ? Yeah the kernel must have used msgsndp which isn't implemented for TCG yet. We use doorbells in linux but only for threads which are on the same core. And when I try to construct a situation with more than 1 thread per core (e.g. -smp 4,threads=4), I get "TCG cannot support more than 1 thread/core on a pseries machine". So I wonder why the guest thinks it can use msgsndp... > > ah. It seems arch_local_irq_restore() / replay_interrupt() generated > some interrupt. > > C. >
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
On 3/27/19 5:37 PM, Cédric Le Goater wrote: > On 3/27/19 1:36 PM, Sebastian Andrzej Siewior wrote: >> With qemu-system-ppc64le -machine pseries -smp 4 I get: >> >> |# chrt 1 hackbench >> |Running in process mode with 10 groups using 40 file descriptors each (== >> 400 tasks) >> |Each sender will pass 100 messages of 100 bytes >> | Oops: Exception in kernel mode, sig: 4 [#1] >> | LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=2048 NUMA pSeries >> | Modules linked in: >> | CPU: 0 PID: 629 Comm: hackbench Not tainted 5.1.0-rc2 #71 >> | NIP: c0046978 LR: c0046a38 CTR: c00b0150 >> | REGS: c001fffeb8e0 TRAP: 0700 Not tainted (5.1.0-rc2) >> | MSR: 80089033 CR: 42000874 XER: >> | CFAR: c0046a34 IRQMASK: 1 >> | GPR00: c00b0170 c001fffebb70 c0a6ba00 2800 >> … >> | NIP [c0046978] doorbell_core_ipi+0x28/0x30 >> | LR [c0046a38] doorbell_try_core_ipi+0xb8/0xf0 >> | Call Trace: >> | [c001fffebb70] [c001fffebba0] 0xc001fffebba0 (unreliable) >> | [c001fffebba0] [c00b0170] smp_pseries_cause_ipi+0x20/0x70 >> | [c001fffebbd0] [c004b02c] >> arch_send_call_function_single_ipi+0x8c/0xa0 >> | [c001fffebbf0] [c01de600] irq_work_queue_on+0xe0/0x130 >> | [c001fffebc30] [c01340c8] rto_push_irq_work_func+0xc8/0x120 >> … >> | Instruction dump: >> | 6000 6000 3c4c00a2 384250b0 3d220009 392949c8 8129 3929 >> | 7d231838 7c0004ac 5463017e 64632800 <7c00191c> 4e800020 3c4c00a2 38425080 >> | ---[ end trace eb842b544538cbdf ]--- >> >> and I was wondering whether this is a qemu bug or the kernel is using an >> opcode it should rather not. If I skip doorbell_try_core_ipi() in >> smp_pseries_cause_ipi() then there is no crash. The comment says "POWER9 >> should not use this handler" so… > > I would say Linux is using a msgsndp instruction which is not implemented > in QEMU TCG. But why have we started using dbells in Linux ? ah. It seems arch_local_irq_restore() / replay_interrupt() generated some interrupt. C.
Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()
On 3/27/19 1:36 PM, Sebastian Andrzej Siewior wrote: > With qemu-system-ppc64le -machine pseries -smp 4 I get: > > |# chrt 1 hackbench > |Running in process mode with 10 groups using 40 file descriptors each (== > 400 tasks) > |Each sender will pass 100 messages of 100 bytes > | Oops: Exception in kernel mode, sig: 4 [#1] > | LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=2048 NUMA pSeries > | Modules linked in: > | CPU: 0 PID: 629 Comm: hackbench Not tainted 5.1.0-rc2 #71 > | NIP: c0046978 LR: c0046a38 CTR: c00b0150 > | REGS: c001fffeb8e0 TRAP: 0700 Not tainted (5.1.0-rc2) > | MSR: 80089033 CR: 42000874 XER: > | CFAR: c0046a34 IRQMASK: 1 > | GPR00: c00b0170 c001fffebb70 c0a6ba00 2800 > … > | NIP [c0046978] doorbell_core_ipi+0x28/0x30 > | LR [c0046a38] doorbell_try_core_ipi+0xb8/0xf0 > | Call Trace: > | [c001fffebb70] [c001fffebba0] 0xc001fffebba0 (unreliable) > | [c001fffebba0] [c00b0170] smp_pseries_cause_ipi+0x20/0x70 > | [c001fffebbd0] [c004b02c] > arch_send_call_function_single_ipi+0x8c/0xa0 > | [c001fffebbf0] [c01de600] irq_work_queue_on+0xe0/0x130 > | [c001fffebc30] [c01340c8] rto_push_irq_work_func+0xc8/0x120 > … > | Instruction dump: > | 6000 6000 3c4c00a2 384250b0 3d220009 392949c8 8129 3929 > | 7d231838 7c0004ac 5463017e 64632800 <7c00191c> 4e800020 3c4c00a2 38425080 > | ---[ end trace eb842b544538cbdf ]--- > > and I was wondering whether this is a qemu bug or the kernel is using an > opcode it should rather not. If I skip doorbell_try_core_ipi() in > smp_pseries_cause_ipi() then there is no crash. The comment says "POWER9 > should not use this handler" so… I would say Linux is using a msgsndp instruction which is not implemented in QEMU TCG. But why have we started using dbells in Linux ? C.
