Link to v1: https://lists.gnu.org/archive/html/qemu-ppc/2022-08/msg00370.html
This series is also available as a git branch:
https://github.com/PPC64/qemu/tree/ferst-interrupt-fix-v2
This version addresses Fabiano's feedback and fixes some issues found
with the tests suggested by Cédric. While working on it, I found two
intermittent problems on master:
i) ~10% of boots with pSeries and 970/970mp/POWER5+ hard lockup after
either SCSI or network initialization when using -smp 4. With
-smp 2, the problem is harder to reproduce but still happens, and I
couldn't reproduce with thread=single.
ii) ~52% of KVM guest initializations on PowerNV hang in different parts
of the boot process when using more than one CPU.
With the complete series applied, I couldn't reproduce (i) anymore, and
(ii) became a little more frequent (~58%).
I've tested each patch of this series with [1], modified to use -smp for
machines that support more than one CPU. The machines I can currently
boot with FreeBSD (970/970,p/POWER5+/POWER7/POWER8/POWER9 pSeries,
POWER8/POWER9 PowerNV, and mpc8544ds) were tested with the images from
[2] and still boot after applying the patch series. Booting nested
guests inside a TCG pSeries machine also seems to be working fine.
Using command lines like:
./qemu-system-ppc64 -M powernv9 -cpu POWER9 -accel tcg,thread=multi \
-m 8G -smp $SMP -vga none -nographic -kernel zImage \
-append 'console=hvc0' -initrdootfs.cpio.xz \
-serial pipe:pipe -monitor unix:mon,server,nowait
and
./qemu-system-ppc64 -M pseries -cpu POWER9 -accel tcg,thread=multi \
-m 8G -smp $SMP -vga none -nographic -kernel zImage \
-append 'console=hvc0' -initrd rootfs.cpio.xz \
-serial pipe:pipe -monitor unix:mon,server,nowait
to measure the time to boot, login, and shut down a compressed kernel
with a buildroot initramfs, with 100 iteration we get:
+-----+------------------------------+-----------------------------+
| | PowerNV | pSeries |
|-smp |------------------------------+-----------------------------+
| | master | patch series | master | patch series |
+-----+------------------------------+-----------------------------+
| 1 | 45,84 ± 0,92 | 38,08 ± 0,66 | 23,56 ± 1,16 | 23,76 ± 1,04 |
| 2 | 80,21 ± 8,03 | 40,81 ± 0,45 | 26,59 ± 0,92 | 26,88 ± 0,99 |
| 4 | 115,98 ± 9,85 | 38,80 ± 0,44 | 28,83 ± 0,84 | 28,46 ± 0,94 |
| 6 | 199,14 ± 6,36 | 39,32 ± 0,50 | 29,22 ± 0,78 | 29,45 ± 0,86 |
| 8 | 47,85 ± 27,50 | 38,98 ± 0,49 | 29,63 ± 0,80 | 29,60 ± 0,78 |
+-----+------------------------------+-----------------------------+
This results shows that the problem reported in [3] is solved, while
pSeries boot time is essentially unchanged.
With a non-compressed kernel, the difference with PowerNV is smaller,
and pSeries stills the same:
+-----+------------------------------+-----------------------------+
| | PowerNV | pSeries |
|-smp |------------------------------+-----------------------------+
| | master | patch series | master | patch series |
+-----+------------------------------+-----------------------------+
| 1 | 42,17 ± 0,92 | 38,13 ± 0,59 | 23,15 ± 1,02 | 23,46 ± 1,02 |
| 2 | 55,72 ± 3,54 | 40,30 ± 0,56 | 26,26 ± 0,82 | 26,38 ± 0,80 |
| 4 | 67,09 ± 3,02 | 38,26 ± 0,47 | 28,36 ± 0,77 | 28,19 ± 0,78 |
| 6 | 98,96 ± 2,49 | 39,01 ± 0,38 | 28,68 ± 0,75 | 29,02 ± 0,88 |
| 8 | 39,68 ± 0,42 | 38,44 ± 0,41 | 29,24 ± 0,81 | 29,44 ± 0,75 |
+-----+------------------------------+-----------------------------+
Finally, using command lines like
./qemu-system-ppc64 -M powernv9 -cpu POWER9 -accel tcg,thread=multi \
-m 8G -smp 4 -device virtio-scsi-pci -boot c -vga none -nographic \
-device nvme,bus=pcie.2,addr=0x0,drive=drive0,serial=1234 \
-drive file=rootfs.ext2,if=none,id=drive0,format=raw,cache=none \
-snapshot -serial pipe:pipe -monitor unix:mon,server,nowait \
-kernel zImage -append 'console=hvc0 rootwait root=/dev/nvme0n1' \
-device virtio-net-pci,netdev=br0,mac=52:54:00:12:34:57,bus=pcie.0 \
-netdev bridge,id=br0
and
./qemu-system-ppc64 -M pseries -cpu POWER9 -accel tcg,thread=multi \
-m 8G -smp 4 -device virtio-scsi-pci -boot c -vga none -nographic \
-drive file=rootfs.ext2,if=scsi,index=0,format=raw -snapshot \
-kernel zImage -append 'console=hvc0 rootwait root=/dev/sda' \
-serial pipe:pipe -monitor unix:mon,server,nowait \
-device virtio-net-pci,netdev=br0,mac=52:54:00:12:34:57 \
-netdev bridge,id=br0
to tests IO performance, with iperf to test network and a 4Gb scp
transfer to test disk+network, in 100 iterations we saw:
+---------------------+---------------+-----------------+
| | scp (s) | iperf (MB/s) |
+---------------------+---------------+-----------------+
|PowerNV master | 166,91 ± 8,37 | 918,06 ± 114,78 |
|PowerNV patch series | 166,25 ± 8,85 | 916,91 ± 107,56 |
|pSeries master | 175,70 ± 8,22 | 958,73 ± 115,09 |
|pSeries patch series | 173,62 ± 8,13 | 893,42 ± 87,77 |
+---------------------+---------------+-----------------+
The scp data shows little difference, while testing just network shows
that it's a bit slower with the patch series applied (although, with
this variation, we'd probably need to repeat this test more times to
have a more robust result...)
[1] https://github.com/legoater/qemu-ppc-boot
[2] https://artifact.ci.freebsd.org/snapshot/14.0-CURRENT/latest_vm/powerpc
[3] https://lists.gnu.org/archive/html/qemu-ppc/2022-06/msg00336.html
Matheus Ferst (29):
target/ppc: define PPC_INTERRUPT_* values directly
target/ppc: always use ppc_set_irq to set env->pending_interrupts
target/ppc: split interrupt masking and delivery from ppc_hw_interrupt
target/ppc: prepare to split interrupt masking and delivery by excp_model
target/ppc: create an interrupt masking method for POWER9/POWER10
target/ppc: remove unused interrupts from p9_pending_interrupt
target/ppc: create an interrupt deliver method for POWER9/POWER10
target/ppc: remove unused interrupts from p9_deliver_interrupt
target/ppc: remove generic architecture checks from p9_deliver_interrupt
target/ppc: move power-saving interrupt masking out of cpu_has_work_POWER9
target/ppc: add power-saving interrupt masking logic to
p9_next_unmasked_interrupt
target/ppc: create an interrupt masking method for POWER8
target/ppc: remove unused interrupts from p8_pending_interrupt
target/ppc: create an interrupt deliver method for POWER8
target/ppc: remove unused interrupts from p8_deliver_interrupt
target/ppc: remove generic architecture checks from p8_deliver_interrupt
target/ppc: move power-saving interrupt masking out of cpu_has_work_POWER8
target/ppc: add power-saving interrupt masking logic to
p8_next_unmasked_interrupt
target/ppc: create an interrupt masking method for POWER7
target/ppc: remove unused interrupts from p7_pending_interrupt
target/ppc: create an interrupt deliver method for POWER7
target/ppc: remove unused interrupts from p7_deliver_interrupt
target/ppc: remove generic architecture checks from p7_deliver_interrupt
target/ppc: move power-saving interrupt masking out of cpu_has_work_POWER7
target/ppc: add power-saving interrupt masking logic to
p7_next_unmasked_interrupt
target/ppc: remove ppc_store_lpcr from CONFIG_USER_ONLY builds
target/ppc: introduce ppc_maybe_interrupt
target/ppc: unify cpu->has_work based on cs->interrupt_request
target/ppc: move the p*_interrupt_powersave methods to excp_helper.c
hw/ppc/pnv_core.c | 1 +
hw/ppc/ppc.c | 17 +-
hw/ppc/spapr_hcall.c | 6 +
hw/ppc/spapr_rtas.c | 2 +-
hw/ppc/trace-events | 2 +-
target/ppc/cpu.c | 4 +
target/ppc/cpu.h | 43 +-
target/ppc/cpu_init.c | 212 +---------
target/ppc/excp_helper.c | 857 ++++++++++++++++++++++++++++++++++-----
target/ppc/helper.h | 1 +
target/ppc/helper_regs.c | 2 +
target/ppc/misc_helper.c | 11 +-
target/ppc/translate.c | 2 +
13 files changed, 803 insertions(+), 357 deletions(-)
--
2.25.1
