Re: [PATCH 1/3] ppc/pnv: Tune the POWER9 PCIe Host bridge model
On 26/11/2021 10:09, Cédric Le Goater wrote:
On 11/16/21 18:01, Frederic Barrat wrote:
The PHB v4 found on POWER9 doesn't request any LSI, so let's clear the
Interrupt Pin register in the config space so that the model matches
the hardware.
If we don't, then we inherit from the default pcie root bridge, which
requests a LSI. And because we don't map it correctly in the device
tree, all PHBs allocate the same bogus hw interrupt. We end up with
inconsistent interrupt controller (xive) data. The problem goes away
if we don't allocate the LSI in the first place.
Signed-off-by: Frederic Barrat
---
hw/pci-host/pnv_phb4.c | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/hw/pci-host/pnv_phb4.c b/hw/pci-host/pnv_phb4.c
index 5c375a9f28..1659d55b4f 100644
--- a/hw/pci-host/pnv_phb4.c
+++ b/hw/pci-host/pnv_phb4.c
@@ -1234,10 +1234,13 @@ static void pnv_phb4_reset(DeviceState *dev)
PCIDevice *root_dev = PCI_DEVICE(>root);
/*
- * Configure PCI device id at reset using a property.
+ * Configure the PCI device at reset:
+ * - set the Vendor and Device ID to for the root bridge
+ * - no LSI
*/
pci_config_set_vendor_id(root_dev->config, PCI_VENDOR_ID_IBM);
pci_config_set_device_id(root_dev->config, phb->device_id);
+ pci_config_set_interrupt_pin(root_dev->config, 0);
}
static const char *pnv_phb4_root_bus_path(PCIHostState *host_bridge,
FYI, I am seeing an issue with FreeBSD when booting from iso :
https://download.freebsd.org/ftp/snapshots/powerpc/powerpc64/ISO-IMAGES/14.0/FreeBSD-14.0-CURRENT-powerpc-powerpc64-20211028-4827bf76bce-250301-disc1.iso.xz
I see what's going on... Since the phb4 model borrows most of its code
from the pcie_root bridge, there are several instances of code such as:
if (msix_enabled(dev)) {
do something;
} else if (msi_enabled(dev)) {
do something else;
} else {
yet something else which assumes a LSI;
}
With this series, I removed the LSI from the phb4 root port to match the
hardware and fixed one such code pattern in patch 3. But there are
others, and we hit one of those when installing from the free bsd iso.
So this is going to need more work.
Fred
Thanks,
C.
SIGTERM received, booting...
KDB: debugger backends: ddb
KDB: current backend: ddb
---<>---
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.0-CURRENT #0 main-n250301-4827bf76bce: Thu Oct 28 06:53:58
UTC 2021
r...@releng1.nyi.freebsd.org:/usr/obj/usr/src/powerpc.powerpc64/sys/GENERIC64
powerpc
FreeBSD clang version 12.0.1 (g...@github.com:llvm/llvm-project.git
llvmorg-12.0.1-0-gfed41342a82f)
WARNING: WITNESS option enabled, expect reduced performance.
VT: init without driver.
ofw_initrd: initrd loaded at 0x2800-0x28c7928c
cpu0: IBM POWER9 revision 2.0, 1000.00 MHz
cpu0: Features
dc007182
cpu0: Features2
bee0
real memory = 1014484992 (967 MB)
avail memory = 117903360 (112 MB)
random: registering fast source PowerISA DARN random number generator
random: fast provider: "PowerISA DARN random number generator"
arc4random: WARNING: initial seeding bypassed the cryptographic random
device because it was not yet seeded and the knob
'bypass_before_seeding' was enabled.
random: entropy device external interface
kbd0 at kbdmux0
ofwbus0: on nexus0
opal0: irq
1048560,1048561,1048562,1048563,1048564,1048565,1048566,1048567,1048568,1048569,1048570,1048571,1048572,1048573
on ofwbus0
opal0: registered as a time-of-day clock, resolution 0.002000s
simplebus0: mem
0x60300-0x60300 on ofwbus0
pcib0: mem
0x600c3c000-0x600c3cfff,0x600c3-0x600c30fff on ofwbus0
pci0: numa-domain 0 on pcib0
qemu-system-ppc64: ../hw/pci/pci.c:1487: pci_irq_handler: Assertion `0
<= irq_num && irq_num < PCI_NUM_PINS' failed.
Re: [PATCH 1/3] ppc/pnv: Tune the POWER9 PCIe Host bridge model
On Fri, Nov 26, 2021 at 06:08:30PM +0100, Cédric Le Goater wrote: > [ Adding Alfredo the thread ] > > On 11/26/21 10:09, Cédric Le Goater wrote: > > On 11/16/21 18:01, Frederic Barrat wrote: > > > The PHB v4 found on POWER9 doesn't request any LSI, so let's clear the > > > Interrupt Pin register in the config space so that the model matches > > > the hardware. > > > > > > If we don't, then we inherit from the default pcie root bridge, which > > > requests a LSI. And because we don't map it correctly in the device > > > tree, all PHBs allocate the same bogus hw interrupt. We end up with > > > inconsistent interrupt controller (xive) data. The problem goes away > > > if we don't allocate the LSI in the first place. > > > > > > Signed-off-by: Frederic Barrat > > > --- > > > hw/pci-host/pnv_phb4.c | 5 - > > > 1 file changed, 4 insertions(+), 1 deletion(-) > > > > > > diff --git a/hw/pci-host/pnv_phb4.c b/hw/pci-host/pnv_phb4.c > > > index 5c375a9f28..1659d55b4f 100644 > > > --- a/hw/pci-host/pnv_phb4.c > > > +++ b/hw/pci-host/pnv_phb4.c > > > @@ -1234,10 +1234,13 @@ static void pnv_phb4_reset(DeviceState *dev) > > > PCIDevice *root_dev = PCI_DEVICE(>root); > > > /* > > > - * Configure PCI device id at reset using a property. > > > + * Configure the PCI device at reset: > > > + * - set the Vendor and Device ID to for the root bridge > > > + * - no LSI > > > */ > > > pci_config_set_vendor_id(root_dev->config, PCI_VENDOR_ID_IBM); > > > pci_config_set_device_id(root_dev->config, phb->device_id); > > > + pci_config_set_interrupt_pin(root_dev->config, 0); > > > } > > > static const char *pnv_phb4_root_bus_path(PCIHostState *host_bridge, > > > > > > > FYI, I am seeing an issue with FreeBSD when booting from iso : > > > > > > https://download.freebsd.org/ftp/snapshots/powerpc/powerpc64/ISO-IMAGES/14.0/FreeBSD-14.0-CURRENT-powerpc-powerpc64-20211028-4827bf76bce-250301-disc1.iso.xz > > > > Thanks, > > > > C. > > > > SIGTERM received, booting... > > KDB: debugger backends: ddb > > KDB: current backend: ddb > > ---<>--- > > Copyright (c) 1992-2021 The FreeBSD Project. > > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > > The Regents of the University of California. All rights reserved. > > FreeBSD is a registered trademark of The FreeBSD Foundation. > > FreeBSD 14.0-CURRENT #0 main-n250301-4827bf76bce: Thu Oct 28 06:53:58 UTC > > 2021 > > > > r...@releng1.nyi.freebsd.org:/usr/obj/usr/src/powerpc.powerpc64/sys/GENERIC64 > > powerpc > > FreeBSD clang version 12.0.1 (g...@github.com:llvm/llvm-project.git > > llvmorg-12.0.1-0-gfed41342a82f) > > WARNING: WITNESS option enabled, expect reduced performance. > > VT: init without driver. > > ofw_initrd: initrd loaded at 0x2800-0x28c7928c > > cpu0: IBM POWER9 revision 2.0, 1000.00 MHz > > cpu0: Features > > dc007182 > > cpu0: Features2 bee0 > > real memory = 1014484992 (967 MB) > > avail memory = 117903360 (112 MB) > > random: registering fast source PowerISA DARN random number generator > > random: fast provider: "PowerISA DARN random number generator" > > arc4random: WARNING: initial seeding bypassed the cryptographic random > > device because it was not yet seeded and the knob 'bypass_before_seeding' > > was enabled. > > random: entropy device external interface > > kbd0 at kbdmux0 > > ofwbus0: on nexus0 > > opal0: irq > > 1048560,1048561,1048562,1048563,1048564,1048565,1048566,1048567,1048568,1048569,1048570,1048571,1048572,1048573 > > on ofwbus0 > > opal0: registered as a time-of-day clock, resolution 0.002000s > > simplebus0: mem > > 0x60300-0x60300 on ofwbus0 > > pcib0: mem > > 0x600c3c000-0x600c3cfff,0x600c3-0x600c30fff on ofwbus0 > > pci0: numa-domain 0 on pcib0 > > qemu-system-ppc64: ../hw/pci/pci.c:1487: pci_irq_handler: Assertion `0 <= > > irq_num && irq_num < PCI_NUM_PINS' failed. > > Frederic? -- MST
Re: [PATCH 1/3] ppc/pnv: Tune the POWER9 PCIe Host bridge model
[ Adding Alfredo the thread ] On 11/26/21 10:09, Cédric Le Goater wrote: On 11/16/21 18:01, Frederic Barrat wrote: The PHB v4 found on POWER9 doesn't request any LSI, so let's clear the Interrupt Pin register in the config space so that the model matches the hardware. If we don't, then we inherit from the default pcie root bridge, which requests a LSI. And because we don't map it correctly in the device tree, all PHBs allocate the same bogus hw interrupt. We end up with inconsistent interrupt controller (xive) data. The problem goes away if we don't allocate the LSI in the first place. Signed-off-by: Frederic Barrat --- hw/pci-host/pnv_phb4.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/hw/pci-host/pnv_phb4.c b/hw/pci-host/pnv_phb4.c index 5c375a9f28..1659d55b4f 100644 --- a/hw/pci-host/pnv_phb4.c +++ b/hw/pci-host/pnv_phb4.c @@ -1234,10 +1234,13 @@ static void pnv_phb4_reset(DeviceState *dev) PCIDevice *root_dev = PCI_DEVICE(>root); /* - * Configure PCI device id at reset using a property. + * Configure the PCI device at reset: + * - set the Vendor and Device ID to for the root bridge + * - no LSI */ pci_config_set_vendor_id(root_dev->config, PCI_VENDOR_ID_IBM); pci_config_set_device_id(root_dev->config, phb->device_id); + pci_config_set_interrupt_pin(root_dev->config, 0); } static const char *pnv_phb4_root_bus_path(PCIHostState *host_bridge, FYI, I am seeing an issue with FreeBSD when booting from iso : https://download.freebsd.org/ftp/snapshots/powerpc/powerpc64/ISO-IMAGES/14.0/FreeBSD-14.0-CURRENT-powerpc-powerpc64-20211028-4827bf76bce-250301-disc1.iso.xz Thanks, C. SIGTERM received, booting... KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2021 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 14.0-CURRENT #0 main-n250301-4827bf76bce: Thu Oct 28 06:53:58 UTC 2021 r...@releng1.nyi.freebsd.org:/usr/obj/usr/src/powerpc.powerpc64/sys/GENERIC64 powerpc FreeBSD clang version 12.0.1 (g...@github.com:llvm/llvm-project.git llvmorg-12.0.1-0-gfed41342a82f) WARNING: WITNESS option enabled, expect reduced performance. VT: init without driver. ofw_initrd: initrd loaded at 0x2800-0x28c7928c cpu0: IBM POWER9 revision 2.0, 1000.00 MHz cpu0: Features dc007182 cpu0: Features2 bee0 real memory = 1014484992 (967 MB) avail memory = 117903360 (112 MB) random: registering fast source PowerISA DARN random number generator random: fast provider: "PowerISA DARN random number generator" arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled. random: entropy device external interface kbd0 at kbdmux0 ofwbus0: on nexus0 opal0: irq 1048560,1048561,1048562,1048563,1048564,1048565,1048566,1048567,1048568,1048569,1048570,1048571,1048572,1048573 on ofwbus0 opal0: registered as a time-of-day clock, resolution 0.002000s simplebus0: mem 0x60300-0x60300 on ofwbus0 pcib0: mem 0x600c3c000-0x600c3cfff,0x600c3-0x600c30fff on ofwbus0 pci0: numa-domain 0 on pcib0 qemu-system-ppc64: ../hw/pci/pci.c:1487: pci_irq_handler: Assertion `0 <= irq_num && irq_num < PCI_NUM_PINS' failed.
Re: [PATCH 1/3] ppc/pnv: Tune the POWER9 PCIe Host bridge model
On 11/16/21 18:01, Frederic Barrat wrote: The PHB v4 found on POWER9 doesn't request any LSI, so let's clear the Interrupt Pin register in the config space so that the model matches the hardware. If we don't, then we inherit from the default pcie root bridge, which requests a LSI. And because we don't map it correctly in the device tree, all PHBs allocate the same bogus hw interrupt. We end up with inconsistent interrupt controller (xive) data. The problem goes away if we don't allocate the LSI in the first place. Signed-off-by: Frederic Barrat --- hw/pci-host/pnv_phb4.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/hw/pci-host/pnv_phb4.c b/hw/pci-host/pnv_phb4.c index 5c375a9f28..1659d55b4f 100644 --- a/hw/pci-host/pnv_phb4.c +++ b/hw/pci-host/pnv_phb4.c @@ -1234,10 +1234,13 @@ static void pnv_phb4_reset(DeviceState *dev) PCIDevice *root_dev = PCI_DEVICE(>root); /* - * Configure PCI device id at reset using a property. + * Configure the PCI device at reset: + * - set the Vendor and Device ID to for the root bridge + * - no LSI */ pci_config_set_vendor_id(root_dev->config, PCI_VENDOR_ID_IBM); pci_config_set_device_id(root_dev->config, phb->device_id); +pci_config_set_interrupt_pin(root_dev->config, 0); } static const char *pnv_phb4_root_bus_path(PCIHostState *host_bridge, FYI, I am seeing an issue with FreeBSD when booting from iso : https://download.freebsd.org/ftp/snapshots/powerpc/powerpc64/ISO-IMAGES/14.0/FreeBSD-14.0-CURRENT-powerpc-powerpc64-20211028-4827bf76bce-250301-disc1.iso.xz Thanks, C. SIGTERM received, booting... KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2021 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 14.0-CURRENT #0 main-n250301-4827bf76bce: Thu Oct 28 06:53:58 UTC 2021 r...@releng1.nyi.freebsd.org:/usr/obj/usr/src/powerpc.powerpc64/sys/GENERIC64 powerpc FreeBSD clang version 12.0.1 (g...@github.com:llvm/llvm-project.git llvmorg-12.0.1-0-gfed41342a82f) WARNING: WITNESS option enabled, expect reduced performance. VT: init without driver. ofw_initrd: initrd loaded at 0x2800-0x28c7928c cpu0: IBM POWER9 revision 2.0, 1000.00 MHz cpu0: Features dc007182 cpu0: Features2 bee0 real memory = 1014484992 (967 MB) avail memory = 117903360 (112 MB) random: registering fast source PowerISA DARN random number generator random: fast provider: "PowerISA DARN random number generator" arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled. random: entropy device external interface kbd0 at kbdmux0 ofwbus0: on nexus0 opal0: irq 1048560,1048561,1048562,1048563,1048564,1048565,1048566,1048567,1048568,1048569,1048570,1048571,1048572,1048573 on ofwbus0 opal0: registered as a time-of-day clock, resolution 0.002000s simplebus0: mem 0x60300-0x60300 on ofwbus0 pcib0: mem 0x600c3c000-0x600c3cfff,0x600c3-0x600c30fff on ofwbus0 pci0: numa-domain 0 on pcib0 qemu-system-ppc64: ../hw/pci/pci.c:1487: pci_irq_handler: Assertion `0 <= irq_num && irq_num < PCI_NUM_PINS' failed.
Re: [PATCH 1/3] ppc/pnv: Tune the POWER9 PCIe Host bridge model
On 11/16/21 18:01, Frederic Barrat wrote: The PHB v4 found on POWER9 doesn't request any LSI, so let's clear the Interrupt Pin register in the config space so that the model matches the hardware. If we don't, then we inherit from the default pcie root bridge, which requests a LSI. And because we don't map it correctly in the device tree, all PHBs allocate the same bogus hw interrupt. We end up with inconsistent interrupt controller (xive) data. The problem goes away if we don't allocate the LSI in the first place. Signed-off-by: Frederic Barrat Reviewed-by: Cédric Le Goater Thanks, C. --- hw/pci-host/pnv_phb4.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/hw/pci-host/pnv_phb4.c b/hw/pci-host/pnv_phb4.c index 5c375a9f28..1659d55b4f 100644 --- a/hw/pci-host/pnv_phb4.c +++ b/hw/pci-host/pnv_phb4.c @@ -1234,10 +1234,13 @@ static void pnv_phb4_reset(DeviceState *dev) PCIDevice *root_dev = PCI_DEVICE(>root); /* - * Configure PCI device id at reset using a property. + * Configure the PCI device at reset: + * - set the Vendor and Device ID to for the root bridge + * - no LSI */ pci_config_set_vendor_id(root_dev->config, PCI_VENDOR_ID_IBM); pci_config_set_device_id(root_dev->config, phb->device_id); +pci_config_set_interrupt_pin(root_dev->config, 0); } static const char *pnv_phb4_root_bus_path(PCIHostState *host_bridge,
