Hi Julia
Not sure if you've noticed my previous email...
After switching from PCIe native hotplug to ACPI PCI hotplug(with
patches
https://lists.gnu.org/archive/html/qemu-devel/2021-07/msg03306.html),
I've run into the secondary virtio_blk hotplugging issue in Windows q35
guest.
Now, it seems Linux q35 guest also runs into issues when
virtio_blk/virtio_net device is hotplugged, both devices fail to get
proper BAR memory assigned. After hot-plugging virtio_blk device, dmesg
shows following error,
[ 111.131377] pci 0000:03:00.0: [1af4:1042] type 00 class 0x010000
[ 111.131815] pci 0000:03:00.0: reg 0x14: [mem 0x00000000-0x00000fff]
[ 111.132206] pci 0000:03:00.0: reg 0x20: [mem 0x00000000-0x00003fff
64bit pref]
[ 111.135050] pci 0000:03:00.0: BAR 4: no space for [mem size
0x00004000 64bit pref]
[ 111.135053] pci 0000:03:00.0: BAR 4: failed to assign [mem size
0x00004000 64bit pref]
[ 111.135055] pci 0000:03:00.0: BAR 1: no space for [mem size 0x00001000]
[ 111.135056] pci 0000:03:00.0: BAR 1: failed to assign [mem size
0x00001000]
[ 111.136332] virtio-pci 0000:03:00.0: virtio_pci: leaving for legacy
driver
After hot-plugging virtio_nic device, dmesg shows following error.
[ 144.932161] pci 0000:04:00.0: [1af4:1041] type 00 class 0x020000
[ 144.932613] pci 0000:04:00.0: reg 0x14: [mem 0x00000000-0x00000fff]
[ 144.932999] pci 0000:04:00.0: reg 0x20: [mem 0x00000000-0x00003fff
64bit pref]
[ 144.933093] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref]
[ 144.935734] pci 0000:04:00.0: BAR 6: no space for [mem size
0x00040000 pref]
[ 144.935737] pci 0000:04:00.0: BAR 6: failed to assign [mem size
0x00040000 pref]
[ 144.935739] pci 0000:04:00.0: BAR 4: no space for [mem size
0x00004000 64bit pref]
[ 144.935741] pci 0000:04:00.0: BAR 4: failed to assign [mem size
0x00004000 64bit pref]
[ 144.935743] pci 0000:04:00.0: BAR 1: no space for [mem size 0x00001000]
[ 144.935744] pci 0000:04:00.0: BAR 1: failed to assign [mem size
0x00001000]
[ 144.937163] virtio-pci 0000:04:00.0: virtio_pci: leaving for legacy
driver
This error in Linux guest looks similar to the one I posted in Windows
guest earlier, there are memory conflicts among these hotplugged devices.
I am using Linux host version 5.15.0, the qemu version 6.1.0(patched
with
https://patchwork.kernel.org/project/qemu-devel/patch/20210916132838.3469580-3-...@anisinha.ca/),
OVMF version stable202108.
If I switch from the ACPI PCI hotplug to PCIe native hotplug with
following option, these errors are gone.
-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off
However, with PCIe native hotplug, we have been seeing other
hotplug/unplug failures, for example, deleted virtio disks still show in
q35 Windows guest.
I am wondering if anyone has seen this kind of errors with ACPI PCI
hotplug in q35 guests with latest QEMU version?
Thanks
Annie
On 11/1/2021 10:06 AM, Annie.li wrote:
Hello,
I've found an issue when hot-plugging the secondary virtio_blk device
into q35 Windows guest(2019) with upstream qemu 6.1.0(+1 patch). The
first disk can be hot-plugged successfully.
The qemu options for PCIe root port is,
-device
pcie-root-port,port=2,chassis=2,id=pciroot2,bus=pcie.0,addr=0x2,multifunction=on
\
-device
pcie-root-port,port=3,chassis=3,id=pciroot3,bus=pcie.0,addr=0x3,multifunction=on
\
-device
pcie-root-port,port=4,chassis=4,id=pciroot4,bus=pcie.0,addr=0x4,multifunction=on
\
-device
pcie-root-port,port=5,chassis=5,id=pciroot5,bus=pcie.0,addr=0x5,multifunction=on
\
-device
pcie-root-port,port=6,chassis=6,id=pciroot6,bus=pcie.0,addr=0x6,multifunction=on
\
The command to hotplug 1st virtio_blk disk is following, the PCI slot
of the 1st virtio_blk is Pci slot 0(PCI bus 1, device 0, function 0).
drive_add auto
file=block_10.qcow2,format=qcow2,if=none,id=drive10,cache=none
device_add virtio-blk-pci,drive=drive10,id=block-disk10,bus=pciroot2
Following is the related "info mtree" after the 1st virtio_blk device
is hot plugged
memory-region: pci_bridge_pci
0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci
00000000febff000-00000000febfffff (prio 1, i/o): virtio-blk-pci-msix
00000000febff000-00000000febff01f (prio 0, i/o): msix-table
00000000febff800-00000000febff807 (prio 0, i/o): msix-pba
0000000fffffc000-0000000fffffffff (prio 1, i/o): virtio-pci
0000000fffffc000-0000000fffffcfff (prio 0, i/o): virtio-pci-common
0000000fffffd000-0000000fffffdfff (prio 0, i/o): virtio-pci-isr
0000000fffffe000-0000000fffffefff (prio 0, i/o): virtio-pci-device
0000000ffffff000-0000000fffffffff (prio 0, i/o): virtio-pci-notify
memory-region: pci_bridge_pci
0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci
memory-region: pci_bridge_pci
0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci
memory-region: pci_bridge_pci
0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci
memory-region: pci_bridge_pci
0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci
Right after the secondary virtio_blk device is hot-plugged, a yellow
mark shows on the first virtio_blk device in the Windows guest. The
PCI slot info of the 2nd virtio_blk is Pci slot 0(PCI bus 2, device 0,
function 0). The debug log of Windows virtio_blk driver shows a
"ScsiStopAdapter" adapter control operation is triggered first, and
then "StorSurpriseRemoval". From the following "info mtree", it seems
the 2nd virtio_blk device is occupying the same memory resource as the
above 1st virtio_blk device. Maybe this causes the failure of the 1st
virtio_blk device and then the system assume it is surprisingly removed?
The command to hotplug 2nd virtio_blk disk,
drive_add auto
file=block_11.qcow2,format=qcow2,if=none,id=drive11,cache=none
device_add virtio-blk-pci,drive=drive11,id=block-disk11,bus=pciroot3
Following is the related "info mtree" after the 2nd virtio_blk device
is hot-plugged,
memory-region: pci_bridge_pci
0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci
memory-region: pci_bridge_pci
0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci
00000000febff000-00000000febfffff (prio 1, i/o): virtio-blk-pci-msix
00000000febff000-00000000febff01f (prio 0, i/o): msix-table
00000000febff800-00000000febff807 (prio 0, i/o): msix-pba
0000000fffffc000-0000000fffffffff (prio 1, i/o): virtio-pci
0000000fffffc000-0000000fffffcfff (prio 0, i/o): virtio-pci-common
0000000fffffd000-0000000fffffdfff (prio 0, i/o): virtio-pci-isr
0000000fffffe000-0000000fffffefff (prio 0, i/o): virtio-pci-device
0000000ffffff000-0000000fffffffff (prio 0, i/o): virtio-pci-notify
memory-region: pci_bridge_pci
0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci
memory-region: pci_bridge_pci
0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci
memory-region: pci_bridge_pci
0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci
Note: I've patched the upstream 6.1.0 qemu with following patch,
https://patchwork.kernel.org/project/qemu-devel/patch/20210916132838.3469580-3-...@anisinha.ca/
the acpi-pci-hotplug memory is following as expected,
0000000000000cc0-0000000000000cd7 (prio 0, i/o): acpi-pci-hotplug
0000000000000cd8-0000000000000ce3 (prio 0, i/o): acpi-mem-hotplug
Thanks
Annie
