On 05/09/2019 14:22, Shawn Anastasio wrote:
On pseries QEMU guests, IOMMU setup for hotplugged PCI devices is currently
broken for all but the first device on a given bus. The culprit is an ordering
issue in the pseries hotplug path (via pci_rescan_bus()) which results in IOMMU
group assigment occuring before device registration in sysfs. This triggers
the following check in arch/powerpc/kernel/iommu.c:
/*
* The sysfs entries should be populated before
* binding IOMMU group. If sysfs entries isn't
* ready, we simply bail.
*/
if (!device_is_registered(dev))
return -ENOENT;
This fails for hotplugged devices since the pcibios_add_device() call in the
pseries hotplug path (in pci_device_add()) occurs before device_add().
Since the IOMMU groups are set up in pcibios_add_device(), this means that a
sysfs entry will not yet be present and it will fail.
I just tried hotplugging 3 virtio-net devices into a guest system with
v5.2 kernel and it seems working (i.e. BARs mapped, a driver is bound):
root@le-dbg:~# lspci -v | egrep -i '(virtio|Memory)'
00:00.0 Ethernet controller: Red Hat, Inc Virtio network device
Memory at 20008004 (32-bit, non-prefetchable) [size=4K]
Memory at 2100 (64-bit, prefetchable) [size=16K]
Kernel driver in use: virtio-pci
00:01.0 Ethernet controller: Red Hat, Inc Virtio network device
Memory at 200080041000 (32-bit, non-prefetchable) [size=4K]
Memory at 21004000 (64-bit, prefetchable) [size=16K]
Kernel driver in use: virtio-pci
00:02.0 Ethernet controller: Red Hat, Inc Virtio network device
Memory at 200080042000 (32-bit, non-prefetchable) [size=4K]
Memory at 21008000 (64-bit, prefetchable) [size=16K]
Kernel driver in use: virtio-pci
Can you explain in detail what you are doing exactly and what is failing
and what qemu/guest kernel/guest distro is used? Thanks,
There is a special case that allows the first hotplugged device on a bus to
succeed, though. The powerpc pcibios_add_device() implementation will skip
initializing the device if bus setup is not yet complete.
Later, the pci core will call pcibios_fixup_bus() which will perform setup
for the first (and only) device on the bus and since it has already been
registered in sysfs, the IOMMU setup will succeed.
My current solution is to introduce another pcibios function, pcibios_fixup_dev,
which is called after device_add() in pci_device_add(). Then in powerpc code,
pcibios_setup_device() was moved from pcibios_add_device() to this new function
which will occur after sysfs registration so IOMMU assignment will succeed.
I added a new pcibios function rather than moving the pcibios_add_device() call
to after the device_add() call in pci_add_device() because there are other
architectures that use it and it wasn't immediately clear to me whether moving
it would break them.
If anybody has more insight or a better way to fix this, please let me know.
Shawn Anastasio (2):
PCI: Introduce pcibios_fixup_dev()
powerpc/pci: Fix IOMMU setup for hotplugged devices on pseries
arch/powerpc/kernel/pci-common.c | 13 ++---
drivers/pci/probe.c | 14 ++
include/linux/pci.h | 1 +
3 files changed, 21 insertions(+), 7 deletions(-)
--
Alexey
