On Sun, 7 Apr 2019 16:02:31 -0700 Bart Van Assche <[email protected]> wrote:
> On 4/7/19 2:06 PM, Alex Williamson wrote: > > On Sun, 7 Apr 2019 12:10:38 -0700 > > Bart Van Assche <[email protected]> wrote: > >> If I tell qemu to use PCI pass-through for a PCI adapter and next load the > >> lpfc driver for an lpfc adapter that has not been passed through to any VM > >> a kernel bug is hit. Do you perhaps know whether it should be possible to > >> a load kernel driver in this scenario? If so, do you know what should > >> change > >> to avoid that this kernel bug is hit? Should the iommu code be modified or > >> should the lpfc code be modified? I'm asking you because I think that you > >> introduced the BUG() statement that was hit. See also commit ab8dfe251571 > >> ("iommu/vt-d: Introduce helper functions to improve code readability"; > >> v3.17). > > > > Do both of these lpfc devices belong to the same IOMMU group? > > (/sys/kernel/iommu_groups/) Thanks, > > Hi Alex, > > Apparently the two Emulex (lpfc) and the two QLogic ports are in the same > IOMMU group: > > # lspci | grep -E 'QLogic|Emulex' > 01:00.0 Fibre Channel: Emulex Corporation Lancer Gen6: LPe32000 Fibre Channel > Host Adapter (rev 01) > 01:00.1 Fibre Channel: Emulex Corporation Lancer Gen6: LPe32000 Fibre Channel > Host Adapter (rev 01) > 02:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI > Express HBA (rev 02) > 02:00.1 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI > Express HBA (rev 02) > > # ls -d /sys/kernel/iommu_groups/*/devices/0000:0[12]:00.* > /sys/kernel/iommu_groups/1/devices/0000:01:00.0 > /sys/kernel/iommu_groups/1/devices/0000:01:00.1 > /sys/kernel/iommu_groups/1/devices/0000:02:00.0 > /sys/kernel/iommu_groups/1/devices/0000:02:00.1 It's not possible to do what you want with this configuration. An IOMMU group represents the smallest set of devices that are isolated from other sets of devices and is also therefore the minimum granularity we can assign devices to userspace (ex. QEMU). The kernel reacts to breaking the isolation of the group with a BUG_ON. If you managed not to hit the BUG_ON here, you'd hit the BUG_ON in vfio code when the loss of isolation is detected there. IOMMU groups are formed at the highest point in the topology which guarantees isolation. This can be indicated either via native PCIe ACS support or ACS-equivalent quirks in the code. If the root port provides neither of these, then all devices downstream are grouped together as well as all peer root ports in the same PCI slot and all devices downstream of those. If a multifunction endpoint does not provide ACS or equivalent quirks, the functions will be grouped together. Not all endpoint devices or systems are designed for minimum possible granularity. You can learn more here[1]. Thanks, Alex [1] http://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
