On Fri, 21 Aug 2020 15:36:40 -0500 Shawn Anastasio <sh...@anastas.io> wrote:
> Hello, > > While developing the userspace VFIO components of libkvmchan[1], I've > run into a dev_WARN in VFIO when hotplugging devices on the same pci > host bridge as other VFIO-managed devices: > > [ 111.220260][ T6281] pci 0001:00:01.0: Device added to live group 1! > [ 111.220390][ T6281] WARNING: CPU: 51 PID: 6281 at > drivers/vfio/vfio.c:709 vfio_iommu_group_notifier+0x250/0x360 [vfio] > > In spite of this warning, everything seems to be working fine. The > daemon that manages these PCI devices and owns the VFIO group fd > is able to connect to the new devices and use them just fine. > > My question is: what exactly is the reason this warning was added? > Is there a subtle issue that could be triggered by hotplugging to > live groups, and if so, what needs to be done to resolve this? > I'd be happy to try to resolve any issues and submit patches if necessary. > > For reference, this is all on a pseries/ppc64 guest on a POWER9 host, > with VFIO operating in the VFIO_SPAPR_TCE_IOMMU mode. In this config, > all PCI devices on a bus exist in the same IOMMU group (in this case #1). When a device is added to a live group there's a risk that it will be auto-probed by a host driver, if that occurs then isolation of the group has been violated and vfio code will BUG_ON to halt the system. The warning is effectively just a notification that we're in a risky situation where the wrong driver binding could escalate the issue. There is a ToDo in the code at that point to prevent driver probing, but ISTR at that time we may not have had a good way to do that. I'm not sure if we do now either. We have the driver_override field for the device that we could write into, but at this point we're looking at a generic device, we don't even know that it's a PCI device. We could determine that, but even then it's not clear that the kernel should set the policy to define that it should be bound to the vfio-pci driver, potentially versus other vfio drivers that could legitimately manage the device safely. If we write a random string to the driver_override field we could prevent automatic binding to any driver, but then we put a barrier to making use of the device, which seems like it has support issues as well. I'm not sure what the best approach is... that's why we currently generate a warning and hope it doesn't happen. On a truly bare metal platform, I don't think this should ever occur in practice without manually removing and re-scanning devices. We'd expect PCIe hotplug to occur on the slot level with isolation to the downstream port providing that slot. Without that isolation, or the increasingly unlikely chance of encountering this with conventional PCI hotplug, we'd probably hand wave the system as inappropriate for the task. Here I think you have a bare metal hypervisor exposing portions of devices to the "host" in unusual ways that can trigger this and are expected to be supported. Sorry, I don't have a good proposal to resolve how we should handle group composition changing while the group is in use... thus we currently just whine about it with a warning. Thanks, Alex _______________________________________________ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
