On Wed, Sep 25, 2013 at 05:39:37AM -0400, Laine Stump wrote: > On 09/25/2013 04:48 AM, Marcel Apfelbaum wrote: > > On Wed, 2013-09-25 at 10:01 +0300, Michael S. Tsirkin wrote: > >> On Tue, Sep 24, 2013 at 06:01:02AM -0400, Laine Stump wrote: > >>> When I added support for the Q35-based machinetypes to libvirt, I > >>> specifically prohibited attaching any PCI devices (with the exception of > >>> graphics controllers) to the PCIe root complex, > >> That's wrong I think. Anything attached to RC is an integrated > >> endpoint, and these can be PCI devices. > > I couldn't find on PCIe spec any mention that "Root Complex Integrated > > EndPoint" > > must be PCIe. But, from spec 1.3.2.3: > > - A Root Complex Integrated Endpoint must not require I/O resources claimed > > through BAR(s). > > - A Root Complex Integrated Endpoint must not generate I/O Requests. > > - A Root Complex Integrated Endpoint is required to support MSI or MSI-X or > > both if an > > interrupt resource is requested. > > So much easier in the real world, where the rule is "if it fits in the > socket and the pins match up, then it's okay" :-) >
Well real world hardware has even more limitations, like different PCIe form-factors. Also I'm not aware of such hardware for PCI/PCIe specifically, but hardware with support for multiple buses does exist, e.g. I have a disk with both USB and eSATA interfaces. Also, it could be same chip with different interfaces on top. At some point intel made hardware that looked almost exactly like a PCI-X part except it had PCI-E interface. We can stick all this info into hardware type but it's a really ugly way to do this. See for example virtio-net-pci which users still can't wrap their heads around. They really want to say use virtio net device. > >> IMO, we really need to grow an interface to query this kind of thing. > > Basically libvirt needs to know: > > 1. for (libvirt) controllers: what kind of devices can be plugged in > > The controllers also need a flag indicating if they supporting having > devices hot-plugged into them. For example, as far as I understand, the > PCI root complex ("pcie-root" in libvirt) doesn't support hot-plugging > devices, Not exactly. It doesn't support native hotplug. > nor does i82801b11-bridge ("dmi-to-pci-bridge" in libvirt), but > pci-bridge, ioh3420 ("root-port" in libvirt), and xio3130-downstream > ("downstream-switch-port" in libvirt) *do* support hot-plugging devices > (as far as I know, none of these controllers can themselves be > hot-plugged into another controller, but that could change in the > future, e.g. I recall someone saying something about hot-plug of a > pci-bridge being a future possibility) > > > > 2. for devices (controller is also a device) > > - to which controllers can it be plugged in > > - does it support hot-plug? > > 3. implicit controllers of the machine types (q35 - "pcie-root", i440fx - > > "pci-root") > > All the above must be exported to libvirt > > > > Implementation options: > > 1. Add a compliance field on PCI/PCIe devices and controllers stating if it > > supports > > PCI/PCIe or both (and maybe hot-plug) > > - consider plug type + compliance to figure out whether a plug can go > > into a socket > > > > 2. Use Markus Armbruster idea of introducing a concept of "plug and > > sockets": > > - dividing the devices into adapters and plugs > > - adding sockets to bridges(buses?). > > In this way it would be clear which devices can connect to bridges > > However it is done, each controller will need to have two sets of flags > - one for what it can plug into, and one for what can be plugged into it. Not just into it - what can be plugged into each slot.