Re: [libvirt] [PATCH v3 19/18] [RFC] qemu: assure there are always at least 4 open pcie-root-ports for hotplug
On Fri, 2016-10-14 at 16:54 +0200, Ján Tomko wrote: > Also, would it be possible to make this feature of leaving > free hot-pluggable slots opt-in? > > E.g. a without an index > would be considered a port where we should not put devices > with auto-assigned addresses. > > (This might actually be more ugly than the proposed solution, > or the 'freeHotplugSlots' attribute, but I dislike implicit > device additions after having model='none' memballoon and usb > controller) If we can find a way to make it opt-in without it turning out incredibly ugly (eg. 'freeHotplugSlots' or similar paramers) or requiring too much knowledge from the users / management applications (eg. "you have x legacy PCI devices and y PCI Express devices, that means adding x + y + z + 3 pcie-root-ports before passing the XML to libvirt") I'm all for that. So far we haven't been able to come up with anything like that, though :) I think the current proposal is fairly okay because it allows for easy opt-out, and if we reduce the number of extra ports to just one as per your comments I don't think people will care at all. As far as I'm concerned, that's a very good balance; it's certainly better than the status quo of adding a completely useless dmi-to-pci-bridge + pci-bridge combination to every single q35 guest. -- Andrea Bolognani / Red Hat / Virtualization -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH v3 19/18] [RFC] qemu: assure there are always at least 4 open pcie-root-ports for hotplug
On Fri, Oct 14, 2016 at 03:13:16PM +0200, Andrea Bolognani wrote: On Thu, 2016-09-29 at 15:20 -0400, Laine Stump wrote: In the absense of anything configurable, we will need to pick a number though. I've done that here, and now we can argue about it (or not :-) Ján raised an interesting point: [in person, there is no mailing list link] whatever number we choose, no matter how wrong doing so is, some user or application will end up relying on us providing at least that many hotpluggable ports, so there's no going back. For that reason, I would recommend leaving just a single empty hotpluggable port for now, and raise the number in the future if we find out that one's definitely not enough. Also, would it be possible to make this feature of leaving free hot-pluggable slots opt-in? E.g. a without an index would be considered a port where we should not put devices with auto-assigned addresses. (This might actually be more ugly than the proposed solution, or the 'freeHotplugSlots' attribute, but I dislike implicit device additions after having model='none' memballoon and usb controller) Jan signature.asc Description: Digital signature -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH v3 19/18] [RFC] qemu: assure there are always at least 4 open pcie-root-ports for hotplug
On 10/14/2016 09:13 AM, Andrea Bolognani wrote: On Thu, 2016-09-29 at 15:20 -0400, Laine Stump wrote: In the absense of anything configurable, we will need to pick a number though. I've done that here, and now we can argue about it (or not :-) Ján raised an interesting point: whatever number we choose, no matter how wrong doing so is, some user or application will end up relying on us providing at least that many hotpluggable ports, so there's no going back. For that reason, I would recommend leaving just a single empty hotpluggable port for now, and raise the number in the future if we find out that one's definitely not enough. That makes sense. I've already updated the patch per our conversation in the other thread, but will update it further to reflect this idea as well. -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH v3 19/18] [RFC] qemu: assure there are always at least 4 open pcie-root-ports for hotplug
On Thu, 2016-09-29 at 15:20 -0400, Laine Stump wrote: > In the absense of anything configurable, we will need to pick a number > though. I've done that here, and now we can argue about it (or not :-) Ján raised an interesting point: whatever number we choose, no matter how wrong doing so is, some user or application will end up relying on us providing at least that many hotpluggable ports, so there's no going back. For that reason, I would recommend leaving just a single empty hotpluggable port for now, and raise the number in the future if we find out that one's definitely not enough. -- Andrea Bolognani / Red Hat / Virtualization -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH v3 19/18] [RFC] qemu: assure there are always at least 4 open pcie-root-ports for hotplug
On Thu, 2016-09-29 at 15:20 -0400, Laine Stump wrote: > For machinetypes with a pci-root bus (all legacy PCI), libvirt will > make a "fake" reservation for one extra slot prior to assigning > addresses to unaddressed PCI endpoint devices in the domain. This will > trigger auto-adding of a pci-bridge for the final device to be > assigned an address *if that device would have otherwise instead been > the last device on the last available pci-bridge*; thus it assures > that there will always be at least one slot left open in the domains > bus topology for expansion (which is important both for hotplug (since > a new pci-bridge can't be added while the guest is running) as well as > for coldplug (since adding a new device might otherwise in some cases > require re-addressing existing devices, which we want to avoid)). > > It's important to note that for this (legacy PCI) case, we must check > for the special case of all slots on all buses being occupied *prior > to assigning any addresses*, and avoid attempting to reserve the extra > address in that case, because there is no free address in the existing > topology, so no place to auto-add a pci-bridge for expansion. Since > that condition can only be reached by manual intervention, this is > acceptable. > > For machinetypes with pcie-root, libvirt's methodology for > automatically expanding the bus topology is different - > pcie-root-ports are plugged into slots (soon to be functions) of > pcie-root as needed, and the new endpoint devices are assigned to the > single slot in each pcie-root-port. This is done so that the devices > are, by default, hotpluggable (the slots of pcie-root don't support > hotplug, but the single slot of the pcie-root-port does). Since > pcie-root-ports can only be plugged into pcie-root, and we don't > auto-assign endpoint devices to those slots, this means topology > expansion doesn't compete with endpoint devices for slots, so we don't > need to worry about checking for all "useful" slots being free prior > to assigning addresses to new endpoint devices - as a matter of fact, > if we attempt to reserve the fake slots before the useful slots, it > can lead to errors. > > Instead this patch just reserves slots for "fake" devices after doing > the assignment for actual devices - if there were already enough > unoccupied pcie-root-ports, then none will be added; if not, then > enough will be added to support the desired (hardcoded) potential > number of hotplugged devices. > > Since there hasn't been any other concrete suggestion for the number > of "available hotpluggable slots" libvirt should assure, this patch > uses "4" as the answer (thanks Drew!) Alternatives could be: > > 0 - hotplug would only work if the user had thought to add an extra > pcie-root to the config *as an extra step after adding and saving each > new device*. (this would preclude creating a new domain that had a > pcie-root-port available for hotplug - the extra modify/save step > would be needed even during initial domain creation, and would need to > be repeated every time a device was added to the domain) > > 1 - this is the *minimum* number of hotpluggable slots libvirt > guarantees for legacy PCI domains (with pci-root). Of course on > average these domains are more likely to have *15* slots available > (half of the slots on a pci-bridge). > > "anything else" - really any choice made for this is going to be > considered wrong by *somebody*. I hope we can all agree that "0" is a > wrong choice, just because it will require so much ongoing > babysitting. > > A couple of us have brought up the idea of having > "availableHotplugSlots" be configurable in the domain. There has also > been sentiment against that. Perhaps it could be configurable in > qemu.conf? (I don't really like that either, though - I think > everything about the domain should be self-contained in the domain > XML, and if something is tunable, it should be tunable separately for > each domain). > > In the absense of anything configurable, we will need to pick a number > though. I've done that here, and now we can argue about it (or not :-) As we have already agreed on a slightly different approach,x eg. doing pretty much exactly this, but only if the configuration provided by the user contains no PCI controller (except possibly for pcie-root), I think it's fair to say: NACK -- Andrea Bolognani / Red Hat / Virtualization -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
