tl;dr - you are correct and I don't know where my head was. :-)
On 10/6/26 12:27 PM, David Vrabel wrote:
On 05/10/2026 20:19, Laine Stump wrote:
When a new domain is created and no busNr is given, the default value
decided on by libvirt at that time is written into the XML, so
subsequent starts of that particular domain using that definition will
use the same busNr - only newly created domains will get a different
default busNr than they used to. So in that case, this change in
default setting has 0 effect.
There are management stacks that re-create the XML for every power on,
Yeah, I hadn't forgotten that, and was even considering those systems,
but...
so every power cycle these VMs would see different SBDFs across the
libvirt upgrade
.. my mind was somehow blocking the fact that the attribute we call
busNr directly determines the *actual* bus number of the controller and
subordinate devices[*]. I mean, it completely makes sense that's what it
would be used for (it's right there in the name!), but for some reason
as I was looking at it, my brain was thinking of it more like the
"chassis" attribute, which (as far as I've been told by lower level
guys) is visible to the guest in a register somewhere, but not used to
enumerate (and sometimes name) devices the way that SBDF is (it was you
saying "SBDF" that finally jogged my memory).
[*] as opposed to the "bus" attribute of the <address type='pci'/>
element in libvirt, which does get passed directly to QEMU, but is
really just an index in a table somewhere in QEMU (I'm assuming), and
not necessarily related to the bus number that is visible to the guest
OS - we've spent a lot of time trying to explain that over the years to
users who expect that when they set "bus='5'" in the XML, the device
will always show up in the guest on bus 5, even if they have no bus 4
(for example))
and some VMs (e.g., Windows) get upset if SBDFs change
(of which I'm sure you're aware).
Painfully! I don't know if it's still the case, but way back when
libvirt began explicitly assigning pci addresses to devices and saving
them in the domain XML, I think one of the reasons was that Windows
could invalidate the guest OS's license key activation if certain/too
many devices' pci addresses changed, and also would pop up a window
about a "new network connection" if the pci address of a network device
changed (and if you think *that's* annoying, it also pops up that dialog
if the MAC address of the next hop default route changes :-/).
And you can't even point the finger only at Windows - in modern Linux
with "predictable" network device naming, the name of a PCI network
device will change when its PCI address changes, thus invalidating any
config based on the device name (which I should have noticed when
testing to verify this patch fixed the originally reported problem, but
was so happy it booted Fedora from the pcie-expander-bus connected disk
image that I only looked to see if the pcie-expander-bus connected NIC
had gotten a an IP from DHCP and was able to communicate).
If you think management stacks that do that should also take care of the
busNr allocations then I personally find that acceptable (as the
management stack I care about already does this),
It's a bit of a digression, but really I think the management layer
shouldn't need to worry about details like PCI addresses at all - they
should be able to just say "make this device hotpluggable, put this
device on NUMA node 1 and not hotpluggable" etc, and [someone else]
would figure out the details of PCI topology for them. That's a bit
outside libvirt's scope though (even though libvirt does automatically
assign PCI addresses for all PCI devices, that was originally done as a
defense mechanism to assure that PCI addresses remained stable as
devices were added and removed, and just encoded what addresses had been
previously auto-assigned by QEMU on startup, and grew organically from
there).
but I'm not sure I'd
want to make assumptions about the behaviour of every one.
So whilst I agree that the current bus number allocation algorithm isn't
very useful, I do think a new algorithm needs to be explicitly requested
by the user or management stack.
Yeah, after you've jogged my memory about what a change in busNr really
leads to, I agree, and am dropping this patch.
Thanks for bringing me back to reality :-)
