> On May 12, 2025, at 5:19 AM, Daniel P. Berrangé <berra...@redhat.com> wrote: > On Fri, May 09, 2025 at 07:29:04PM +0000, Matt Ochs wrote: >> >> Would it make sense to just use the existing pcihole64 since [I think] >> it more or less represents the same concept (setting 64bit MMIO window)? > > I'm not sure. I've been struggling to reproduce an effect wit hseting > the existing -global q35-pcihost.pci-hole64-size=1048576K settings > on x86, and also wondering how it interacts with the previously > mentioned -fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=262144 > > Possibly the former only works with SeaBIOS, and the latter only > works with EDK2, but I've not figured out how to prove this.
The qemu docs mention opt/ovmf is specifically for OVMF firmware: https://github.com/qemu/qemu/blob/7be29f2f1a3f5b037d27eedbd5df9f441e8c8c16/docs/specs/fw_cfg.rst#L279 The pcihole64 setting can be used with OVMF (see below) and with SEABIOS: https://github.com/libvirt/libvirt/blob/master/docs/formatdomain.rst (see pcihole64) The X-PciMmio64Mb parameter isn't directly supported in libvirt IIUC. The libvirt XML would need to directly pass qemu command line arguments to use it. > > I'm curious if there's a good way to identify the guest memory > map impact, as I'm not finding a clear marker in 'dmesg' that > correlates ? We were able to test this by using OVMF without the dynamic mmio window size patch (i.e. a version older than edk2-stable202211) and guest kernel parameters that are not set to allow re-calculating the MMIO window size by deferring guest resource allocations to the guest kernel (i.e. pci=realloc and pci=nocrs aren't set). With this we could reproduce a 4 GPU VM launch with guest BARs not mapped properly due to running out of space/resources. The BAR mapping failures will be clear in dmesg, with no BAR region mappings in /proc/iomem or output of lspci for the GPUs. From there we added the pcihole64 attribute to the VM's libvirt definition, setting a 2 TB hole size, and the VM booted with guest GPU BARs mapped properly in dmesg + GPU BAR mappings visible in /proc/iomem and lspci output. Lastly, observed the same behavior by removing the pcihole64 attribute and setting the X-PciMmio64Mb configuration to 2TB. > >> Or perhaps that would be too messy or x86-centric and it’s better to go >> with what you proposed (pcimmio64)? > > If the 'pcihole64' setting really is setting the MMIO64 window, then it > would be preferrable to re-use the existing setting field. Per the tests above, pcihole64 is setting the MMIO64 window. The only concern I have with using it is that to date, it has been an x86-centric attribute and tied closely with the qemu -global parameter. I don’t think this is a show-stopper, but will require some code changes to allow it to work with the virt machine and connect it up to a different qemu parameter for that machine. -matt
