On Mon, May 12, 2025 at 07:33:37PM +0000, Matt Ochs wrote: > > On May 12, 2025, at 5:19 AM, Daniel P. Berrangé <berra...@redhat.com> wrote: > > On Fri, May 09, 2025 at 07:29:04PM +0000, Matt Ochs wrote: > >> > >> Would it make sense to just use the existing pcihole64 since [I think] > >> it more or less represents the same concept (setting 64bit MMIO window)? > > > > I'm not sure. I've been struggling to reproduce an effect wit hseting > > the existing -global q35-pcihost.pci-hole64-size=1048576K settings > > on x86, and also wondering how it interacts with the previously > > mentioned -fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=262144 > > > > Possibly the former only works with SeaBIOS, and the latter only > > works with EDK2, but I've not figured out how to prove this. > > The qemu docs mention opt/ovmf is specifically for OVMF firmware: > https://github.com/qemu/qemu/blob/7be29f2f1a3f5b037d27eedbd5df9f441e8c8c16/docs/specs/fw_cfg.rst#L279 > > The pcihole64 setting can be used with OVMF (see below) and with SEABIOS: > https://github.com/libvirt/libvirt/blob/master/docs/formatdomain.rst (see > pcihole64) > > The X-PciMmio64Mb parameter isn't directly supported in libvirt IIUC. The > libvirt > XML would need to directly pass qemu command line arguments to use it.
I'm wondering what the semantic difference is between setting the pcihole64 property and the X-PciMmio64Mb fwcfg, in the context of OVMF. The fact that both exist, suggests that there is a meaningful difference, which in turn would mean libvirt might need separate XML attributes for each, which in turn influences how we might choose to design the aarch64 solution. > > > > > I'm curious if there's a good way to identify the guest memory > > map impact, as I'm not finding a clear marker in 'dmesg' that > > correlates ? > > We were able to test this by using OVMF without the dynamic mmio > window size patch (i.e. a version older than edk2-stable202211) and > guest kernel parameters that are not set to allow re-calculating the > MMIO window size by deferring guest resource allocations to the guest > kernel (i.e. pci=realloc and pci=nocrs aren't set). With this we could > reproduce a 4 GPU VM launch with guest BARs not mapped properly > due to running out of space/resources. The BAR mapping failures will > be clear in dmesg, with no BAR region mappings in /proc/iomem or > output of lspci for the GPUs. > > From there we added the pcihole64 attribute to the VM's libvirt definition, > setting a 2 TB hole size, and the VM booted with guest GPU BARs mapped > properly in dmesg + GPU BAR mappings visible in /proc/iomem and lspci output. > > Lastly, observed the same behavior by removing the pcihole64 attribute and > setting the X-PciMmio64Mb configuration to 2TB. > > > > >> Or perhaps that would be too messy or x86-centric and it’s better to go > >> with what you proposed (pcimmio64)? > > > > If the 'pcihole64' setting really is setting the MMIO64 window, then it > > would be preferrable to re-use the existing setting field. > > Per the tests above, pcihole64 is setting the MMIO64 window. The only concern > I have with using it is that to date, it has been an x86-centric attribute > and tied > closely with the qemu -global parameter. I don’t think this is a > show-stopper, but > will require some code changes to allow it to work with the virt machine and > connect it up to a different qemu parameter for that machine. > > > -matt > > With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
