[Bug 1849563] Re: Unable to passthrough GPUs to guest, due to PCI64 aperture limitation
The Groovy Gorilla has reached end of life, so this bug will not be fixed for that release ** Changed in: edk2 (Ubuntu Groovy) Status: Confirmed => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest, due to PCI64 aperture limitation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1849563] Re: Unable to passthrough GPUs to guest, due to PCI64 aperture limitation
** Changed in: edk2 (Ubuntu Eoan) Status: Confirmed => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest, due to PCI64 aperture limitation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1849563] Re: Unable to passthrough GPUs to guest, due to PCI64 aperture limitation
** Tags added: sts -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest, due to PCI64 aperture limitation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1849563] Re: Unable to passthrough GPUs to guest, due to PCI64 aperture limitation
Upstream discussion: https://bugzilla.tianocore.org/show_bug.cgi?id=2796 ** Summary changed: - Unable to passthrough GPUs to guest + Unable to passthrough GPUs to guest, due to PCI64 aperture limitation ** No longer affects: edk2 (Ubuntu Disco) ** Also affects: edk2 (Ubuntu Groovy) Importance: Undecided Status: New ** Changed in: edk2 (Ubuntu Eoan) Status: New => Confirmed ** Changed in: edk2 (Ubuntu Focal) Status: New => Confirmed ** Changed in: edk2 (Ubuntu Groovy) Status: New => Confirmed ** Changed in: edk2 (Ubuntu Bionic) Importance: Undecided => Medium ** Changed in: edk2 (Ubuntu Eoan) Importance: Undecided => Medium ** Changed in: edk2 (Ubuntu Focal) Importance: Undecided => Medium ** Changed in: edk2 (Ubuntu Groovy) Importance: Undecided => Medium ** Changed in: edk2 (Ubuntu Bionic) Assignee: (unassigned) => Guilherme G. Piccoli (gpiccoli) ** Changed in: edk2 (Ubuntu Eoan) Assignee: (unassigned) => Guilherme G. Piccoli (gpiccoli) ** Changed in: edk2 (Ubuntu Groovy) Assignee: (unassigned) => Guilherme G. Piccoli (gpiccoli) ** Changed in: edk2 (Ubuntu Focal) Assignee: (unassigned) => Guilherme G. Piccoli (gpiccoli) ** Bug watch added: bugzilla.tianocore.org/ #2796 https://bugzilla.tianocore.org/show_bug.cgi?id=2796 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest, due to PCI64 aperture limitation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1849563] Re: Unable to passthrough GPUs to guest
Hi Dann, thanks a lot for your logs and pretty great bug report here.
The thread with edk2 folks is really informative!
I've been doing some research on this topic, and will share here in order to
document it.
So, first thing is about "pci=nocrs" "pci=realloc". When setting "pci=nocrs",
we are telling kernel to disregard ACPI resource information (CRS == Current
Resource Settings object). In this mode, all memory is available to PCI Host
Bridge allocations except the RAM and other detected reservations, so it
bypasses the limitations of the OVMF firmware. This mode was the default 10
years ago, but it was changed due to some incompatibilities, for example
systems with more than 1 PCI host bridge - the PCI subsystem maintainer decided
then to "trust" more in the FW resource mapping, and allowed a kernel fallback
through the option "nocrs". This is well-explained in [0] and [1].
The option "pci=realloc" is somewhat orthogonal to it; basically it
allows kernel to perform PCI endpoints (aka, devices) memory
(re-)assignments under their PCI host bridge memory space. So, an
analogy would be: the PCI host bridge resource is a pile of memory in
which devices will take some and consume for their BARs. With
"pci=realloc", we allow kernel to retry this memory mapping for PCI
devices some times, until it works (or eventually fail). It's natural to
use "pci=realloc" and this option is somewhat automatic, due to kernel
build-time configuration PCI_REALLOC_ENABLE_AUTO, which is default in
Ubuntu kernels. In summary, "pci=realloc" is the way the memory of PCI
host bridge is distributed to the PCI devices.
Now, regarding the firmware differences between OVMF and seabios. As per
the ed2k thread mentioned in the above comment, OVMF has a strict
limitation of the PCI64 aperture size. In seabios, things are a bit
different - the ACPI table passed to Linux containing the PCI64 aperture
information is DSST, this table is built dynamically based on SSDT
construction on boot time (build_ssdt() on seabios code). This is
ultimately based on PCI initialization routines that construct the BARs'
sizes and sum all of them, given the information in the PCI devices'
configuration space. The functions involved in this process are:
pci_setup() -> pci_bios_check_devices()/pci_bios_map_devices()
There's no limit on the aperture size, which is variable and can
accommodate as many devices the guest memory allows. In a way, this is
similar to the way Linux would perform the PCI resource allocations with
"pci=nocrs" parameter.
Now, OVMF is more complex in nature. The source tree of OVMF is composed
by multiple modules. The module MdeModulePkg is responsible for the PCI
enumeration for OVMF. There are 2 parts involved in that:
- the aperture is calculated on submodule PciHostBridgeDxe; it comes
from the early portions of the firmware code (submodule
OvmfPkg/PlatformPei), in the memory detection routine (and in that point
we can hijack into it using the experimental parameter X-PciMmio64Mb).
This is then passed to PciHostBridgeDxe which will create a bridge with
the memory resources' limits set.
- The PCI enumeration itself (and specially the device dropping in case the
aperture is exceeded) comes in the submodule PciBusDxe, through the following
functions:
PciBusDriverBindingStart() -> PciEnumerator() ->
PciHostBridgeResourceAllocator()
The function PciHostBridgeResourceAllocator() is the one that tries to allocate
effectively the memory through what's called Global Coherency Domain (GCD), the
edk2/UEFI generic memory/IO manager. It's done in the PCI Bridge "level" and if
it fails due to lack of resources then it'll go through the following functions
to free resources in the bridge:
PciHostBridgeAdjustAllocation() -> GetMaxResourceConsumerDevice()
In this point, the GPU is discarded on benefit of other devices in case
its BAR is too large based on the limitation of OVMF PCI64 aperture. For
reference, this is the edk2/OVMF commit that limits by default the PCI64
aperture size: 7e5b1b670c ("OvmfPkg: PlatformPei: determine the 64-bit
PCI host aperture for X64 DXE)
Cheers,
Guilherme
[0] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/341681/comments/97
[1] https://bugzilla.kernel.org/show_bug.cgi?id=14183
** Bug watch added: Linux Kernel Bug Tracker #14183
https://bugzilla.kernel.org/show_bug.cgi?id=14183
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1849563
Title:
Unable to passthrough GPUs to guest
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1849563] Re: Unable to passthrough GPUs to guest
Discussion thread on edk2 list: https://edk2.groups.io/g/discuss/topic/ovmf_resource_assignment/59340711?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,59340711 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1849563] Re: Unable to passthrough GPUs to guest
** Attachment added: "lspci.seabios" https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+attachment/5305737/+files/lspci.seabios -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1849563] Re: Unable to passthrough GPUs to guest
** Attachment added: "lspci.uefi" https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+attachment/5305738/+files/lspci.uefi -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1849563] Re: Unable to passthrough GPUs to guest
Attaching a boot log using an OVMF built w/ DEBUG enabled, which adds some more runes, such as: PciBus: HostBridge->NotifyPhase(AllocateResources) - Out of Resources PciBus: [01|00|00] was rejected due to resource confliction. ** Attachment added: "uefi boot log w/ OVMF DEBUG enabled" https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+attachment/5305740/+files/q35-uefidbg.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1849563] Re: Unable to passthrough GPUs to guest
** Attachment added: "lspci.uefi.pci=realloc,pci=nocrs" https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+attachment/5305739/+files/lspci.uefi.pci%3Drealloc%2Cpci%3Dnocrs -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1849563] Re: Unable to passthrough GPUs to guest
fyi, passing both pci=realloc pci=nocrs works as a workaround for me for q35 guests. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1849563] Re: Unable to passthrough GPUs to guest
As an experiment, I retried this test with focal host/guest (on the off- chance that e.g. we were missing something from QEMU or some topology logic in virtinst), but the results were the same. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1849563] Re: Unable to passthrough GPUs to guest
** Changed in: edk2 (Ubuntu Bionic) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1849563] Re: Unable to passthrough GPUs to guest
On Thu, Nov 7, 2019 at 7:50 AM Dimitri John Ledkov wrote: > > What is host architecture? amd64 - specifically, an Nvidia DGX2 system. > Does host support gpu passthrough? > Was it enabled in Bios settings, and to the kernel commandline? > > I.e. VT-d enabled in BIOS + intel_iommu=on passed on the kernel > commandline for an Intel based machine. Similarish things need to happen > on other platforms, ie. AMD. As these passthrough support are host- > vendor-hw specific. I think the important piece here - and the reason I filed the issue against edk2 - is the following from the Description: "Neither issue is reproducible when booting in non-UEFI mode." Since I can passthrough devices just fine in legacy BIOS mode, I assume all of my host BIOS/host kernel configuration is OK. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1849563] Re: Unable to passthrough GPUs to guest
What is host architecture? Does host support gpu passthrough? Was it enabled in Bios settings, and to the kernel commandline? I.e. VT-d enabled in BIOS + intel_iommu=on passed on the kernel commandline for an Intel based machine. Similarish things need to happen on other platforms, ie. AMD. As these passthrough support are host- vendor-hw specific. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1849563] Re: Unable to passthrough GPUs to guest
Urgh, sorry - ignore comment #5. The only reason I didn't see issues is that I hadn't yet installed nvidia-dkms in the guest :( Even with q35 I see the "NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:" and following messages. Further, in q35 mode, pci=nocrs no longer seems to be a functioning workaround. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1849563] Re: Unable to passthrough GPUs to guest
I retested w/ q35 and did not have an issue. Here's the command I used: virt-install --name q35 --machine q35 --memory 4096 --boot uefi --disk /home/ubuntu/q35.img --disk q35-seed.img --hostdev pci__34_00_0 --hostdev pci__36_00_0 --graphics none But, by default, virt-install placed the devices on different BDFs in the guest when selecting q35: virsh edit wouldn't let me place the device at slot >= 0, so I couldn't do an apples/apples comparison w/ the i440FX -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1849563] Re: Unable to passthrough GPUs to guest
cpaelzer recommends that we retest w/ the q35 machine type as a next step. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1849563] Re: Unable to passthrough GPUs to guest
** Attachment added: "guest XML (hits issue #1, no output)" https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+attachment/5299614/+files/foo.xml -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1849563] Re: Unable to passthrough GPUs to guest
** Attachment added: "dmesg when booting in non-UEFI mode (OK)" https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+attachment/5299616/+files/dmesg.seabios -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1849563] Re: Unable to passthrough GPUs to guest
** Attachment added: "dmesg of guest after hitting issue #2 (invalid I/O region)" https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+attachment/5299615/+files/dmesg -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1849563 Title: Unable to passthrough GPUs to guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
