Re: [vfio-users] Question about integrated GPU passthrough and initialization
This is helpful, thanks for the info! On Wed, Jun 12, 2019 at 9:41 AM Alex Williamson wrote: > > On Wed, 12 Jun 2019 08:49:36 -0700 > Micah Morton wrote: > > > Hi Alex, > > > > Thanks for the help on this earlier, I was able to get IGD passthrough > > working on my device (In case you're interested, crbug.com/970820 has > > further details on the changes we needed to make to the kernel/i915 > > driver to get things working). > > > > I have a couple follow up questions that hopefully you can get to when > > you get a chance: > > > > 1) I see in https://patchwork.kernel.org/patch/9115161/ that you wrote > > a patch for SeaBIOS to reserve stolen memory for the IGD device. It > > seems that it is hard-coded to reserve 8MB of stolen memory. I wanted > > to reserve 64MB, but wasn't able to locate the "etc/igd-bdsm-size" > > file that should be able to configure this. Where does this file live? > > Its easy enough for me to hard-code SeaBIOS to 64MB, but I was curious > > if there's a cleaner way to set this. > > There's a QEMU vfio-pci device option x-igd-gms= which (according to > the commit log[1], because I've forgotten how this works 3yrs later) > specifies the number of 32MB chunks of additional stolen memory > allocated. So theoretically you could just add x-igd-gms=2 for 64MB. > > > > 2) Even when SeaBIOS reserves a region for stolen memory, the VM > > kernel has a hard time realizing the region is there and available. So > > far I just hard-coded the kernel/i915 driver since I know the > > address/range of the memory region that SeaBIOS will reserve for > > stolen memory in my case. It requires some hard-coding both in the > > kernel > > (https://elixir.bootlin.com/linux/v4.14.114/source/arch/x86/kernel/early-quirks.c#L436) > > and in the driver > > (https://elixir.bootlin.com/linux/v4.14.114/source/drivers/gpu/drm/i915/i915_gem_stolen.c#L404). > > Are you aware of any discussions around this problem? I wanted to see > > if it has already been discussed before looking at proposing some kind > > of patch to the kernel/driver. > > The fact that there are genX versions of those sorts of sizing and init > routines is exactly part of the problem with assigning IGD. There's > no standard and the hardware folks change things as they please. Maybe > the QEMU code doesn't do quite the right thing for your hardware > generation. IIRC I developed the QEMU quirks on Broadwell and older > hardware and I don't have the time or hardware access to tweak it for > every new chip. Intel also can't seem to maintain a consistent story > about whether they're reducing or increasing their dependencies on > external components like chipset registers, stolen memory, and firmware > tables. I expect the best hope for reliable IGD in a VM with physical > display output might come through GVT-g (ie. vGPU), but the physical > display part of that isn't supported yet and I don't see a strong > commitment to enable GVT-g across the product line. Thanks, > > Alex > > [1]https://git.qemu.org/?p=qemu.git;a=commit;h=c4c45e943e519f5ac220f7af1afb2a0025d03c54 ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Question about integrated GPU passthrough and initialization
On Wed, 12 Jun 2019 08:49:36 -0700 Micah Morton wrote: > Hi Alex, > > Thanks for the help on this earlier, I was able to get IGD passthrough > working on my device (In case you're interested, crbug.com/970820 has > further details on the changes we needed to make to the kernel/i915 > driver to get things working). > > I have a couple follow up questions that hopefully you can get to when > you get a chance: > > 1) I see in https://patchwork.kernel.org/patch/9115161/ that you wrote > a patch for SeaBIOS to reserve stolen memory for the IGD device. It > seems that it is hard-coded to reserve 8MB of stolen memory. I wanted > to reserve 64MB, but wasn't able to locate the "etc/igd-bdsm-size" > file that should be able to configure this. Where does this file live? > Its easy enough for me to hard-code SeaBIOS to 64MB, but I was curious > if there's a cleaner way to set this. There's a QEMU vfio-pci device option x-igd-gms= which (according to the commit log[1], because I've forgotten how this works 3yrs later) specifies the number of 32MB chunks of additional stolen memory allocated. So theoretically you could just add x-igd-gms=2 for 64MB. > 2) Even when SeaBIOS reserves a region for stolen memory, the VM > kernel has a hard time realizing the region is there and available. So > far I just hard-coded the kernel/i915 driver since I know the > address/range of the memory region that SeaBIOS will reserve for > stolen memory in my case. It requires some hard-coding both in the > kernel > (https://elixir.bootlin.com/linux/v4.14.114/source/arch/x86/kernel/early-quirks.c#L436) > and in the driver > (https://elixir.bootlin.com/linux/v4.14.114/source/drivers/gpu/drm/i915/i915_gem_stolen.c#L404). > Are you aware of any discussions around this problem? I wanted to see > if it has already been discussed before looking at proposing some kind > of patch to the kernel/driver. The fact that there are genX versions of those sorts of sizing and init routines is exactly part of the problem with assigning IGD. There's no standard and the hardware folks change things as they please. Maybe the QEMU code doesn't do quite the right thing for your hardware generation. IIRC I developed the QEMU quirks on Broadwell and older hardware and I don't have the time or hardware access to tweak it for every new chip. Intel also can't seem to maintain a consistent story about whether they're reducing or increasing their dependencies on external components like chipset registers, stolen memory, and firmware tables. I expect the best hope for reliable IGD in a VM with physical display output might come through GVT-g (ie. vGPU), but the physical display part of that isn't supported yet and I don't see a strong commitment to enable GVT-g across the product line. Thanks, Alex [1]https://git.qemu.org/?p=qemu.git;a=commit;h=c4c45e943e519f5ac220f7af1afb2a0025d03c54 ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Question about integrated GPU passthrough and initialization
Hi Alex, Thanks for the help on this earlier, I was able to get IGD passthrough working on my device (In case you're interested, crbug.com/970820 has further details on the changes we needed to make to the kernel/i915 driver to get things working). I have a couple follow up questions that hopefully you can get to when you get a chance: 1) I see in https://patchwork.kernel.org/patch/9115161/ that you wrote a patch for SeaBIOS to reserve stolen memory for the IGD device. It seems that it is hard-coded to reserve 8MB of stolen memory. I wanted to reserve 64MB, but wasn't able to locate the "etc/igd-bdsm-size" file that should be able to configure this. Where does this file live? Its easy enough for me to hard-code SeaBIOS to 64MB, but I was curious if there's a cleaner way to set this. 2) Even when SeaBIOS reserves a region for stolen memory, the VM kernel has a hard time realizing the region is there and available. So far I just hard-coded the kernel/i915 driver since I know the address/range of the memory region that SeaBIOS will reserve for stolen memory in my case. It requires some hard-coding both in the kernel (https://elixir.bootlin.com/linux/v4.14.114/source/arch/x86/kernel/early-quirks.c#L436) and in the driver (https://elixir.bootlin.com/linux/v4.14.114/source/drivers/gpu/drm/i915/i915_gem_stolen.c#L404). Are you aware of any discussions around this problem? I wanted to see if it has already been discussed before looking at proposing some kind of patch to the kernel/driver. Thanks! Micah On Tue, Jun 4, 2019 at 1:44 PM Micah Morton wrote: > > Ok thanks, I figured out my issue. Turns out SeaBIOS in the VM tries > to map from 0x10-0x800 as "System RAM" (which holds the kernel > code/data/bss sections). On my device the sbios usually only maps > 0x10-0x7a988fff for this purpose. So when the i915 driver tries to > reserve 0x7c00-0x7fff for the "Graphics Stolen Memory" this > normally succeeds on my device but fails in the VM since the kernel > sees something already mapping this area of memory. This can be fixed > by patching SeaBIOS to only reserve up through 0x7bff for the > "System RAM" for the kernel. Somehow the 4.14 kernel is smart enough > to remap 0x7c00-0x7fff for the Graphics Stolen Memory even if > it has been claimed by SeaBIOS for System RAM (so this problem doesn't > show up when running 4.14 in the VM), but 4.4 is not. I'm trying to > get the 4.4 kernel working in the VM since I think those drivers have > been more thoroughly tested for my hardware. > > On Mon, Jun 3, 2019 at 3:51 PM Alex Williamson > wrote: > > > > On Mon, 3 Jun 2019 14:38:49 -0700 > > Micah Morton wrote: > > > > > Hi Alex, > > > > > > Could you remind me whether there is a minimum recommended kernel > > > version to be running in the VM guest when doing GPU passthrough? > > > > > > I'm fine running 4.14 in the host, but was looking to see if I could > > > run 4.4 in the guest and couldn't remember if it is advised to use a > > > newer kernel in the guest or if there is any reason to have the guest > > > kernel match the host kernel version? > > > > I don't recall any specific guest version dependency. I've got an old > > 4.9 guest around and that's about the only direct IGD assignment VM I > > test on a semi-regular basis. I do keep that host relatively updated, > > so it's currently a 5.0/4.9 combination. No reason to keep them in sync > > that I know of. Thanks, > > > > Alex ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Question about integrated GPU passthrough and initialization
Ok thanks, I figured out my issue. Turns out SeaBIOS in the VM tries to map from 0x10-0x800 as "System RAM" (which holds the kernel code/data/bss sections). On my device the sbios usually only maps 0x10-0x7a988fff for this purpose. So when the i915 driver tries to reserve 0x7c00-0x7fff for the "Graphics Stolen Memory" this normally succeeds on my device but fails in the VM since the kernel sees something already mapping this area of memory. This can be fixed by patching SeaBIOS to only reserve up through 0x7bff for the "System RAM" for the kernel. Somehow the 4.14 kernel is smart enough to remap 0x7c00-0x7fff for the Graphics Stolen Memory even if it has been claimed by SeaBIOS for System RAM (so this problem doesn't show up when running 4.14 in the VM), but 4.4 is not. I'm trying to get the 4.4 kernel working in the VM since I think those drivers have been more thoroughly tested for my hardware. On Mon, Jun 3, 2019 at 3:51 PM Alex Williamson wrote: > > On Mon, 3 Jun 2019 14:38:49 -0700 > Micah Morton wrote: > > > Hi Alex, > > > > Could you remind me whether there is a minimum recommended kernel > > version to be running in the VM guest when doing GPU passthrough? > > > > I'm fine running 4.14 in the host, but was looking to see if I could > > run 4.4 in the guest and couldn't remember if it is advised to use a > > newer kernel in the guest or if there is any reason to have the guest > > kernel match the host kernel version? > > I don't recall any specific guest version dependency. I've got an old > 4.9 guest around and that's about the only direct IGD assignment VM I > test on a semi-regular basis. I do keep that host relatively updated, > so it's currently a 5.0/4.9 combination. No reason to keep them in sync > that I know of. Thanks, > > Alex ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Question about integrated GPU passthrough and initialization
Hi, I have had fun recently trying to get vega gpu passthru working. After a lot of testing and some kernel patching I would say: 1) if you want vega gpu reset working, use kernel 5.1.5 or above in the guest as well as the host. 2) if you want passthru working instead of a blank screen at vm start, use OVMF efi instead of seabios. 3) you may or may not need a vfio patch that stops the graphics card resizing the BAR. Kind regards James On Mon, 3 Jun 2019, 22:40 Micah Morton, wrote: > Hi Alex, > > Could you remind me whether there is a minimum recommended kernel > version to be running in the VM guest when doing GPU passthrough? > > I'm fine running 4.14 in the host, but was looking to see if I could > run 4.4 in the guest and couldn't remember if it is advised to use a > newer kernel in the guest or if there is any reason to have the guest > kernel match the host kernel version? > > On Wed, May 29, 2019 at 10:00 AM Micah Morton > wrote: > > > > Ah my bad. Just realized I was using my own copy of SeaBIOS that I had > > built. When I use the copy from qemu-3.0.0/pc-bios/bios-256k.bin I see > > the i915 driver finding the OpRegion: > > [0.269341] in i915_driver_init_hw > > [0.269374] [drm] Memory usable by graphics device = 4096M > > [0.269585] in intel_opregion_setup > > [0.269600] graphic opregion physical addr: 0x7fffe000 > > > > Still working on getting the screen to light up > > > > On Wed, May 29, 2019 at 9:44 AM Alex Williamson > > wrote: > > > > > > On Wed, 29 May 2019 09:25:59 -0700 > > > Micah Morton wrote: > > > > > > > So as I mentioned, the ChromeOS firmware writes the location of the > > > > OpRegion to the ASLS PCI config register > > > > ( > https://github.com/coreboot/coreboot/blob/master/src/drivers/intel/gma/opregion.c#L88 > ). > > > > The i915 driver then gets the address for the OpRegion from that > > > > register here: > https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/i915/intel_opregion.c#L910 > . > > > > This all works for Chrome OS, but when we run a VM with SeaBIOS the > > > > ASLS PCI config register doesn't get written with the location of the > > > > OpRegion.: > > > > [0.263640] in i915_driver_init_hw (I added this) > > > > ... > > > > [0.263922] in intel_opregion_setup (and this) > > > > [0.263954] graphic opregion physical addr: 0x0 <-- This is > > > > supposed to point to the OpRegion, not be zero. > > > > [0.263954] ACPI OpRegion not supported! > > > > ... > > > > [0.267727] Failed to find VBIOS tables (VBT) > > > > > > > > I'm also not sure if the OpRegion is actually in VM memory or not. Do > > > > you think I need to find a way to put the OpRegion in VM memory as we > > > > have seen coreboot (Chrome OS firmware) do above? Or should using > > > > "x-igd-opregion=on" somehow ensure that the OpRegion makes it into VM > > > > memory? Clearly I at least need to find a way to set that ASLS PCI > > > > config register in the VM or modify the i915 driver that runs in the > > > > guest so it can find the OpRegion. > > > > > > In QEMU, vfio_pci_igd_opregion_init() adds the opregion to a fw_cfg > > > file "etc/igd-opregion" and makes the (virtual) ASLS register > > > writable. Then in SeaBIOS, any Intel vendor ID, PCI class VGA device > > > will trigger the intel_igd_setup() function, which looks for the fw_cfg > > > file, allocates space for it, and writes the GPA back to the ASLS > > > register. That's at least how it's supposed to work, which again > > > reminds me for the umpteenth time that x-igd-opregion only works with > > > SeaBIOS as OVMF has rejected this support in favor of an option ROM > > > based solution, which Intel never provided. I think you're using > > > SeaBIOS though so, so as long as that's not an ancient version it > > > should do the little dance here. The ASLS is writable though, we don't > > > do any write-once tricks, so something could blindly stomp on it. You > > > might enable logging in SeaBIOS, it will emit some spew for the > > > OpRegion support. You could also enable tracing to see the write of > > > the ASLS into QEMU. Thanks, > > > > > > Alex > > ___ > vfio-users mailing list > vfio-users@redhat.com > https://www.redhat.com/mailman/listinfo/vfio-users > ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Question about integrated GPU passthrough and initialization
On Mon, 3 Jun 2019 14:38:49 -0700 Micah Morton wrote: > Hi Alex, > > Could you remind me whether there is a minimum recommended kernel > version to be running in the VM guest when doing GPU passthrough? > > I'm fine running 4.14 in the host, but was looking to see if I could > run 4.4 in the guest and couldn't remember if it is advised to use a > newer kernel in the guest or if there is any reason to have the guest > kernel match the host kernel version? I don't recall any specific guest version dependency. I've got an old 4.9 guest around and that's about the only direct IGD assignment VM I test on a semi-regular basis. I do keep that host relatively updated, so it's currently a 5.0/4.9 combination. No reason to keep them in sync that I know of. Thanks, Alex ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Question about integrated GPU passthrough and initialization
Hi Alex, Could you remind me whether there is a minimum recommended kernel version to be running in the VM guest when doing GPU passthrough? I'm fine running 4.14 in the host, but was looking to see if I could run 4.4 in the guest and couldn't remember if it is advised to use a newer kernel in the guest or if there is any reason to have the guest kernel match the host kernel version? On Wed, May 29, 2019 at 10:00 AM Micah Morton wrote: > > Ah my bad. Just realized I was using my own copy of SeaBIOS that I had > built. When I use the copy from qemu-3.0.0/pc-bios/bios-256k.bin I see > the i915 driver finding the OpRegion: > [0.269341] in i915_driver_init_hw > [0.269374] [drm] Memory usable by graphics device = 4096M > [0.269585] in intel_opregion_setup > [0.269600] graphic opregion physical addr: 0x7fffe000 > > Still working on getting the screen to light up > > On Wed, May 29, 2019 at 9:44 AM Alex Williamson > wrote: > > > > On Wed, 29 May 2019 09:25:59 -0700 > > Micah Morton wrote: > > > > > So as I mentioned, the ChromeOS firmware writes the location of the > > > OpRegion to the ASLS PCI config register > > > (https://github.com/coreboot/coreboot/blob/master/src/drivers/intel/gma/opregion.c#L88). > > > The i915 driver then gets the address for the OpRegion from that > > > register here: > > > https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/i915/intel_opregion.c#L910. > > > This all works for Chrome OS, but when we run a VM with SeaBIOS the > > > ASLS PCI config register doesn't get written with the location of the > > > OpRegion.: > > > [0.263640] in i915_driver_init_hw (I added this) > > > ... > > > [0.263922] in intel_opregion_setup (and this) > > > [0.263954] graphic opregion physical addr: 0x0 <-- This is > > > supposed to point to the OpRegion, not be zero. > > > [0.263954] ACPI OpRegion not supported! > > > ... > > > [0.267727] Failed to find VBIOS tables (VBT) > > > > > > I'm also not sure if the OpRegion is actually in VM memory or not. Do > > > you think I need to find a way to put the OpRegion in VM memory as we > > > have seen coreboot (Chrome OS firmware) do above? Or should using > > > "x-igd-opregion=on" somehow ensure that the OpRegion makes it into VM > > > memory? Clearly I at least need to find a way to set that ASLS PCI > > > config register in the VM or modify the i915 driver that runs in the > > > guest so it can find the OpRegion. > > > > In QEMU, vfio_pci_igd_opregion_init() adds the opregion to a fw_cfg > > file "etc/igd-opregion" and makes the (virtual) ASLS register > > writable. Then in SeaBIOS, any Intel vendor ID, PCI class VGA device > > will trigger the intel_igd_setup() function, which looks for the fw_cfg > > file, allocates space for it, and writes the GPA back to the ASLS > > register. That's at least how it's supposed to work, which again > > reminds me for the umpteenth time that x-igd-opregion only works with > > SeaBIOS as OVMF has rejected this support in favor of an option ROM > > based solution, which Intel never provided. I think you're using > > SeaBIOS though so, so as long as that's not an ancient version it > > should do the little dance here. The ASLS is writable though, we don't > > do any write-once tricks, so something could blindly stomp on it. You > > might enable logging in SeaBIOS, it will emit some spew for the > > OpRegion support. You could also enable tracing to see the write of > > the ASLS into QEMU. Thanks, > > > > Alex ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Question about integrated GPU passthrough and initialization
Ah my bad. Just realized I was using my own copy of SeaBIOS that I had built. When I use the copy from qemu-3.0.0/pc-bios/bios-256k.bin I see the i915 driver finding the OpRegion: [0.269341] in i915_driver_init_hw [0.269374] [drm] Memory usable by graphics device = 4096M [0.269585] in intel_opregion_setup [0.269600] graphic opregion physical addr: 0x7fffe000 Still working on getting the screen to light up On Wed, May 29, 2019 at 9:44 AM Alex Williamson wrote: > > On Wed, 29 May 2019 09:25:59 -0700 > Micah Morton wrote: > > > So as I mentioned, the ChromeOS firmware writes the location of the > > OpRegion to the ASLS PCI config register > > (https://github.com/coreboot/coreboot/blob/master/src/drivers/intel/gma/opregion.c#L88). > > The i915 driver then gets the address for the OpRegion from that > > register here: > > https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/i915/intel_opregion.c#L910. > > This all works for Chrome OS, but when we run a VM with SeaBIOS the > > ASLS PCI config register doesn't get written with the location of the > > OpRegion.: > > [0.263640] in i915_driver_init_hw (I added this) > > ... > > [0.263922] in intel_opregion_setup (and this) > > [0.263954] graphic opregion physical addr: 0x0 <-- This is > > supposed to point to the OpRegion, not be zero. > > [0.263954] ACPI OpRegion not supported! > > ... > > [0.267727] Failed to find VBIOS tables (VBT) > > > > I'm also not sure if the OpRegion is actually in VM memory or not. Do > > you think I need to find a way to put the OpRegion in VM memory as we > > have seen coreboot (Chrome OS firmware) do above? Or should using > > "x-igd-opregion=on" somehow ensure that the OpRegion makes it into VM > > memory? Clearly I at least need to find a way to set that ASLS PCI > > config register in the VM or modify the i915 driver that runs in the > > guest so it can find the OpRegion. > > In QEMU, vfio_pci_igd_opregion_init() adds the opregion to a fw_cfg > file "etc/igd-opregion" and makes the (virtual) ASLS register > writable. Then in SeaBIOS, any Intel vendor ID, PCI class VGA device > will trigger the intel_igd_setup() function, which looks for the fw_cfg > file, allocates space for it, and writes the GPA back to the ASLS > register. That's at least how it's supposed to work, which again > reminds me for the umpteenth time that x-igd-opregion only works with > SeaBIOS as OVMF has rejected this support in favor of an option ROM > based solution, which Intel never provided. I think you're using > SeaBIOS though so, so as long as that's not an ancient version it > should do the little dance here. The ASLS is writable though, we don't > do any write-once tricks, so something could blindly stomp on it. You > might enable logging in SeaBIOS, it will emit some spew for the > OpRegion support. You could also enable tracing to see the write of > the ASLS into QEMU. Thanks, > > Alex ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Question about integrated GPU passthrough and initialization
On Wed, 29 May 2019 09:25:59 -0700 Micah Morton wrote: > So as I mentioned, the ChromeOS firmware writes the location of the > OpRegion to the ASLS PCI config register > (https://github.com/coreboot/coreboot/blob/master/src/drivers/intel/gma/opregion.c#L88). > The i915 driver then gets the address for the OpRegion from that > register here: > https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/i915/intel_opregion.c#L910. > This all works for Chrome OS, but when we run a VM with SeaBIOS the > ASLS PCI config register doesn't get written with the location of the > OpRegion.: > [0.263640] in i915_driver_init_hw (I added this) > ... > [0.263922] in intel_opregion_setup (and this) > [0.263954] graphic opregion physical addr: 0x0 <-- This is > supposed to point to the OpRegion, not be zero. > [0.263954] ACPI OpRegion not supported! > ... > [0.267727] Failed to find VBIOS tables (VBT) > > I'm also not sure if the OpRegion is actually in VM memory or not. Do > you think I need to find a way to put the OpRegion in VM memory as we > have seen coreboot (Chrome OS firmware) do above? Or should using > "x-igd-opregion=on" somehow ensure that the OpRegion makes it into VM > memory? Clearly I at least need to find a way to set that ASLS PCI > config register in the VM or modify the i915 driver that runs in the > guest so it can find the OpRegion. In QEMU, vfio_pci_igd_opregion_init() adds the opregion to a fw_cfg file "etc/igd-opregion" and makes the (virtual) ASLS register writable. Then in SeaBIOS, any Intel vendor ID, PCI class VGA device will trigger the intel_igd_setup() function, which looks for the fw_cfg file, allocates space for it, and writes the GPA back to the ASLS register. That's at least how it's supposed to work, which again reminds me for the umpteenth time that x-igd-opregion only works with SeaBIOS as OVMF has rejected this support in favor of an option ROM based solution, which Intel never provided. I think you're using SeaBIOS though so, so as long as that's not an ancient version it should do the little dance here. The ASLS is writable though, we don't do any write-once tricks, so something could blindly stomp on it. You might enable logging in SeaBIOS, it will emit some spew for the OpRegion support. You could also enable tracing to see the write of the ASLS into QEMU. Thanks, Alex ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Question about integrated GPU passthrough and initialization
So as I mentioned, the ChromeOS firmware writes the location of the OpRegion to the ASLS PCI config register (https://github.com/coreboot/coreboot/blob/master/src/drivers/intel/gma/opregion.c#L88). The i915 driver then gets the address for the OpRegion from that register here: https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/i915/intel_opregion.c#L910. This all works for Chrome OS, but when we run a VM with SeaBIOS the ASLS PCI config register doesn't get written with the location of the OpRegion.: [0.263640] in i915_driver_init_hw (I added this) ... [0.263922] in intel_opregion_setup (and this) [0.263954] graphic opregion physical addr: 0x0 <-- This is supposed to point to the OpRegion, not be zero. [0.263954] ACPI OpRegion not supported! ... [0.267727] Failed to find VBIOS tables (VBT) I'm also not sure if the OpRegion is actually in VM memory or not. Do you think I need to find a way to put the OpRegion in VM memory as we have seen coreboot (Chrome OS firmware) do above? Or should using "x-igd-opregion=on" somehow ensure that the OpRegion makes it into VM memory? Clearly I at least need to find a way to set that ASLS PCI config register in the VM or modify the i915 driver that runs in the guest so it can find the OpRegion. On Tue, May 28, 2019 at 1:43 PM Micah Morton wrote: > > Hey Alex, > > I'm seeing the firmware get a hold of the VBT > (https://github.com/coreboot/coreboot/blob/master/src/drivers/intel/gma/opregion.c#L253) > and write the location of the OpRegion to the ASLS PCI register > (https://github.com/coreboot/coreboot/blob/master/src/drivers/intel/gma/opregion.c#L88). > To sanity check, I booted and Chrome OS firmware put the OpRegion at > 0x7aa9b520 (I can do `mem rm 0x7aa9b520 16` in the host and see it > print out "IntelGraphicsMem"). So like you said, OpRegion is > definitely there and used in the host. > > This line > (https://github.com/coreboot/coreboot/blob/master/src/drivers/intel/gma/opregion.c#L312) > seems to imply that the VBT is being included in the OpRegion, so not > sure whats going wrong. I am right in the middle of debugging this so > I'll follow up on here if I have further specific questions. > > Thanks! > > On Tue, May 28, 2019 at 1:23 PM Alex Williamson > wrote: > > > > On Tue, 28 May 2019 09:35:16 -0700 > > Micah Morton wrote: > > > > > Ah ok thanks! > > > > > > The qemu command line i was using is here: `qemu-system-x86_64 > > > -chardev stdio,id=seabios -device > > > isa-debugcon,iobase=0x402,chardev=seabios -m 2G -smp 2 -M pc -vga none > > > -usbdevice tablet -cpu host,-invpcid,-tsc-deadline,check -drive > > > 'file=/path/to/image.bin,index=0,media=disk,cache=unsafe,format=raw' > > > -enable-kvm -device > > > vfio-pci,x-igd-opregion=on,host=00:02.0,id=hostdev0,bus=pci.0,addr=0x2,rombar=0 > > > -device 'virtio-net,netdev=eth0' -netdev > > > 'user,id=eth0,net=10.0.2.0/27,hostfwd=tcp:127.0.0.1:9222-:22'` > > > > > > It didn't work, but now at least I know why: > > > [0.316117] i915 :00:02.0: No more image in the PCI ROM > > > [0.316261] [drm] Failed to find VBIOS tables (VBT) > > > > > > If I can expose the VBT to the VM maybe it will work :) > > > > Hmm, looking at i915 it seems it didn't find this VBT thing in the > > OpRegion so tried to look at the ROM, which comments indicate would > > only be the VBT location on an older device. QEMU should fail if > > x-igd-opregion=on is specified but the host kernel didn't provide an > > OpRegion at all, so we've at least done some minimal sanity checking at > > the host kernel before exposing it, but maybe the OpRegion is missing > > some things on this chrome device vs a standard pc? Maybe Chrome OS > > uses a modified i915 driver that doesn't depend on it so the firmware > > guys stripped it? You could write a minimal vfio driver to dump > > the opregion data if you want to parse it by hand. Thanks, > > > > Alex ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Question about integrated GPU passthrough and initialization
Hey Alex, I'm seeing the firmware get a hold of the VBT (https://github.com/coreboot/coreboot/blob/master/src/drivers/intel/gma/opregion.c#L253) and write the location of the OpRegion to the ASLS PCI register (https://github.com/coreboot/coreboot/blob/master/src/drivers/intel/gma/opregion.c#L88). To sanity check, I booted and Chrome OS firmware put the OpRegion at 0x7aa9b520 (I can do `mem rm 0x7aa9b520 16` in the host and see it print out "IntelGraphicsMem"). So like you said, OpRegion is definitely there and used in the host. This line (https://github.com/coreboot/coreboot/blob/master/src/drivers/intel/gma/opregion.c#L312) seems to imply that the VBT is being included in the OpRegion, so not sure whats going wrong. I am right in the middle of debugging this so I'll follow up on here if I have further specific questions. Thanks! On Tue, May 28, 2019 at 1:23 PM Alex Williamson wrote: > > On Tue, 28 May 2019 09:35:16 -0700 > Micah Morton wrote: > > > Ah ok thanks! > > > > The qemu command line i was using is here: `qemu-system-x86_64 > > -chardev stdio,id=seabios -device > > isa-debugcon,iobase=0x402,chardev=seabios -m 2G -smp 2 -M pc -vga none > > -usbdevice tablet -cpu host,-invpcid,-tsc-deadline,check -drive > > 'file=/path/to/image.bin,index=0,media=disk,cache=unsafe,format=raw' > > -enable-kvm -device > > vfio-pci,x-igd-opregion=on,host=00:02.0,id=hostdev0,bus=pci.0,addr=0x2,rombar=0 > > -device 'virtio-net,netdev=eth0' -netdev > > 'user,id=eth0,net=10.0.2.0/27,hostfwd=tcp:127.0.0.1:9222-:22'` > > > > It didn't work, but now at least I know why: > > [0.316117] i915 :00:02.0: No more image in the PCI ROM > > [0.316261] [drm] Failed to find VBIOS tables (VBT) > > > > If I can expose the VBT to the VM maybe it will work :) > > Hmm, looking at i915 it seems it didn't find this VBT thing in the > OpRegion so tried to look at the ROM, which comments indicate would > only be the VBT location on an older device. QEMU should fail if > x-igd-opregion=on is specified but the host kernel didn't provide an > OpRegion at all, so we've at least done some minimal sanity checking at > the host kernel before exposing it, but maybe the OpRegion is missing > some things on this chrome device vs a standard pc? Maybe Chrome OS > uses a modified i915 driver that doesn't depend on it so the firmware > guys stripped it? You could write a minimal vfio driver to dump > the opregion data if you want to parse it by hand. Thanks, > > Alex ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Question about integrated GPU passthrough and initialization
On Tue, 28 May 2019 09:35:16 -0700 Micah Morton wrote: > Ah ok thanks! > > The qemu command line i was using is here: `qemu-system-x86_64 > -chardev stdio,id=seabios -device > isa-debugcon,iobase=0x402,chardev=seabios -m 2G -smp 2 -M pc -vga none > -usbdevice tablet -cpu host,-invpcid,-tsc-deadline,check -drive > 'file=/path/to/image.bin,index=0,media=disk,cache=unsafe,format=raw' > -enable-kvm -device > vfio-pci,x-igd-opregion=on,host=00:02.0,id=hostdev0,bus=pci.0,addr=0x2,rombar=0 > -device 'virtio-net,netdev=eth0' -netdev > 'user,id=eth0,net=10.0.2.0/27,hostfwd=tcp:127.0.0.1:9222-:22'` > > It didn't work, but now at least I know why: > [0.316117] i915 :00:02.0: No more image in the PCI ROM > [0.316261] [drm] Failed to find VBIOS tables (VBT) > > If I can expose the VBT to the VM maybe it will work :) Hmm, looking at i915 it seems it didn't find this VBT thing in the OpRegion so tried to look at the ROM, which comments indicate would only be the VBT location on an older device. QEMU should fail if x-igd-opregion=on is specified but the host kernel didn't provide an OpRegion at all, so we've at least done some minimal sanity checking at the host kernel before exposing it, but maybe the OpRegion is missing some things on this chrome device vs a standard pc? Maybe Chrome OS uses a modified i915 driver that doesn't depend on it so the firmware guys stripped it? You could write a minimal vfio driver to dump the opregion data if you want to parse it by hand. Thanks, Alex ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Question about integrated GPU passthrough and initialization
Ah ok thanks! The qemu command line i was using is here: `qemu-system-x86_64 -chardev stdio,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios -m 2G -smp 2 -M pc -vga none -usbdevice tablet -cpu host,-invpcid,-tsc-deadline,check -drive 'file=/path/to/image.bin,index=0,media=disk,cache=unsafe,format=raw' -enable-kvm -device vfio-pci,x-igd-opregion=on,host=00:02.0,id=hostdev0,bus=pci.0,addr=0x2,rombar=0 -device 'virtio-net,netdev=eth0' -netdev 'user,id=eth0,net=10.0.2.0/27,hostfwd=tcp:127.0.0.1:9222-:22'` It didn't work, but now at least I know why: [0.316117] i915 :00:02.0: No more image in the PCI ROM [0.316261] [drm] Failed to find VBIOS tables (VBT) If I can expose the VBT to the VM maybe it will work :) On Fri, May 24, 2019 at 2:19 PM Alex Williamson wrote: > > On Fri, 24 May 2019 14:10:03 -0700 > Micah Morton wrote: > > > On Fri, May 24, 2019 at 12:50 PM Alex Williamson > > wrote: > > > > > > On Fri, 24 May 2019 11:12:41 -0700 > > > Micah Morton wrote: > > > > > > > I’ve been working with an Intel Chrome OS device to see if integrated > > > > GPU passthrough works. The device is a 7th Generation (Kaby Lake) > > > > Intel Core i5-7Y57 with HD Graphics 615. So no discrete GPU. I think > > > > iGPU passthrough should work on this device. > > > > > > > > Initializing the graphics hardware has proven to be the trickiest part > > > > in all of this, and I have a question I thought someone on this list > > > > might be able to answer: > > > > > > > > Why does vfio enforce that I _need_ a VGA rom to be available in order > > > > to boot the guest in legacy passthrough mode > > > > (https://github.com/qemu/qemu/blob/master/hw/vfio/pci-quirks.c#L1616)? > > > > > > "Legacy" mode is largely about setting up components to make the VGA > > > ROM work. Some of the components of legacy mode are faking stolen > > > memory (which really only works for the ROM, not the OS driver), adding > > > the LPC bridge (not sure of the OS driver dependency on this, but I > > > think it can be manually added if needed), and adding the OpRegion > > > (which can also be manually added). If you only care about OS level > > > graphics initialization then you probably only need the OpRegion > > > support, which Intel vGPU also needs but Intel can't provide a > > > consistent story about how it should work. So why do you think you > > > need legacy assignment mode? Granted UPT mode is just about a > > > forgotten ideal at Intel, but legacy mode doesn't really solve the > > > stolen memory dependencies for anyone but the VGA ROM. > > > > Ah ok. Yeah I have no reason to think I need "legacy" mode apart from > > the fact that I wanted the GPU to be the "primary and exclusive > > graphics device in the VM". Is the only alternative UPT mode, which > > "does not support direct video output" > > (https://android.googlesource.com/platform/external/qemu/+/emu-master-dev/docs/igd-assign.txt#14)? > > So I can definitely pass through the device to the guest and get the > > i915 driver in the guest to attach to it on some level: > > > > ~ # lspci -nnk (in the guest) > > ... > > 00:02.0 VGA compatible controller [0300]: Intel Corporation Device > > [8086:591e] (rev 02) > > Subsystem: Intel Corporation Device [8086:2212] > > Kernel driver in use: i915 > > > > I guess the question for me now is whether I need OpRegion support in > > the guest (or anything else) for the guest driver to successfully use > > the GPU (I just see a black screen right now and haven't yet enabled > > much in the way of meaningful logs for i915). Do you have any pointers > > to resources on how the OpRegion can be added? Or am I better off > > contacting Intel engineers? > > If you're using QEMU by command line, just add x-igd-opregion=on to the > set of device parameters for the vfio-pci device. libvirt does not > provide a supported way to add this (the 'x-' prefix indicates an > experimental option and is used here because it's rather a hack), but > you can use elements to pass it, such as documented here: > http://vfio.blogspot.com/2016/09/passing-qemu-command-line-options.html > It's entirely possible that this alone will be enough to light up the > display. Thanks, > > Alex ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Question about integrated GPU passthrough and initialization
On Fri, 24 May 2019 14:10:03 -0700 Micah Morton wrote: > On Fri, May 24, 2019 at 12:50 PM Alex Williamson > wrote: > > > > On Fri, 24 May 2019 11:12:41 -0700 > > Micah Morton wrote: > > > > > I’ve been working with an Intel Chrome OS device to see if integrated > > > GPU passthrough works. The device is a 7th Generation (Kaby Lake) > > > Intel Core i5-7Y57 with HD Graphics 615. So no discrete GPU. I think > > > iGPU passthrough should work on this device. > > > > > > Initializing the graphics hardware has proven to be the trickiest part > > > in all of this, and I have a question I thought someone on this list > > > might be able to answer: > > > > > > Why does vfio enforce that I _need_ a VGA rom to be available in order > > > to boot the guest in legacy passthrough mode > > > (https://github.com/qemu/qemu/blob/master/hw/vfio/pci-quirks.c#L1616)? > > > > "Legacy" mode is largely about setting up components to make the VGA > > ROM work. Some of the components of legacy mode are faking stolen > > memory (which really only works for the ROM, not the OS driver), adding > > the LPC bridge (not sure of the OS driver dependency on this, but I > > think it can be manually added if needed), and adding the OpRegion > > (which can also be manually added). If you only care about OS level > > graphics initialization then you probably only need the OpRegion > > support, which Intel vGPU also needs but Intel can't provide a > > consistent story about how it should work. So why do you think you > > need legacy assignment mode? Granted UPT mode is just about a > > forgotten ideal at Intel, but legacy mode doesn't really solve the > > stolen memory dependencies for anyone but the VGA ROM. > > Ah ok. Yeah I have no reason to think I need "legacy" mode apart from > the fact that I wanted the GPU to be the "primary and exclusive > graphics device in the VM". Is the only alternative UPT mode, which > "does not support direct video output" > (https://android.googlesource.com/platform/external/qemu/+/emu-master-dev/docs/igd-assign.txt#14)? > So I can definitely pass through the device to the guest and get the > i915 driver in the guest to attach to it on some level: > > ~ # lspci -nnk (in the guest) > ... > 00:02.0 VGA compatible controller [0300]: Intel Corporation Device > [8086:591e] (rev 02) > Subsystem: Intel Corporation Device [8086:2212] > Kernel driver in use: i915 > > I guess the question for me now is whether I need OpRegion support in > the guest (or anything else) for the guest driver to successfully use > the GPU (I just see a black screen right now and haven't yet enabled > much in the way of meaningful logs for i915). Do you have any pointers > to resources on how the OpRegion can be added? Or am I better off > contacting Intel engineers? If you're using QEMU by command line, just add x-igd-opregion=on to the set of device parameters for the vfio-pci device. libvirt does not provide a supported way to add this (the 'x-' prefix indicates an experimental option and is used here because it's rather a hack), but you can use elements to pass it, such as documented here: http://vfio.blogspot.com/2016/09/passing-qemu-command-line-options.html It's entirely possible that this alone will be enough to light up the display. Thanks, Alex ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Question about integrated GPU passthrough and initialization
On Fri, 24 May 2019 11:12:41 -0700 Micah Morton wrote: > I’ve been working with an Intel Chrome OS device to see if integrated > GPU passthrough works. The device is a 7th Generation (Kaby Lake) > Intel Core i5-7Y57 with HD Graphics 615. So no discrete GPU. I think > iGPU passthrough should work on this device. > > Initializing the graphics hardware has proven to be the trickiest part > in all of this, and I have a question I thought someone on this list > might be able to answer: > > Why does vfio enforce that I _need_ a VGA rom to be available in order > to boot the guest in legacy passthrough mode > (https://github.com/qemu/qemu/blob/master/hw/vfio/pci-quirks.c#L1616)? "Legacy" mode is largely about setting up components to make the VGA ROM work. Some of the components of legacy mode are faking stolen memory (which really only works for the ROM, not the OS driver), adding the LPC bridge (not sure of the OS driver dependency on this, but I think it can be manually added if needed), and adding the OpRegion (which can also be manually added). If you only care about OS level graphics initialization then you probably only need the OpRegion support, which Intel vGPU also needs but Intel can't provide a consistent story about how it should work. So why do you think you need legacy assignment mode? Granted UPT mode is just about a forgotten ideal at Intel, but legacy mode doesn't really solve the stolen memory dependencies for anyone but the VGA ROM. > The Intel device I’m working with normally doesn’t initialize the GPU > at all until the kernel is running, at which point the i915 driver > does the initialization. So there is never actually any VGA rom > anywhere on the system at any time for me to grab. I suppose I maybe > could get one for this hardware from Intel or the web somewhere, but > seems like if the device can normally initialize the GPU hardware from > the kernel driver then it wouldn’t be too unreasonable to pass through > the GPU and let the guest (with the same i915 driver) initialize the > GPU without using a VGA rom. Do you think it would be reasonable for > me to find a way to patch qemu/vfio/SeaBIOS to allow for this > scenario? I'm still not sure what you're missing versus the options we provide otherwise. If your dependency is actually on stolen memory, then trying to re-enable the hack for legacy mode is likely not to work for the OS driver. The fix for that would mean having the host dictate to the VM an address space hole such that stolen memory could be identity mapped into the VM (plus safety things like clearing that memory between users to avoid leaking data). Thanks, Alex ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Question about integrated GPU passthrough and initialization
On Fri, 24 May 2019, at 7:17 PM, Micah Morton wrote: > I’ve been working with an Intel Chrome OS device to see if integrated > GPU passthrough works. The device is a 7th Generation (Kaby Lake) > Intel Core i5-7Y57 with HD Graphics 615. So no discrete GPU. I think > iGPU passthrough should work on this device. > > Initializing the graphics hardware has proven to be the trickiest part > in all of this, and I have a question I thought someone on this list > might be able to answer: > > Why does vfio enforce that I _need_ a VGA rom to be available in order > to boot the guest in legacy passthrough mode > (https://github.com/qemu/qemu/blob/master/hw/vfio/pci-quirks.c#L1616)? > The Intel device I’m working with normally doesn’t initialize the GPU > at all until the kernel is running, at which point the i915 driver > does the initialization. So there is never actually any VGA rom > anywhere on the system at any time for me to grab. I suppose I maybe > could get one for this hardware from Intel or the web somewhere, but > seems like if the device can normally initialize the GPU hardware from > the kernel driver then it wouldn’t be too unreasonable to pass through > the GPU and let the guest (with the same i915 driver) initialize the > GPU without using a VGA rom. Do you think it would be reasonable for > me to find a way to patch qemu/vfio/SeaBIOS to allow for this > scenario? > > I’ve tried commenting out some blocks in this function > (https://github.com/qemu/qemu/blob/master/hw/vfio/pci-quirks.c#L1557) > and patching SeaBIOS to avoid doing any kind of graphics init stuff in > the guest, and am still working through the debugging, but was > wondering if this is a reasonable approach? Or something that has been > done before? > > Thanks, > Micah > > ___ > vfio-users mailing list > vfio-users@redhat.com > https://www.redhat.com/mailman/listinfo/vfio-users > I don't know about iGPU but for nVidia Quadro cards which I am passing through, I do not have a Rom. I use "" element in the device definition, and it just works, with an obvious limitation that guest doesn't show any image onscreen until the system is fully started. Perhaps it also helps that all my virtual machines are UEFI, not sure. B. -- Bronek Kozicki b...@incorrekt.com ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
[vfio-users] Question about integrated GPU passthrough and initialization
I’ve been working with an Intel Chrome OS device to see if integrated GPU passthrough works. The device is a 7th Generation (Kaby Lake) Intel Core i5-7Y57 with HD Graphics 615. So no discrete GPU. I think iGPU passthrough should work on this device. Initializing the graphics hardware has proven to be the trickiest part in all of this, and I have a question I thought someone on this list might be able to answer: Why does vfio enforce that I _need_ a VGA rom to be available in order to boot the guest in legacy passthrough mode (https://github.com/qemu/qemu/blob/master/hw/vfio/pci-quirks.c#L1616)? The Intel device I’m working with normally doesn’t initialize the GPU at all until the kernel is running, at which point the i915 driver does the initialization. So there is never actually any VGA rom anywhere on the system at any time for me to grab. I suppose I maybe could get one for this hardware from Intel or the web somewhere, but seems like if the device can normally initialize the GPU hardware from the kernel driver then it wouldn’t be too unreasonable to pass through the GPU and let the guest (with the same i915 driver) initialize the GPU without using a VGA rom. Do you think it would be reasonable for me to find a way to patch qemu/vfio/SeaBIOS to allow for this scenario? I’ve tried commenting out some blocks in this function (https://github.com/qemu/qemu/blob/master/hw/vfio/pci-quirks.c#L1557) and patching SeaBIOS to avoid doing any kind of graphics init stuff in the guest, and am still working through the debugging, but was wondering if this is a reasonable approach? Or something that has been done before? Thanks, Micah ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
