On 2015-03-25 18:29, Alex Williamson wrote: > On Wed, 2015-03-25 at 18:00 +0000, Gordan Bobic wrote: >> On 2015-03-25 17:09, Alex Williamson wrote: >> > On Wed, 2015-03-25 at 16:00 +0000, Gordan Bobic wrote: >> >> On 2015-03-25 15:46, Alex Williamson wrote: >> >> > On Wed, 2015-03-25 at 16:25 +0100, Laszlo Ersek wrote: >> >> >> On 03/25/15 11:12, Gordan Bobic wrote: >> >> >> > Hi, >> >> >> > >> >> >> > I'm trying to solve what should be a relatively simple problem >> >> >> > of side-loading a GPU VBIOS but I am running into an unexpected >> >> >> > problem. >> >> >> > >> >> >> > The setup is a laptop with Optimus setup (Intel primary + Nvidia >> >> >> > headless), and I want to pass the headless Nvidia GPU to the KVM >> >> >> > virtual machine. Unfortunately, the GPU appears to be >> >> >> > uninitialized in the VM, judging by the GPU-Z symptoms. >> >> >> > >> >> >> > The VBIOS in question isn't on an EEPROM attached to the GPU, it >> >> >> > is instead in the main system BIOS payload (which actually contains >> >> >> > 3 Nvidia BIOS-es, presumably for different compatible GPUs). I >> >> >> > extracted all the BIOS blobs using PhoenixTool and deduced the >> >> >> > correct one, but the blob in question appears to have the EFI >> >> >> > header missing - which means it looks much like a legacy BIOS, >> >> >> > rather than an EFI one (type 0 header is there, but type 3 >> >> >> > header is not. But the crypto certs are in fact there in the >> >> >> > BIOS payload, which indicates it is an EFI BIOS after all, >> >> >> > only with the type 3 header stripped out. >> >> >> >> >> >> You can use the EfiRom utility from the edk2 git tree, with the >> >> >> "--dump" >> >> >> option, to "Dump the headers of an existing option ROM image". >> >> >> >> >> >> I don't know of any utility offhand that extracts images from >> >> >> expansion >> >> >> ROMs (splitting off the expansion ROM header and the PCI Data >> >> >> Structure >> >> >> etc). >> >> >> >> >> >> > Is the EFI header as described (usually 1KB on Nvidia Kepler >> >> >> > BIOS-es and 1.5KB on Nvidia Maxwell BIOS-es) only applicable >> >> >> > to discrete EFI BIOS-es? Is it handled differently if the >> >> >> > GPU BIOS is integrated into the main system BIOS payload? >> >> >> >> >> >> Normally the UEFI driver is built as a UEFI executable. The EfiRom >> >> >> utility (or another equivalent tool) is used to build it (possibly >> >> >> together with other such executables) into an expansion ROM. This >> >> >> procedure formats the "PCI Data Structure" and "PCI Expansion ROM >> >> >> Header" structs into the output file, as specified in the "PCI (TM) >> >> >> Firmware Specification Revision 3.1". >> >> >> >> >> >> > If so, is there a method for incorporating the GPU firmware >> >> >> > I have extracted into the OVMF core instead of side-loading >> >> >> > it? >> >> >> >> >> >> Yes, there is. (Well, assuming that whatever you extracted is indeed a >> >> >> UEFI binary.) >> >> >> >> >> >> Method #1 is to place the binary somewhere appropriate in the edk2 >> >> >> tree, >> >> >> write an INF file for it, include the INF file in >> >> >> "OvmfPkg/OvmfPkgX64.fdf", and rebuild OVMF. Since this is going to be >> >> >> a >> >> >> binary-only module, you can refer to the following examples: >> >> >> - FatBinPkg/EnhancedFatDxe/Fat.inf >> >> >> - EdkShellBinPkg/FullShell/FullShell.inf >> >> >> >> >> >> Method #2 is similar, except you don't need to write an INF file. >> >> >> Search >> >> >> the same FDF file for "Intel3.5/EFIX64/E3507X2.EFI". >> >> >> >> >> >> Method #3 is to prepare a disk image with a FAT32 partition. Then copy >> >> >> the extracted binary to this filesystem, using mtools or guestfish. >> >> >> During OVMF boot, enter the UEFI shell, and load the driver binary >> >> >> with >> >> >> the "load" command from this filesystem. (This should also connect the >> >> >> driver to matching devices at once.) >> >> >> >> >> >> Hm, I'm just noticing method #4: the UEFI shell has a command called >> >> >> "loadpcirom", which "Loads a UEFI driver from a file in the format of >> >> >> a >> >> >> PCI Option ROM". >> >> > >> >> > That's really cool that there are so many ways to do it, but IMHO the >> >> > most maintenance-free solution seems like it would be to stuff that >> >> > UEFI >> >> > binary into a PCI option ROM format and expose it to the VM as if the >> >> > GPU had a PCI option ROM, using the romfile= parameter (or <rom >> >> > file=''/> in libvirt). Then you don't need a custom OVMF build, nor do >> >> > you need to do any prep work to pre-load the UEFI driver into the EFI >> >> > partition. Thanks, >> >> >> >> That was the first thing I tried, but I'm finding it difficult to >> >> spot from the OVMF debug log to even tell if it is being loaded >> >> and executed, let alone of it works. >> > >> > If the ROM doesn't start with AA55, then it's not a valid PCI option >> > ROM >> > and won't be processed. >> >> It does begin with AA55 (well, 55AA, but it's in little-endian order): >> >> # xxd GTX860M.rom | head -20 >> 0000000: 55aa cbeb 4b37 3430 30e9 4c19 77cc 5649 U...K7400.L.w.VI >> 0000010: 4445 4f20 0d00 0000 9001 f71a 0000 4942 DEO ..........IB >> 0000020: 4d20 5647 4120 436f 6d70 6174 6962 6c65 M VGA Compatible >> 0000030: 0100 0000 4000 5a57 3033 2f31 382f 3134 [email protected]/18/14 >> 0000040: 0000 0000 0000 0000 8010 5000 51fc 0000 ..........P.Q... >> 0000050: e978 2b00 aa17 1238 ffff ffff 0000 0000 .x+....8........ >> 0000060: efff ffff 0000 0080 5c56 a541 e925 49e9 ........\V.A.%I. >> 0000070: 2c49 504d 4944 6c00 6f00 0000 00a0 00b0 ,IPMIDl.o....... >> 0000080: 00b8 00c0 0033 474d 3130 3720 4532 3730 .....3GM107 E270 >> 0000090: 3420 534b 5520 3130 2056 4741 2042 494f 4 SKU 10 VGA BIO >> 00000a0: 5320 2848 5744 4941 4729 0d0a 0000 0000 S (HWDIAG)...... >> 00000b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ >> 00000c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ >> 00000d0: 0000 0000 0000 0056 6572 7369 6f6e 2038 .......Version 8 >> 00000e0: 322e 3037 2e33 342e 3030 2e30 3820 0d0a 2.07.34.00.08 .. >> 00000f0: 0043 6f70 7972 6967 6874 2028 4329 2031 .Copyright (C) 1 >> 0000100: 3939 362d 3230 3134 204e 5649 4449 4120 996-2014 NVIDIA >> 0000110: 436f 7270 2e0d 0a00 0000 ffff 0000 0000 Corp............ >> 0000120: ffff 474d 3130 3720 426f 6172 6420 2d20 ..GM107 Board - >> 0000130: 3237 3034 3030 3130 0000 0000 0000 0000 27040010........ >> >> >> > If you're running a version of OVMF without CSM >> > support and the ROM doesn't include a valid type 3 header, then it will >> > not be processed, regardless of whether you believe it's actually a >> > UEFI >> > payload. What you have in the other reply looks like a legacy BIOS, >> > and >> > it very well may work as such, but it's going to require the CSM and >> > likely make callbacks that require VGA support, which is what I expect >> > you're trying to avoid by using OVMF. >> >> I tried the OVMF-with-csm firmware, and that seems to take quite a >> long >> time to get to the shell, but in the end the result is the same. > > What are you expecting to happen when the VM firmware executes the > option ROM? Typically this would be the point where the screen > connected to the GPU gets sync and we get some kind of splash screen > from the VBIOS or seabios/OVMF output. But your GPU is headless, so > how > would we know if it worked?
I was hoping something might appear in the debug log via port 0x402, but I couldn't spot any indication either way last time I looked. > If you're using the CSM firmware and the option ROM is legacy, that > also > likely means that it's trying to access legacy VGA addresses, which it > cannot do unless you've enabled VGA support for the assigned device > _and_ patched your host i915 drivers to obey VGA arbitration. If dmesg > on the host indicates some i915 errors coincident with starting the VM, > then we can be pretty sure that's a legacy BIOS and it might work if > the > i915 driver wasn't lying about claiming VGA resources. I was kind of hoping to go with either legacy-free or secondary passthrough to avoid all that. The problem is that the standard Nvidia driver fails to install: 1) Nvidia installer refuses to install the driver saying it couldn't find any compatible hardware 2) Installing the driver manually makes Windows say the driver is not the this OS (hardly unlikely) 3) Hacking the inf (actually, getting one from laptopvideo2go.com) makes the driver install just fine, and it shows up as working with relevant resource allocations in the Device Manager, but it doesn't _actually_ work. GPU-Z shows 2990MB of RAM, no BIOS version, and no clock speeds, all typical symptoms of exactly what happens when the card isn't initialized properly (e.g. when flashing a BIOS with an incorrect checksum onto the card - I have done enough hacking away at Nvidia BIOS-es to be familiar with it). And Nvidia control panel silently fails to load. Hence why I am looking in the direction of POST-ing the card using it's BIOS. I tried even using a 750Ti UEFI BIOS (same GPU, so I would expect it to work, apart from maybe initializing only half the VRAM). All that happens when I use a ROM file is that at startup I get an error that says that Nvidia streaming failed to start (which doesn't happen when I boot into the VM with a card without supplying a ROM). I tried removing the QXL emulated GPU, but I just get serial OVMF console and nothing at all after booting, nothing on the VNC server in the guest that I'd set up. Which leads me to think that the GPU really isn't initializing properly. Or maybe it just doesn't work without at least a fake monitor - only I cannot set up a resolution on it manually because Nvidia control panel fails to load. >> On Nvidia cards starting with 6xx series, the legacy BIOS is >> essentially wrapped - there is a type 3 header (1024 bytes on Kepler >> GPUs, 1536 bytes on Maxwell GPUs), followed by the legacy BIOS >> payload, >> followed by the EFI crypto certs. >> >> What appears to have been extracted from the system BIOS on my laptop >> is a front-truncated EFI VBIOS, with the EFI header missing. But >> looking at what PhoenixTool reports about the structures, the numbers >> are correct - there is no extra payload being reported. >> >> Which makes me wonder if Insyde UEFI system firmware is just being >> weird and proprietary. >> >> > If the blob you suspect as being >> > a raw UEFI binary (I'm not as hopeful as you and Laszlo seem to be) can >> > be used as a ROM, it would at least need to be wrapped in a PCI option >> > ROM header. Thanks, >> >> I'm pretty sure it is a valid PCI option ROM, it's just the type 3 >> header that seems to be missing. :( Does that mean I am plain out >> of luck? > > You've gone to a lot of effort so far, so I would put it past you to > create a PCI option ROM header for that blob, but that's the only way I > see making progress unless you want to go the legacy route. I don't think this can be just "cooked up". The 1536 bytes of EFI header in Nvidia BIOS-es includes a lot of important stuff, such as the software straps (and that's just 16 of the 1536 bytes). > It might be > worthwhile to try to legacy route and see if you can make the GPU do > what you want before going to so much effort to get a UEFI ROM for it. > The GPUs in hybrid laptops have a much lower success rate for > assignment > than do discrete cards. > > If it were my experiment, I'd patch my host kernel with the i915 VGA > arbitration patch, add the module option to enable it, setup a VM (w/o > the GPU at first) with remote access because the GPU is headless > (tightvnc server), then add the GPU and the ROM using romfile=, then > try > to install the drivers and see what works in the guest. If you can do > what you need with legacy, then start hacking on how to get a UEFI ROM > on the thing so you can avoid legacy VGA. Note that your VM is also > going to be headless. You will need to use remote graphics solutions > to > get to it. Thanks, Indeed, I am aware of this - installing the VNC server was the first thing I did after installing the guest OS. Unfortunatelym that doesn't seem to quite work, either due to the GPU not being properly initialized or due to it not having a monitor (fake or otherwise) attached to set the resolution, so all VNC returns when I remove the QXL GPU is a black screen. Maybe I was too optimistic in expecting the concept to just work with a hybrid GPU setup. I was trying to replicate a similar configuration from my desktop system, except there I am using a full fat desktop GPU rather than an Optimus hybrid, so initializing the GPU in the VM "just works". Gordan ------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ edk2-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/edk2-devel
