Thanks for that info Stephan. Yup it's an UEFI install since I'm using the following line in my bhyve script:
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu And yea it may be possible something is going on with AMD CPU hosts... Jonathan Vasquez PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 Sent with ProtonMail Secure Email Sent with [Proton Mail](https://proton.me/mail/home) secure email. -------- Original Message -------- On Saturday, 08/30/25 at 20:11 Jonathan Vasquez <j...@xyinn.org> wrote: > Hey all, > > I've updated my blog post with more information and some useful screenshots: > https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd > > The important stuff to mention is that I've also gotten a chance to play with > vm-bhyve, but for the purposes of making sure we get as clear information as > possible without having to unwrap the vm-bhyve scripts, I'll continue to use > bhyve directly. With that said, some useful info: > > - The vBIOS is not required when running on Windows as mentioned by Corvin > here: > https://lists.freebsd.org/archives/freebsd-virtualization/2022-June/000797.html, > thus I've stopped using it. I was however also able to successfully use his > "echo 1 ..." recommendation to dump the bios from within Linux (Fedora 42). I > actually failed to extract the bios before (without using amdvbflash) because > most likely my Linux Mint 22.1's Kernel was too old to extract the ROM. When > I tried to do the same exact thing today with Fedora 42, it worked, and I was > able to see the "rom" file from Fedora 42, but not from Linux Mint 22.1. > - I was able to get the GPU to load without a Code 43 error. But it's still > unusable. I was actually able to get to this state before but it was > inconsistent. Through more research I was able to get it to be reproducible. > There are a few things to mention here: > > - The card will always start in a Code 43 failed state regardless of what I > mention below (including if I reboot the host). > - If you use RDP, the card will always be in a Code 43 state. > - If I use VNC, the card will start in a failed state (as mentioned above), > HOWEVER, you will be able to go into the Device Manager, Disable the card, > and then Enable the card, and it will be enabled successfully as you can see > in my screenshots. The OS will also give it resources from what we can see in > the Device's Properties' Resources Tab. > > Given what I mentioned above, I would recommend (as some other people > recommended in other links I've found), to disable the card in the VM before > you shut it down, and then enable it from inside of Windows when you turn it > back on. Although I don't know how necessary this is given that if I use VNC, > even if it starts off in a failed state (let's say from a fresh host reboot), > doing the "device re-nabling trick" will let it load successfully again. At > this point since I've gotten the card to say it worked, I'm not sure how to > tell Windows to use that as the main GPU when I run my apps. I've looked at > some examples online of using display settings to change the preferred card, > but I don't see it listed there. Am I suppose to see the card in dxdiag if > I'm using VNC? I'm just running the bhyve VM on my server, and I VNC over > from my laptop on the same LAN network. I'm guessing if things are working > correctly I should be able to do that right? I know there are also some > benchmarking apps that show you what GPUs its detected and maybe allow you to > pick one to run a benchmark on. I tried opening FurMark but it fails saying > that it only detects Open GL 1.1 and all of that... I actually tried > installing Mesa for windows to see what happens and that does help but it's > slow and it's clearly not using the card and most likely doing some sort of > software rendering (maybe like a Windows llvmpipe). > > Jonathan Vasquez > PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 > Sent with ProtonMail Secure Email > > On Monday, August 25th, 2025 at 14:51, Jonathan Vasquez <j...@xyinn.org> > wrote: > >> No go for my testing inside a Linux Mint 22.1 VM. Fully updated, and passing >> in both 3/0/0 and 3/0/1 for my AMD Radeon card. I was able to extract some >> of the "dmesg" and "eglinfo" output. Running "glxinfo" also showed that the >> OpenGL renderer was set to llvmpipe, so software rendering :(. "lspci -k" >> detected it and showed that it was using the "amdgpu" kernel driver. Since >> I'm using vm-bhyve atm I didn't test this with the dumped rom, just directly >> passing them through. I've uploaded some of the logs here for now in case >> anyone wants to take a glance at them: >> >> https://xyinn.org/freebsd/files/gpu_pass/2025-08-25/ >> >> Jonathan Vasquez >> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >> Sent with ProtonMail Secure Email >> >> On Monday, August 25th, 2025 at 14:37, Robert Cocker <rcocke...@gmail.com> >> wrote: >> >>> Hey group has bhvye core dev baked out an update for this >>> >>> Robert >>> --------------------------------------------------------------- >>> >>> From: owner-freebsd-virtualizat...@freebsd.org >>> <owner-freebsd-virtualizat...@freebsd.org> on behalf of Jonathan Vasquez >>> <j...@xyinn.org> >>> Sent: Monday, August 25, 2025 11:35:31 AM >>> To: Jonathan Vasquez <j...@xyinn.org> >>> Cc: Stephan Althaus <stephan.alth...@duedinghausen.eu>; >>> virtualization@freebsd.org <virtualization@freebsd.org> >>> Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and >>> Windows 10 Pro) >>> >>> I was able to get a fresh Win 10 Pro VM running through vm-bhyve but no >>> progress. Still same Code 43 error as before. I wasn't able to figure out >>> how to pass the rom=bios.rom through vm-bhyve, I didn't see anything about >>> it in the config.sample file and I also tried to bypass that by adding it >>> in bhyve_options but no go, the VM wouldn't even start if I did that. I see >>> there is a "bhyve_devices" that contains all of the slot devices but I >>> don't see that as an adjustable variable based on the config.sample info. >>> I've pasted my current vm-bhyve template below. My next test is to try the >>> same thing inside a Linux guest to see what happens. >>> >>> ---------------- >>> >>> loader="uefi" >>> cpu="4" >>> cpu_sockets="1" >>> cpu_cores="4" >>> cpu_threads="1" >>> memory="16G" >>> wired_memory="yes" >>> >>> utctime="no" >>> ahci_device_limit="8" >>> >>> disk0_type="nvme" >>> disk0_name="disk0.img" >>> >>> network0_type="e1000" >>> network0_switch="public" >>> >>> passthru0="3/0/0=2:0" >>> passthru1="3/0/1=2:1" >>> >>> #bhyve_options="-s 2:0,passthru,3/0/0,rom=bios.rom" >>> >>> # Using RDP post-install. >>> graphics="no" >>> graphics_port="5900" >>> graphics_listen="0.0.0.0" >>> graphics_res="1920x1080" >>> graphics_wait="yes" >>> >>> xhci_mouse="yes" >>> uuid="535d54b5-81cb-11f0-93a2-98b78501bda8" >>> network0_mac="58:9c:fc:04:5a:eb" >>> >>> ---------------- >>> >>> Jonathan Vasquez >>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >>> Sent with ProtonMail Secure Email >>> >>> On Monday, August 25th, 2025 at 11:05, Jonathan Vasquez <j...@xyinn.org> >>> wrote: >>> >>>> Unfortunately I don't have an intel machine with an available PCIe slot. >>>> However I'm trying another thing which is to passthrough the card to a >>>> Linux guest. I need to run more experiments but a preliminary boot of a >>>> Linux Mint ISO did show the AMD Radeon RX 6900 XT card in lspci and >>>> showed that it was using the amdgpu driver. I then did a basic glxgears >>>> and it opened up in my VNC window and showed the FPS counter at over 2000 >>>> fps. The only issue I noticed was when shutting down the live CD >>>> environment there were amdgpu failure messages where it was failing to >>>> write something to it. I'm currently redoing my "basic bhyve script" and >>>> moving it over to "vm-bhyve" since it seems a lot of people are using that >>>> instead of the more manual approach. This should hopefully help simplify >>>> things for debugging. If the card works on Linux via passthrough this >>>> would also be useful info and can allow me to see the performance of >>>> Cyperpunk 2077, and some other games (even if I'm not getting 100% of the >>>> card's performance). >>>> >>>> Jonathan Vasquez >>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >>>> Sent with ProtonMail Secure Email >>>> >>>> On Monday, August 25th, 2025 at 11:00, Stephan Althaus >>>> <stephan.alth...@duedinghausen.eu> wrote: >>>> >>>>> On 8/25/25 16:56, Jonathan Vasquez wrote: >>>>> >>>>>> Hey Stephan, >>>>>> >>>>>> Yup, I've tried passing both 3/0/0 and 3/0/1 and no go, and I also make >>>>>> sure to reboot the host in between runs due to the amd hardware issue. I >>>>>> don't believe 3/0/1 is necessary though based on the examples by Corvin >>>>>> in his talk. He only passed in 3/0/0 (I'm guessing his card also has a >>>>>> 3/0/1 function that he isn't using since that's the audio channel). >>>>>> >>>>>> Jonathan Vasquez >>>>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >>>>>> Sent with ProtonMail Secure Email >>>>>> >>>>>> On Monday, August 25th, 2025 at 10:52, Stephan Althaus >>>>>> <stephan.alth...@duedinghausen.eu> wrote: >>>>>> >>>>>>> On 8/25/25 16:47, Stephan Althaus wrote: >>>>>>> >>>>>>>> On 8/24/25 23:58, Jonathan Vasquez wrote: >>>>>>>> >>>>>>>>> Hey all, >>>>>>>>> >>>>>>>>> I've been experimenting with trying to get my server's dedicated AMD >>>>>>>>> Radeon RX 6900 XT working on a bhyve Windows 10 Pro client for gaming >>>>>>>>> purposes. I've been reading a bunch of blog posts, sites, files, and >>>>>>>>> videos (thanks Corvin) to try and piece things together. I've been >>>>>>>>> able to get the VM to recognize the card, but still stuck on the Code >>>>>>>>> 43 issue (with or without using the vBIOS I've extracted from my >>>>>>>>> dedicated card and also ensuring any options needed to be on in the >>>>>>>>> UEFI settings are enabled). This machine is running on an AMD Ryzen 9 >>>>>>>>> 7950X 16-Core Processor, so it's a full AMD setup, no Intel stuff. It >>>>>>>>> technically should be relatively "easy" to get AMD working from what >>>>>>>>> I've read (excluding the known AMD Hardware reset issue). I've >>>>>>>>> documented everything I've done so far, and will be documenting more >>>>>>>>> and simplifying the instructions as I make progress. Please let me >>>>>>>>> know if you see any thing weird or anything I could try based on what >>>>>>>>> I've rewritten. I'll try to give it a go and provide feedback. Thank >>>>>>>>> you! >>>>>>>>> >>>>>>>>> https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd >>>>>>>>> >>>>>>>>> Jonathan Vasquez >>>>>>>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >>>>>>>>> Sent with ProtonMail Secure Email >>>>>>>> >>>>>>>> Hi! >>>>>>>> >>>>>>>> i have a AMD RX6400 GPU setup running on 14.2 with win10 guest, host >>>>>>>> is a intel system. >>>>>>>> >>>>>>>> i think you have to passthrough the 3/0/1 device as well! >>>>>>>> >>>>>>>> The bios.rom is extracted from this card in place ? >>>>>>>> >>>>>>>> Regards, >>>>>>>> Stephan >>>>>>> >>>>>>> Ah, and be shure to reboot the whole setup, inclusive host, for every >>>>>>> new test-case, >>>>>>> at least as long at is not guaranteed that we can live without.. >>>>>>> >>>>>>> Regars, >>>>>>> Stephan >>>>> >>>>> Hi! >>>>> >>>>> My only next assumption would be that this only does not work on AMD host. >>>>> >>>>> You don't have an intel system to test, you would havedone it, if so.. (?) >>>>> >>>>> Stephan
