Hi Mario, tried options similar to what you suggested, no difference to the outcome. Stuck where it was before.
Best regards SS On Mon, 24 Mar 2025 at 15:29, Shamim Shahriar <[email protected]> wrote: > Hi Mario > > I will try the options you have used later and let you know how it goes. > Not sure if this is intentional or typo, but I don't recall any bhyve-win > binary in my system. > > Best regards > SS > > On Mon, 24 Mar 2025 at 14:59, Mario Marietto <[email protected]> > wrote: > >> Oh sorry. I forgot to add the following bhyve parameters necessary to >> passthru my GPU :D >> >> -s 8:0,passthru,2/0/0 \ >> -s 8:1,passthru,2/0/1 \ >> -s 8:2,passthru,2/0/2 \ >> -s 8:3,passthru,2/0/3 \ >> >> On Mon, Mar 24, 2025 at 3:57 PM Mario Marietto <[email protected]> >> wrote: >> >>> Usually I use this kind of script to launch a bhyve vm : >>> >>> #!/bin/sh >>> >>> setxkbmap it >>> kldload vmm.ko >>> vms="$(ls /dev/vmm/*)" >>> vncs="$(ps ax | awk '/vncviewer [0]/{print $6}')" >>> kldload vmm.ko >>> >>> if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then >>> echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..." >>> kldload nvidia-modeset >>> devctl clear driver -f pci0:2:0:0 >>> devctl set driver -f pci0:2:0:0 ppt >>> else >>> echo "rtx 2080ti slot 2/0/0 is already attached to ppt" >>> fi >>> >>> if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then >>> echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..." >>> devctl clear driver -f pci0:2:0:1 >>> devctl set driver -f pci0:2:0:1 ppt >>> else >>> echo "rtx 2080ti slot 2/0/1 is already attached to ppt" >>> fi >>> >>> if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then >>> echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..." >>> devctl clear driver -f pci0:2:0:2 >>> devctl set driver -f pci0:2:0:2 ppt >>> else >>> echo "rtx 2080ti slot 2/0/2 is already attached to ppt" >>> fi >>> >>> if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then >>> echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..." >>> devctl clear driver -f pci0:2:0:3 >>> devctl set driver -f pci0:2:0:3 ppt >>> else >>> echo "rtx 2080ti slot 2/0/3 is already attached to ppt" >>> fi >>> >>> echo "rtx 2080ti is fully attached to ppt" >>> >>> for vm in $vms; do >>> session="${vm##*/}" >>> echo "bhyve session = $session" >>> echo "vnc session = $vncs" >>> if ! printf '%s\n' "${vncs}" | grep "${session#vm}"; >>> then >>> printf 'VNC session not found,destroying ghost vms\n' >>> bhyvectl --vm=$session --destroy >>> else >>> printf 'Found VNC session %s\n' >>> "${session},no ghost vms found,not destroying them" >>> fi >>> done >>> >>> vmdisk0=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: >>> (AM13N4CZ)/ && d{print d}'` >>> echo "Seagate M3 Portable 1.8T ; $vmdisk0" >>> >>> vmdisk1=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: >>> (2015040204055E)/ && d{print d}'` >>> echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1" >>> >>> vmdisk2=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: >>> (2027285F1175)/ && d{print d}'` >>> echo "CT1000P1SSD8 ; $vmdisk2" >>> >>> vmdisk3=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: >>> (20130508005976F)/ && d{print d}'` >>> echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3" >>> >>> vmdisk4=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: >>> (BE0191510218)/ && d{print d}'` >>> echo "G-DRIVE USB ; $vmdisk4" >>> >>> vmdisk5=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: >>> (38234B4237354B45)/ && d{print d}'` >>> echo "Elements 25A3 ; $vmdisk5" >>> >>> vmdisk6=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: >>> (WD-WCAV2X797309)/ && d{print d}'` >>> echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6" >>> >>> vmdisk7=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: >>> (20140108006C)/ && d{print d}'` >>> echo "Corsair Force 3 SSD ; $vmdisk7" >>> >>> vmdisk8=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: >>> (1924E50B2AE5)/ && d{print d}'` >>> echo "CT500MX500SSD4 ; $vmdisk8" >>> >>> vmdisk9=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: >>> (0774921DDC4200A6)/ && d{print d}'` >>> echo "SanDisk Cruzer-15GB ; $vmdisk9" >>> >>> vmdisk10=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: >>> (60A44D4138D8F311190A0149)/ && d{print d}'` >>> echo "Kingston DataTraveler 2.0 ; $vmdisk10" >>> >>> vmdisk11=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: >>> (575845483038524844323238)/ && d{print d}'` >>> echo "WD 2500BMV External ; $vmdisk11" >>> >>> vmdisk12=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: >>> (57442E575845323039544433303334)/ && d{print d}'` >>> echo "WD 3200BMV External ; $vmdisk12" >>> >>> vmdisk13=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: >>> (2414E989076B)/ && d{print d}'` >>> echo "CT500BX500SSD1 ; $vmdisk13" >>> >>> /usr/sbin/./bhyve-win -S -c sockets=4,cores=1,threads=1 -m 4G -w -H -A \ >>> -s 0,hostbridge \ >>> -s 1,ahci-hd,/dev/$vmdisk13 \ >>> -s 10,virtio-net,tap6 \ >>> -s 11,virtio-9p,sharename=/ \ >>> -s 29,fbuf,tcp=0.0.0.0:5906,w=1600,h=950 \ >>> -s 31,lpc \ >>> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ >>> vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 & >>> >>> As you can see,each vm has a unique ID (connected with the vncviewer >>> instance) and the vncviewer window is launched as soon as the bhyve vm is >>> called...and my Geforce RTX 2080 ti is passed through inside the vm. >>> >>> >>> On Mon, Mar 24, 2025 at 3:47 PM Mario Marietto <[email protected]> >>> wrote: >>> >>>> First of all,you could use a more flexible technique than using the >>>> pptdevs on /boot/loader.conf. I would use a script like this : >>>> >>>> 1) >>>> >>>> if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then >>>> echo "18/0/0 is not attached to ppt,attaching..." >>>> devctl clear driver -f pci0:18:0:0 >>>> devctl set driver -f pci0:18:0:0 ppt >>>> else >>>> echo "18/0/0 already attached to ppt" >>>> fi >>>> >>>> 2) I would not like to use vm-bhyve. It adds only some confusion... >>>> >>>> On Mon, Mar 24, 2025 at 3:22 PM Shamim Shahriar < >>>> [email protected]> wrote: >>>> >>>>> Good afternoon everyone. >>>>> >>>>> I am trying to have VMs with GPU passthrough. The setup is a Dell >>>>> server with NVidia Quadro RTX 6000/8000 installed already. I have checked >>>>> the device IDs and put pptdevs in place >>>>> >>>>> # cat /boot/loader.conf >>>>> pptdevs="18/0/0 19/0/0" >>>>> pptdevs2="72/0/0 73/0/0" >>>>> >>>>> this is showing the GPUs as pptdev on the pciconf >>>>> >>>>> ppt0@pci0:18:0:0: class=0x030200 rev=0xa1 hdr=0x00 >>>>> vendor=0x10de device=0x1e78 subvendor=0x10de subdevice=0x13d8 >>>>> vendor = 'NVIDIA Corporation' >>>>> device = 'TU102GL [Quadro RTX 6000/8000]' >>>>> class = display >>>>> subclass = 3D >>>>> ppt1@pci0:19:0:0: class=0x030200 rev=0xa1 hdr=0x00 >>>>> vendor=0x10de device=0x1e78 subvendor=0x10de subdevice=0x13d8 >>>>> vendor = 'NVIDIA Corporation' >>>>> device = 'TU102GL [Quadro RTX 6000/8000]' >>>>> class = display >>>>> subclass = 3D >>>>> >>>>> as I am using vm-bhyve, I have put the configuration as below >>>>> >>>>> # cat /mnt/VMs/jagadish/jagadish.conf >>>>> loader="uefi" >>>>> cpu=16 >>>>> memory=128G >>>>> xhci_mouse="yes" >>>>> debug="true" >>>>> >>>>> graphics="yes" >>>>> graphics_listen="127.0.0.1" >>>>> graphics_port="5920" >>>>> graphics_res="1024x768" >>>>> graphics_wait="no" >>>>> #graphics_vga="io" >>>>> >>>>> network0_type="virtio-net" >>>>> network0_switch="swUNI" >>>>> network0_mac="58:9c:fc:06:3f:af" >>>>> >>>>> disk0_type="nvme" >>>>> disk0_name="jagadish-disk0.img" >>>>> #disk0_size="128G" >>>>> >>>>> uuid="966e909b-1293-11ef-a9a4-e4434bfe34de" >>>>> >>>>> passthru0="19/0/0=6:0" >>>>> >>>>> bhyve_options="-A -H -P" >>>>> #END >>>>> >>>>> however, when I start the vm (for OS installation to start with), it >>>>> shows it is running but I am unable to access the VNC for a while, then >>>>> when I finally manage to connect via VNC, there is nothing on the screen, >>>>> just blank dark screen >>>>> >>>>> the vm-bhyve.log shows >>>>> >>>>> Mar 24 13:59:26: initialising >>>>> Mar 24 13:59:26: [loader: uefi] >>>>> Mar 24 13:59:26: [cpu: 16] >>>>> Mar 24 13:59:26: [memory: 128G] >>>>> Mar 24 13:59:26: [hostbridge: standard] >>>>> Mar 24 13:59:26: [com ports: com1] >>>>> Mar 24 13:59:26: [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de] >>>>> Mar 24 13:59:26: [debug mode: true] >>>>> Mar 24 13:59:26: [primary disk: jagadish-disk0.img] >>>>> Mar 24 13:59:26: [primary disk dev: file] >>>>> Mar 24 13:59:26: initialising network device tap0 >>>>> Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm) >>>>> Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm) >>>>> Mar 24 13:59:26: booting >>>>> Mar 24 13:59:26: [bhyve options: -c 16 -m 128G -AHPw -l >>>>> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U >>>>> 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S] >>>>> Mar 24 13:59:26: [bhyve devices: -s 0,hostbridge -s 31,lpc -s >>>>> 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s >>>>> 5:0,virtio-net,tap0,mac=58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s >>>>> 7:0,fbuf,tcp=127.0.0.1:5920,w=1024,h=768 -s 8:0,xhci,tablet] >>>>> Mar 24 13:59:26: [bhyve console: -l com1,/dev/nmdm-jagadish.1A] >>>>> Mar 24 13:59:26: [bhyve iso device: -s >>>>> 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro] >>>>> Mar 24 13:59:26: starting bhyve (run 1) >>>>> >>>>> based on what I can see, and a little out of desparation, I decided to >>>>> run the installer in the foreground, and below is what I got >>>>> >>>>> # vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso >>>>> Starting jagadish >>>>> * found guest in /mnt/VMs/jagadish >>>>> * booting... >>>>> fbuf frame buffer base: 0x112245400000 [sz 16777216] >>>>> >>>>> it stays there for as long as I wait until I poweroff the vm. >>>>> >>>>> tried with Debian installer, >>>>> # vm install -f jagadish debian-12.5.0-amd64-netinst.iso >>>>> Starting jagadish >>>>> * found guest in /mnt/VMs/jagadish >>>>> * booting... >>>>> fbuf frame buffer base: 0x2747e2400000 [sz 16777216] >>>>> >>>>> >>>>> ideally I need to install Debian for my user base on these GPU based >>>>> systems. But even that is proving to be impossible since the installer >>>>> does >>>>> not move any further from where it is. >>>>> >>>>> Any thoughts/ideas/suggestions what I else I can try to make this >>>>> work? Anything that you have tried that worked? have I missed something? >>>>> >>>>> Would appreciate any and all thoughts/suggestions >>>>> >>>>> best regards >>>>> SS >>>>> >>>> >>>> >>>> -- >>>> Mario. >>>> >>> >>> >>> -- >>> Mario. >>> >> >> >> -- >> Mario. >> >
