On 6/11/25 11:03, Stephan Althaus via oi-dev wrote:
Hello!
I've created a script to get the relevant system parameters around
bhyve, to get the corresponding config for each of my tests.
Please let me know if i miss something relevant here!
i call it check_bhyve_config.sh
echo "######### grep -v '#' /etc/ppt_aliases"
grep -v "#" /etc/ppt_aliases | awk NF
echo "######### grep -v '#' /etc/ppt_matches "
grep -v "#" /etc/ppt_matches | awk NF
echo "######### grep -v '#' /etc/mach"
grep -v "#" /etc/mach | awk NF
echo "######### grep -v '#' /platform/i86pc/kernel/drv/rootnex.conf"
grep -v "#" /platform/i86pc/kernel/drv/rootnex.conf | awk NF
echo "######### apic_mode::print | mdb -k"
echo apic_mode::print | mdb -k
echo "######### pptadm list -ao all"
pptadm list -ao all
echo "######### /opt/bhyve-tests/tests/vmm/check_iommu (Openindiana
package pkg://openindiana.org/system/bhyve/tests
<http://openindiana.org/system/bhyve/tests>)"
/opt/bhyve-tests/tests/vmm/check_iommu
echo "######### dmesg | grep -iE '(vmm|ppt|APIC|DMAR|IOMMU|mapp)'"
grep -iE '(vmm|ppt|APIC|DMAR|IOMMU|mapp)' /var/adm/messages | tail -n 100
Would you be so kind, and run above script on your machines with a
working GPU passthrough configuration, i have no clue why i get "error
43" within my win10 vm. The GPUs are visible as PCI device within
Windows, but are not usable. (with FreeBSD 14.2 host-OS this is
working well on my hardware/BIOS combination with both GPUs AMD RX6400
and MSI NVidia GT1030).
Thank you!
Stephan
-------------------------
output on my machine fpr my first test today:
# ./check_bhyve_config.sh
######### grep -v '#' /etc/ppt_aliases
ppt "pci1002,743f"
ppt "pci1002,ab28"
ppt "pci10de,1d01"
ppt "pci10de,fb8"
######### grep -v '#' /etc/ppt_matches
pciex1da2,e458
pciex1002,ab28
pciex10de,1d01
pciex10de,fb8
######### grep -v '#' /etc/mach
pcplusmp
apix
xpv_psm
######### grep -v '#' /platform/i86pc/kernel/drv/rootnex.conf
immu-enable="false";
immu-intrmap-enable="false";
######### apic_mode::print | mdb -k
3 (LOCAL_X2APIC)
######### pptadm list -ao all
DEV VENDOR DEVICE SUBVENDOR SUBDEVICE REV PATH
LABEL
/dev/ppt2 10de 1d01 1462 8c98 a1
/pci@0,0/pci8086,6f04@2/display@0 NVIDIA Corporation
GP108 [GeForce GT 1030]
/dev/ppt3 10de fb8 1462 8c98 a1
/pci@0,0/pci8086,6f04@2/pci1462,8c98@0,1 NVIDIA Corporation
GP108 High Definition Audio Controller
/dev/ppt0 1002 743f 1da2 e458 c7
/pci@ff,0/pci8086,6f08@3/pci1002,1478@0/pci1002,1479@0/display@0
Advanced Micro Devices, Inc. [AMD/ATI] Navi 24 [Radeon RX 6400/6500
XT/6500M]
/dev/ppt1 1002 ab28 1002 ab28 0
/pci@ff,0/pci8086,6f08@3/pci1002,1478@0/pci1002,1479@0/pci1002,ab28@0,1
Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller
######### /opt/bhyve-tests/tests/vmm/check_iommu (Openindiana package
pkg://openindiana.org/system/bhyve/tests)
check_iommu PASS
######### dmesg | grep -iE '(vmm|ppt|APIC|DMAR|IOMMU|mapp)'
Jun 11 10:10:20 smhost genunix: [ID 936769 kern.info] vmm0 is
/pseudo/vmm@0
Jun 11 10:19:19 smhost pseudo: [ID 129642 kern.info] pseudo-device:
vmm_drv_test0
Jun 11 10:19:19 smhost genunix: [ID 936769 kern.info] vmm_drv_test0 is
/pseudo/vmm_drv_test@0
_______________________________________________
oi-dev mailing list
oi-dev@openindiana.org
https://openindiana.org/mailman/listinfo/oi-dev
Hello!
My longterm-goal is to have a bhyve Win10 guest with GPU passthrough on
an OI host.
Passthrough works 'somehow', i see the devices in the guest, but i have
the win error code 43 within the guest.
With FreeBSD host OS this is working on the hardware/BIOS, so these seem
to be ok.
! With OI host OS, I have a working ubuntu guest with the nvidia GPU,
that's promising!
I've tried this and that, to get some knowledge and some details about
what's happening,
and i now chose an OI guest to maybe get even more details about what
could be the cause of the windows error.
With OI host OS and OI guest, i have errors on the host, and within the
guest. (details below)
### OI HOST output of the bhyve executable
bhyve: pci_passthru: map_pptdev_mmio failed
bhyve: pci_passthru: unmap_pptdev_mmio failed
### OI GUEST from dmesg
Jun 13 12:07:10 openindiana-vm nvidia: [ID 702911 kern.notice] NOTICE:
NVRM: BAR1 @ 0x14: BAR is 0 (invalid)
Jun 13 12:07:10 openindiana-vm nvidia: [ID 702911 kern.notice] NOTICE:
NVRM: nv_attach: nvidia_pci_probe failed
Maybe this has something to do with the 4GB 'barrier' ("Above 4G
Decoding") ??
# ~max Address 32 Bit (=2^32-1)
4294967295
# BAR1 Address
0x800000000 <- this is corresponding to what the OI host gives to the
ubuntu-vm "pci 0000:00:08.0: BAR 1 [mem 0x800000000-0x80fffffff 64bit pref]"
Please give me a hint what i should look at,
maybe there's a bhyve param or OI-guest or nvidia param i haven't seen??
Thanks!
Regards,
Stephan
================================================================================
################################################################################
### Openindiana-HOST
################################################################################
######### grep -v '#' /etc/ppt_aliases
ppt "pci1002,743f"
ppt "pci1002,ab28"
ppt "pci10de,1d01"
ppt "pci10de,fb8"
######### grep -v '#' /etc/ppt_matches
pciex1da2,e458
pciex1002,ab28
pciex10de,1d01
pciex10de,fb8
######### grep -v '#' /etc/mach
pcplusmp
apix
xpv_psm
######### grep -v '#' /platform/i86pc/kernel/drv/rootnex.conf
immu-enable="false";
immu-intrmap-enable="false";
immu-blacklist=
"SMBIOS","Supermicro","X10DRI-T","",
"DRIVER", "xhci",
"DRIVER", "ehci",
"DRIVER", "nvme",
;
immu-dvma-mapping="xlate";
immu-dvma-unity-drivers="igb","ahci","ixgbe","ehci","xhci","nvme","mpt";
######### apic_mode::print | mdb -k
3 (LOCAL_X2APIC)
######### pptadm list -ao all
DEV VENDOR DEVICE SUBVENDOR SUBDEVICE REV PATH
LABEL
/dev/ppt2 10de 1d01 1462 8c98 a1
/pci@0,0/pci8086,6f04@2/display@0 NVIDIA Corporation
GP108 [GeForce GT 1030]
/dev/ppt3 10de fb8 1462 8c98 a1
/pci@0,0/pci8086,6f04@2/pci1462,8c98@0,1 NVIDIA Corporation
GP108 High Definition Audio Controller
/dev/ppt0 1002 743f 1da2 e458 c7
/pci@ff,0/pci8086,6f08@3/pci1002,1478@0/pci1002,1479@0/display@0
Advanced Micro Devices, Inc. [AMD/ATI] Navi 24 [Radeon RX 6400/6500
XT/6500M]
/dev/ppt1 1002 ab28 1002 ab28 0
/pci@ff,0/pci8086,6f08@3/pci1002,1478@0/pci1002,1479@0/pci1002,ab28@0,1
Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller
######### /opt/bhyve-tests/tests/vmm/check_iommu (Openindiana package
pkg://openindiana.org/system/bhyve/tests)
check_iommu PASS
# /usr/X11/bin/scanpci -v
pci bus 0x0002 cardnum 0x00 function 0x00: vendor 0x10de device 0x1d01
NVIDIA Corporation GP108 [GeForce GT 1030]
CardVendor 0x1462 card 0x8c98 (Micro-Star International Co., Ltd.
[MSI], Card unknown)
STATUS 0x0010 COMMAND 0x0040
CLASS 0x03 0x00 0x00 REVISION 0xa1
BIST 0x00 HEADER 0x80 LATENCY 0x00 CACHE 0x10
BASE0 0xbe000000 SIZE 16777216 MEM
BASE1 0x0000383fe0000000 SIZE 268435456 MEM64 PREFETCHABLE
BASE3 0x0000383ff0000000 SIZE 33554432 MEM64 PREFETCHABLE
BASE5 0x00006000 SIZE 128 I/O
MAX_LAT 0x00 MIN_GNT 0x00 INT_PIN 0x01 INT_LINE 0x0b
pci bus 0x0002 cardnum 0x00 function 0x01: vendor 0x10de device 0x0fb8
NVIDIA Corporation GP108 High Definition Audio Controller
CardVendor 0x1462 card 0x8c98 (Micro-Star International Co., Ltd.
[MSI], Card unknown)
STATUS 0x0010 COMMAND 0x0040
CLASS 0x04 0x03 0x00 REVISION 0xa1
BIST 0x00 HEADER 0x80 LATENCY 0x00 CACHE 0x10
BASE0 0xbf080000 SIZE 16384 MEM
MAX_LAT 0x00 MIN_GNT 0x00 INT_PIN 0x02 INT_LINE 0x0a
### Openindiana-HOST
# prtconf -vD
display, instance #2 (driver name: ppt)
Driver properties:
name='fm-errcb-capable' type=boolean dev=none
name='fm-ereport-capable' type=boolean dev=none
Hardware properties:
name='pcie-aspm-state' type=string items=1
value='l1'
name='pcie-aspm-support' type=string items=1
value='l0s,l1'
name='pcie-link-supported-speeds' type=int64 items=3
value=000000009502f900.000000012a05f200.00000001dcd65000
name='pcie-link-target-speed' type=int64 items=1
value=00000001dcd65000
name='pcie-link-maximum-speed' type=int64 items=1
value=00000001dcd65000
name='pcie-link-current-speed' type=int64 items=1
value=00000001dcd65000
name='pcie-link-current-width' type=int items=1
value=00000004
name='pcie-link-maximum-width' type=int items=1
value=00000004
name='acpi-namespace' type=string items=1
value='\_SB_.PCI0.BR2A.D07B'
name='assigned-addresses' type=int items=35
value=82020010.00000000.be000000.00000000.01000000.c3020014.0000383f.e0000000.00000000.10000000.c302001c.0000383f.f0000000.00000000.02000000.8102002
4.00000000.00006000.00000000.00000080.a1020000.00000000.000003b0.00000000.0000000c.a1020000.00000000.000003c0.00000000.00000020.a2020000.00000000.000a0000.00000000.00020000
name='reg' type=int items=40
value=00020000.00000000.00000000.00000000.00000000.02020010.00000000.00000000.00000000.01000000.43020014.00000000.00000000.00000000.10000000.4302001
c.00000000.00000000.00000000.02000000.01020024.00000000.00000000.00000000.00000080.a1020000.00000000.000003b0.00000000.0000000c.a1020000.00000000.000003c0.00000000.00000020
.a2020000.00000000.000a0000.00000000.00020000
name='compatible' type=string items=13
value='pciex10de,1d01.1462.8c98.a1' +
'pciex10de,1d01.1462.8c98' + 'pciex10de,1d01.a1' + 'pciex10de,1d01' +
'pciexclass,030000' + 'pciexclass,0300'
+ 'pci10de,1d01.1462.8c98.a1' + 'pci10de,1d01.1462.8c98' +
'pci10de,1d01.a1' + 'pci10de,1d01,p' + 'pci10de,1d01' +
'pciclass,030000' + 'pciclass,0300'
...
### Openindiana-HOST
# cat testoigui.sh
/usr/sbin/bhyve -U bd60bf59-0ed6-61a1-f0ed-b17e99ada165 -A -H \
-B
1,manufacturer=OpenIndiana,product=OpenIndianaHVM,version=1.0,serial=69a18a85-c1df-c266-f620-a99a632d3c11,sku=001,family=VirtualMachine
\
-c 8 -m 16G -w \
-s 31:0,lpc -l bootrom,/usr/share/bhyve/firmware/BHYVE_DEBUG.fd \
-s 0,hostbridge,model=i440fx \
-s 4:0,nvme,/dev/zvol/rdsk/rpool1/zones/oivm \
-s 6:0,virtio-net-viona,vnic0 \
-s 30:0,fbuf,tcp=0.0.0.0:5905,w=1900,h=900 \
-s 30:1,xhci,tablet \
-S -s 8:0,passthru,/dev/ppt2,rom=/home2/vm/GP108.rom \
-s 8:1,passthru,/dev/ppt3 \
oivm
### Openindiana-HOST
# ./testoigui.sh
fbuf frame buffer base: 7fffa7e00000 [sz 33554432]
rfb0: connection from 192.168.2.63
bhyve: pci_passthru: map_pptdev_mmio failed
bhyve: pci_passthru: unmap_pptdev_mmio failed
bhyve: pci_passthru: map_pptdev_mmio failed
bhyve: pci_passthru: unmap_pptdev_mmio failed
bhyve: pci_passthru: map_pptdev_mmio failed
bhyve: pci_passthru: unmap_pptdev_mmio failed
bhyve: pci_passthru: map_pptdev_mmio failed
bhyve: pci_passthru: unmap_pptdev_mmio failed
bhyve: pci_passthru: map_pptdev_mmio failed
bhyve: pci_passthru: unmap_pptdev_mmio failed
bhyve: pci_passthru: map_pptdev_mmio failed
bhyve: pci_passthru: unmap_pptdev_mmio failed
bhyve: pci_passthru: map_pptdev_mmio failed
bhyve: pci_passthru: unmap_pptdev_mmio failed
bhyve: pci_passthru: map_pptdev_mmio failed
bhyve: pci_passthru: unmap_pptdev_mmio failed
################################################################################
### openindiana-vm
################################################################################
# /usr/X11/bin/scanpci -v
pci bus 0x0000 cardnum 0x08 function 0x00: vendor 0x10de device 0x1d01
NVIDIA Corporation GP108 [GeForce GT 1030]
CardVendor 0x1462 card 0x8c98 (Micro-Star International Co., Ltd.
[MSI], Card unknown)
STATUS 0x0010 COMMAND 0x0047
CLASS 0x03 0x00 0x00 REVISION 0xa1
BIST 0x00 HEADER 0x80 LATENCY 0x00 CACHE 0x10
BASE0 0xc4000000 SIZE 16777216 MEM
BASE1 0x0000000800000000 SIZE 268435456 MEM64 PREFETCHABLE
BASE3 0x00000000c0000000 SIZE 33554432 MEM64 PREFETCHABLE
BASE5 0x00002000 SIZE 128 I/O
MAX_LAT 0x00 MIN_GNT 0x00 INT_PIN 0x01 INT_LINE 0x0b
pci bus 0x0000 cardnum 0x08 function 0x01: vendor 0x10de device 0x0fb8
NVIDIA Corporation GP108 High Definition Audio Controller
CardVendor 0x1462 card 0x8c98 (Micro-Star International Co., Ltd.
[MSI], Card unknown)
STATUS 0x0010 COMMAND 0x0047
CLASS 0x04 0x03 0x00 REVISION 0xa1
BIST 0x00 HEADER 0x80 LATENCY 0x00 CACHE 0x10
BASE0 0xc5004000 SIZE 16384 MEM
MAX_LAT 0x00 MIN_GNT 0x00 INT_PIN 0x02 INT_LINE 0x0a
### openindiana-vm
# lspci -vv
00:08.0 Class 0300: Device 10de:1d01 (rev a1)
Subsystem: Device 1462:8c98
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 11
Region 0: Memory at c4000000 (32-bit, non-prefetchable)
Region 1: Memory at <unassigned> (64-bit, prefetchable)
Region 2: Memory at <unassigned> (32-bit, prefetchable)
Region 3: Memory at c0000000 (64-bit, prefetchable)
Region 5: I/O ports at 2000
Expansion ROM at bf000000 [disabled]
### openindiana-vm
# prtconf -vD
display, instance #1 (driver name: nvidia)
Hardware properties:
name='pcie-aspm-state' type=string items=1
value='l1'
name='pcie-aspm-support' type=string items=1
value='l0s,l1'
name='pcie-link-supported-speeds' type=int64 items=3
value=000000009502f900.000000012a05f200.00000001dcd65000
name='pcie-link-target-speed' type=int64 items=1
value=00000001dcd65000
name='pcie-link-maximum-speed' type=int64 items=1
value=00000001dcd65000
name='pcie-link-current-speed' type=int64 items=1
value=00000001dcd65000
name='pcie-link-current-width' type=int items=1
value=00000004
name='pcie-link-maximum-width' type=int items=1
value=00000004
name='assigned-addresses' type=int items=35
value=82004010.00000000.c4000000.00000000.01000000.c3004014.00000008.00000000.00000000.10000000.c300401c.00000000.c0000000.00000000.02000000.81004024.00000000.00002000.00000000.00000080.a1004000.00000000.000003b0.00000000.0000000c.a1004000.00000000.000003c0.00000000.00000020.a2004000.00000000.000a0000.00000000.00020000
name='reg' type=int items=40
value=00004000.00000000.00000000.00000000.00000000.02004010.00000000.00000000.00000000.01000000.43004014.00000000.00000000.00000000.10000000.4300401c.00000000.00000000.00000000.02000000.01004024.00000000.00000000.00000000.00000080.a1004000.00000000.000003b0.00000000.0000000c.a1004000.00000000.000003c0.00000000.00000020.a2004000.00000000.000a0000.00000000.00020000
name='compatible' type=string items=13
value='pciex10de,1d01.1462.8c98.a1' +
'pciex10de,1d01.1462.8c98' + 'pciex10de,1d01.a1' + 'pciex10de,1d01' +
'pciexclass,030000' + 'pciexclass,0300' + 'pci10de,1d01.1462.8c98.a1' +
'pci10de,1d01.1462.8c98' + 'pci10de,1d01.a1' + 'pci10de,1d01,p' +
'pci10de,1d01' + 'pciclass,030000' + 'pciclass,0300'
### openindiana-vm
# dmesg|grep NOTICE
Jun 13 12:07:10 openindiana-vm iommulib: [ID 321598 kern.info] NOTICE:
iommulib_nexus_register: rootnex-1: Succesfully registered NEXUS i86pc
nexops=fffffffffbd39040
Jun 13 12:07:10 openindiana-vm apix: [ID 615120 kern.info] NOTICE: apic:
local nmi: 255 0x5 1
Jun 13 12:07:10 openindiana-vm apix: [ID 468339 kern.info] NOTICE: apic:
Using ACPI (MADT) for SMP configuration
Jun 13 12:07:10 openindiana-vm apix: [ID 179576 kern.info] NOTICE: apic:
Using APIC interrupt routing mode
Jun 13 12:07:10 openindiana-vm blkdev: [ID 643073 kern.info] NOTICE:
blkdev0: dynamic LUN expansion
Jun 13 12:07:10 openindiana-vm unix: [ID 494986 kern.info] NOTICE: CPU
power management will not function.
Jun 13 12:07:10 openindiana-vm nvidia: [ID 702911 kern.notice] NOTICE:
NVRM: BAR1 @ 0x14: BAR is 0 (invalid)
Jun 13 12:07:10 openindiana-vm nvidia: [ID 702911 kern.notice] NOTICE:
NVRM: nv_attach: nvidia_pci_probe failed
Jun 13 12:07:10 openindiana-vm nvidia: [ID 702911 kern.notice] NOTICE:
NVRM: BAR1 @ 0x14: BAR is 0 (invalid)
Jun 13 12:07:10 openindiana-vm nvidia: [ID 702911 kern.notice] NOTICE:
NVRM: nv_attach: nvidia_pci_probe failed
Jun 13 12:07:11 openindiana-vm unix: [ID 494986 kern.info] NOTICE: CPU
power management will not function.
Jun 13 12:07:11 openindiana-vm unix: [ID 494986 kern.info] NOTICE: CPU
power management will not function.
Jun 13 12:07:11 openindiana-vm unix: [ID 494986 kern.info] NOTICE: CPU
power management will not function.
Jun 13 12:07:11 openindiana-vm unix: [ID 494986 kern.info] NOTICE: CPU
power management will not function.
Jun 13 12:07:11 openindiana-vm unix: [ID 494986 kern.info] NOTICE: CPU
power management will not function.
Jun 13 12:07:11 openindiana-vm unix: [ID 494986 kern.info] NOTICE: CPU
power management will not function.
Jun 13 12:07:11 openindiana-vm unix: [ID 494986 kern.info] NOTICE: CPU
power management will not function.
Jun 13 12:07:11 openindiana-vm unix: [ID 434834 kern.info] NOTICE:
hma_vmx_init: CPU does not support VMX
Jun 13 12:07:14 openindiana-vm mac: [ID 469746 kern.info] NOTICE: vioif0
registered
Jun 13 12:07:14 openindiana-vm mac: [ID 435574 kern.info] NOTICE: vioif0
link up, 1000 Mbps, full duplex
Jun 13 12:07:16 openindiana-vm nvidia: [ID 702911 kern.notice] NOTICE:
NVRM: BAR1 @ 0x14: BAR is 0 (invalid)
Jun 13 12:07:16 openindiana-vm nvidia: [ID 702911 kern.notice] NOTICE:
NVRM: nv_attach: nvidia_pci_probe failed
Jun 13 12:07:30 openindiana-vm nvidia: [ID 702911 kern.notice] NOTICE:
NVRM: BAR1 @ 0x14: BAR is 0 (invalid)
Jun 13 12:07:30 openindiana-vm nvidia: [ID 702911 kern.notice] NOTICE:
NVRM: nv_attach: nvidia_pci_probe failed
_______________________________________________
oi-dev mailing list
oi-dev@openindiana.org
https://openindiana.org/mailman/listinfo/oi-dev
