First, `nvidia-smi -q` output diff [0] is interesting. It suggests that
 the card may be in some incompletely initialized state: notice the
 "Unknown Error" instead of real UUID, and the P8 power state. Could it
 be that the driver doesn't put the card's BIOS in the right state?

That is extremely likely. bhyve itself doesn't have a BIOS, though bhyve/UEFI could be modified to handle options ROMs (see http://awilliam.github.io/presentations/KVM-Forum-2014/#/)

 The command was run in both host and guest without Xorg loaded.

 Thanks for the diff; this is very useful.

-    GPU UUID                        : GPU-f6c71b8e-f6c8-5a42-260d-1164720bf4f2
+    GPU UUID                        : Unknown Error

That implies some type of h/w access isn't working, either MMIO registers or response from a DMA command.

-    Board ID                        : 0x100
+    Board ID                        : 0x4

 The same ?

             PCIe Generation
                 Max                 : 2
-                Current             : 2
+                Current             : 1

bhyve's emulated PCI hostbridge only advertises gen-1 - that could be easily changed to gen2. That could make a difference for some of the clock issues below

 (source is pci_emul.c:pci_emul_add_pciecap())

             Link Width
                 Max                 : 16x
                 Current             : 16x

That's a bit unexpected since the hostbridge only advertises 1x, but the driver is probably exporting the host value here.

-    Performance State               : P0
+    Performance State               : P8

 Note sure what's happening here.

-        Graphics                    : 625 MHz
-        SM                          : 1251 MHz
-        Memory                      : 1304 MHz
-        Video                       : 540 MHz
+        Graphics                    : 405 MHz
+        SM                          : 810 MHz
+        Memory                      : 324 MHz
+        Video                       : 405 MHz

 This may be related to the gen1 vs gen2 issue above.

When rebooting, I get this:
nvidia-modeset: ERROR: GPU:0: Idling display engine timed out: 

 This may be DMA not working.

A general issue with PCI passthrough is that often MMIO from the guest works, since that is just VT-x remapping, but DMA doesn't work due to issues with IOMMU programming (or incorrect mappings being used). This gives a device that partially works in that registers can be read, but data transfer doesn't work.

Jan 11 11:34:49 fbsd12tst kernel: nvidia-modeset: ERROR: GPU:0: Display engine 
push buffer channel allocation failed
Jan 11 11:34:49 fbsd12tst kernel: nvidia-modeset: ERROR: GPU:0: Failed to 
allocate display engine core DMA push buffer

 Not sure what's happening with those.

Would it be possible to try the nouveau driver ? At least the source is available, so it may be easier to determine what is broken.


freebsd-virtualization@freebsd.org mailing list
To unsubscribe, send any mail to 

Reply via email to