nvidia GTX960 in PCIe slot
intel i7-4790K CPU
FreeBSD 11-RC2 host
CentOS 7 guest with kernel 3.10.0-327.28.3.el7.x86_64
Using vm-bhyve port

I've hit two issues:

1. BAR allocation

Workaround (for me) is adding "pci=nocrs" to linux guest's kernel command line. Without "pci=nocrs" (or with "pci=use_crs"), the GTX960 doesn't get its 256MB block allocated.

Bhyve's ACPI table produces this error linux-side regardless of "pci=" setting:

[ 0.163085] acpi PNP0A03:00: host bridge window [0xd000000000-0xd0100fffff] (ignored, not CPU addressable)

which then leads to this:

[ 0.215369] pci 0000:00:06.0: can't claim BAR 1 [mem 0xd000000000-0xd00fffffff 64bit pref]: no compatible bridge window

and then, with "pci=use_crs" (i.e. use ACPI host bridge windows):

[    0.164030] pci_bus 0000:00: root bus resource [bus 00]
[    0.164379] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7]
[    0.164799] pci_bus 0000:00: root bus resource [io 0x0d00-0x1fff]
[    0.165206] pci_bus 0000:00: root bus resource [io 0x2000-0x211f]
[ 0.165623] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xc41fffff]
[ 0.231762] pci 0000:00:06.0: BAR 1: no space for [mem size 0x10000000 64bit pref] [ 0.232263] pci 0000:00:06.0: BAR 1: trying firmware assignment [mem size 0x10000000 64bit pref] [ 0.232855] pci 0000:00:06.0: BAR 1: [mem size 0x10000000 64bit pref] conflicts with PCI mem [mem 0x00000000-0x7fffffffff] [ 0.233579] pci 0000:00:06.0: BAR 1: failed to assign [mem size 0x10000000 64bit pref]

but with "pci=nocrs" (i.e. ignore ACPI host bridge windows):

[    0.163967] pci_bus 0000:00: root bus resource [bus 00]
[    0.164323] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 0.164745] pci_bus 0000:00: root bus resource [mem 0x00000000-0x7fffffffff]
[ 0.230203] pci 0000:00:06.0: BAR 1: assigned [mem 0x140000000-0x14fffffff 64bit pref]

2. "can't derive routing for PCI INT"

Linux-side dmesg related output:

[    1.677168] nvidia 0000:00:06.0: can't derive routing for PCI INT A
[    1.677600] nvidia 0000:00:06.0: PCI INT A: no GSI

Host-side info (when GTX960 is NOT configured as a pass-thru dev):

vgapci0@pci0:1:0:0: class=0x030000 card=0x19623842 chip=0x140110de rev=0xa1 hdr=0x00
    vendor     = 'NVIDIA Corporation'
    device     = 'GM206 [GeForce GTX 960]'
    class      = display
    subclass   = VGA
bar [10] = type Memory, range 32, base 0xf6000000, size 16777216, enabled bar [14] = type Prefetchable Memory, range 64, base 0xe0000000, size 268435456, enabled bar [1c] = type Prefetchable Memory, range 64, base 0xf0000000, size 33554432, enabled
    bar   [24] = type I/O Port, range 32, base 0xe000, size 128, enabled
    cap 01[60] = powerspec 3  supports D0 D3  current D0
    cap 05[68] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 10[78] = PCI-Express 2 legacy endpoint max data 128(256) RO
                 link x4(x16) speed 2.5(8.0)
    ecap 0002[100] = VC 1 max VC0
    ecap 001e[258] = unknown 1
    ecap 0004[128] = Power Budgeting 1
    ecap 0001[420] = AER 2 0 fatal 0 non-fatal 4 corrected
    ecap 000b[600] = Vendor 1 ID 1
    ecap 0019[900] = PCIe Sec 1 lane errors 0xf

Linux-side info (when GTX960 IS configured as a pass-thru dev, also with "pci=nocrs"):

00:06.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: eVga.com. Corp. Device 1962
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at c1000000 (32-bit, non-prefetchable) [size=16M]
        Region 1: Memory at 140000000 (64-bit, prefetchable) [size=256M]
        Region 3: Memory at c2000000 (64-bit, prefetchable) [size=32M]
        Region 5: I/O ports at 2080 [size=128]
        Expansion ROM at f7000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM not supported, Exit Latency L0s <1us, L1 <4us
                        ClockPM+ Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR+, OBFF Via message DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
        Kernel driver in use: nvidia

Any ideas?

freebsd-virtualization@freebsd.org mailing list
To unsubscribe, send any mail to 

Reply via email to