Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-15 Thread soralx
> Some screens attached (hopefully not too heavy). > Didn't have time to do better. Select your favourite ones. > > I upgraded Linux to newer version (Ubuntu 16.10, kernel 4.8), > and it broke the driver. OpenCL does not work at all anymore. > The screens were made on newer system -- nothing

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-15 Thread soralx
> When `glxgears` is run in fullscreen mode, what's on screen depends > on each run. Framerate varies from run to run, is mostly stable within > one run but can change abruptly by 100's of FPS. When the screen is > blank, the framerate is slowest (750~1500 FPS). When only parts of > gears are

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-13 Thread Peter Grehan
Hi, Does bhyve not execute peripheral cards' option ROMs? Not yet. I guess it doesn't. This could explain a lot of strange behaviour seen resulting from running in a VM. Yes. How does UEFI work in this regard? My guess is that cards have to explicitly support the new boot method

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-13 Thread Peter Grehan
Hi, That is extremely likely. bhyve itself doesn't have a BIOS, though bhyve/UEFI could be modified to handle options ROMs (see http://awilliam.github.io/presentations/KVM-Forum-2014/#/) Hm, interesting. I wonder if a card that's not designed for use with UEFI is destined not to work

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-13 Thread Rodney W. Grimes
... > > > -Performance State : P0 > > > +Performance State : P8 > > > > Note sure what's happening here. > > Driver not kicking the card's BIOS into the right mode > to switch to dynamic power state selection? I suspect this is a failure to run the BIOS

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-13 Thread soralx
> -- VDPAU works, but I suspect it's not using the GPU [3][4]; > I haven't figure a way how to force the use of GPU. Also, > the main window with text looks OK most of the time (when > doing the video test and in the end, in particular), but > show a smaller black rectangle

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-13 Thread soralx
> > First, `nvidia-smi -q` output diff [0] is interesting. It suggests > > that the card may be in some incompletely initialized state: notice > > the "Unknown Error" instead of real UUID, and the P8 power state. > > Could it be that the driver doesn't put the card's BIOS in the right > > state?

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-12 Thread Peter Grehan
Hi, BTW, is it [generally] safe to decrease the BAR base address further? > My workstation has a CPU with just 36 address bits... Yes. The only potential conflict is with the top of guest RAM, and 36 bits is a lot of RAM :) later, Peter. ___

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-11 Thread soralx
> This gives me the idea to try a different driver version in Linux... Tried the same driver version in Linux as in FreeBSD. The driver seems to talk to the card now, but not sure whether I can call this progress: [0.536988] PCI host bridge to bus :00 [0.537291] pci_bus :00:

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-11 Thread soralx
> That's a different issue - it's unlikely, if not impossible, to > configure bhyve with enough RAM to hit 37 bits worth where that would > become a problem. No need to worry about that. Well, there may be peripheral cards that have less bits... Anyway, I see what you mean: memory manager

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-11 Thread soralx
> > Removing another signature of detecting virtualization and increasing > > compatibility would be negligible gain? Just asking... > I don't think we are going to try and defeat the NVidia virtualization > checks, and I can probably assure you that they would patch them as > fast as we

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-11 Thread soralx
> I hope you keep it up or at least figure out what the driver is doing. Not in my plans at the moment. I prefer AMD GPUs over nV for OpenCL. nVidia did (and does) serve me well for the last 10 year with their excellent FreeBSD graphics driver: I had very few problems with it; it's stable and

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-11 Thread K. Macy
I hope you keep it up or at least figure out what the driver is doing. If they haven't explicitly put in the license terms that virtualization is forbidden for consumer cards, there's nothing wrong with hot patching the driver ... assuming that they don't do things like Skype does where it

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-11 Thread soralx
I had a bit more play with nVidia and FreeBSD guest. First, `nvidia-smi -q` output diff [0] is interesting. It suggests that the card may be in some incompletely initialized state: notice the "Unknown Error" instead of real UUID, and the P8 power state. Could it be that the driver doesn't

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-11 Thread Rodney W. Grimes
> > > Hi, > > > > >> The problem appears to be in the area of assigning memory-mapped > > >> I/O ranges by bhyve for the VGA card to a region outside of the > > >> CPU's addressable space; i.e., bhyve does not check CPUID's > > >> 0x8008 AL value (0x27 for my CPU, which is 39 bits -- while >

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-11 Thread soralx
> Hi, > > >> The problem appears to be in the area of assigning memory-mapped > >> I/O ranges by bhyve for the VGA card to a region outside of the > >> CPU's addressable space; i.e., bhyve does not check CPUID's > >> 0x8008 AL value (0x27 for my CPU, which is 39 bits -- while > >> bhyve

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-11 Thread soralx
> IIRC the 367.44 version of the nvidia drivers do NOT support the > Quadro 2000, you need to be using the 340.xx version of them. I > ran into problems on native hardware. I pulled the Quadro 2000 out of my workstation [and put the 600 in], which is running fine with the latest driver from

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-11 Thread soralx
> As far as I can tell it's the Hypervisor extension flags list. The lack > of these extensions/optimisations might explain why your FreeBSD VM > runs slow The guest isn't slow, actually -- just the `nvidia-smi` tool was much slower than normal to produce output. CPU speed in the guest is less

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-11 Thread Peter Grehan
Hi, There doesn't seem to be support for CPUID 0x4001 in bhyve either. What is it supposed to do? As far as I can tell it's the Hypervisor extension flags list. The lack of these extensions/optimisations might explain why your FreeBSD VM runs slow but their presence also causes the

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-11 Thread K. Macy
Is the VM checking documented in the driver notes somewhere? I have a Titan X that I need to run CUDA on and would be much happier if I didn't have to actually switch back and forth between FreeBSD and Ubuntu on my desktop. Are we new fairly certain that this won't work? (Yet another reason to go

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-11 Thread Dom
On 11/01/2017 02:01, sor...@cydem.org wrote: Dom wote: There doesn't seem to be support for CPUID 0x4001 in bhyve either. What is it supposed to do? As far as I can tell it's the Hypervisor extension flags list. The lack of these extensions/optimisations might explain why your FreeBSD

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-11 Thread Rodney W. Grimes
IIRC the 367.44 version of the nvidia drivers do NOT support the Quadro 2000, you need to be using the 340.xx version of them. I ran into problems on native hardware. Also before you attempt to get VGA passthrough working it is best to make sure you can run native, have you tried running your

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-11 Thread Peter Grehan
Hi, The problem appears to be in the area of assigning memory-mapped I/O ranges by bhyve for the VGA card to a region outside of the CPU's addressable space; i.e., bhyve does not check CPUID's 0x8008 AL value (0x27 for my CPU, which is 39 bits -- while bhyve assigns 0xd0 & above for

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-10 Thread soralx
> Found my original attempt by modifying /usr/src/sys/amd64/vmm/x86.c > Unified diff follows, but this didn't work for me. > ("bhyve_id[]" commented out to prevent compiler complaints) Who knows what sort of trickery nVidia's driver is up to besides CPUID when determining the presence of

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-10 Thread Dom
Found my original attempt by modifying /usr/src/sys/amd64/vmm/x86.c Unified diff follows, but this didn't work for me. ("bhyve_id[]" commented out to prevent compiler complaints) There doesn't seem to be support for CPUID 0x4001 in bhyve either. --- x86.c.orig 2016-09-11 14:40:22.410462000

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-10 Thread Dom
With QEMU, they have the "kvm=off" option which hides hypervisor info from the guest. See: https://www.redhat.com/archives/libvir-list/2014-August/msg00512.html I did try to replicate this a while back but didn't have much success - maybe I missed a flag? The QEMU diff seems relatively small,

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2017-01-10 Thread soralx
Howdy, virtualization zealots! This is in reply to maillist thread [0]. It so happens that I have to get GPU-accelerated OpenCL working on my machine, so I had a play with bhyve & PCI-e passthrough for VGA. I was using nVidia Quadro 600 (GF108) for testing (planning to use AMD/ATI for

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2016-09-09 Thread Dom
It looks like there may not be an issue with MSI after all. The nvidia driver is issued an IRQ when first used, not at boot time. If I run the CUDA "deviceQuery" sample then this appears in dmesg: [ 67.207929] nvidia :00:06.0: irq 29 for MSI/MSI-X [ 67.646207] NVRM: RmInitAdapter

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2016-09-07 Thread Peter Grehan
Hi Dom, Bhyve's ACPI table produces this error linux-side regardless of "pci=" setting: [0.163085] acpi PNP0A03:00: host bridge window [0xd0-0xd0100f] (ignored, not CPU addressable) That one is most likely a bug in bhyve, where the space used for 64-bit BAR placement isn't

Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

2016-09-02 Thread Dom
Hello, Setup: nvidia GTX960 in PCIe slot intel i7-4790K CPU FreeBSD 11-RC2 host CentOS 7 guest with kernel 3.10.0-327.28.3.el7.x86_64 Using vm-bhyve port I've hit two issues: 1. BAR allocation Workaround (for me) is adding "pci=nocrs" to linux guest's kernel command line. Without