Re: bhyve PCI pass-through to Linux guest
Excerpts from Peter Grehan's message from Tue 22-Dec-15 13:09: > > I pass trough a PCI device (USB controller) to a Linux guest. It works > > properly. Then I halt the VM, make sure that bhyve destroyed it and run > > Windows guest with the same PCI device passed-through. > > > > Windows device manager does show the device, however, e.g. a flash drive > > plugged in is not presented to Windows, instead it's being processed by > > FreeBSD. > > > > After that it does not work in Linux guest as well. Kernel module (vmm) > > unloading and reloading does not help. > > The flash drive being processed by FreeBSD would indicate that it has > ownership of the device. Would you be able to try a 'pciconf -vl' after > the Linux guest exists, and after the Windows guest exits ? First of all I found that I do not need to switch between guests. It is 100% reproducible with a single Linux VM. In freshly booted FreeBSD the USB ports (ones passed through) do not sense any device plugged in. When I boot Linux VM, it fully controls those ports, and I see the USB flash in Linux. As soon as I halt Linux, the flash drive appears in the host! Today I've checked out the latest sources and recompiled the whole system. Nothing has changed. Here is the corresponding output of pciconf iand devinfo after Linux exits: $ pciconf -vl ppt0@pci0:0:20:0: class=0x0c0330 card=0x21f317aa chip=0x1e318086 rev=0x04 hdr=0x00 vendor = 'Intel Corporation' device = '7 Series/C210 Series Chipset Family USB xHCI Host Controller' class = serial bus subclass = USB $ devinfo | grep -B15 mass pcib1 pci1 sdhci_pci0 pcib2 pci2 iwn0 pcib3 pci3 ehci1 usbus1 uhub1 uhub3 umass0 -- S. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve PCI pass-through to Linux guest
Excerpts from Anish's message from Mon 21-Dec-15 21:48: > >ppt0@pci0:0:20:0: class=0x0c0330 card=0x21f317aa chip=0x1e318086 > Passthrough stub driver, part of vmm, controls the USB controller. > > Can you share output of /usr/sbin/devinfo from FreeBSD host, highlighting > the usb mass/flash device in tree? > Before I start the guest VM the flash drive is not in the list. Right after shuting down the guest OS it appears: pcib1 pci1 sdhci_pci0 pcib2 pci2 iwn0 pcib3 pci3 ehci1 usbus1 uhub1 uhub3 umass0 As you can see it's under "ehci", it looks that my USB3 controller is fully USB2 backward compatible and work with "ehci" driver. I tried (as Trent suggested in this thread) to recompile the kernel without "xhci", and nothing changed, still behaves exactly the same way.. S. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: {Spam?} Re: bhyve PCI pass-through to Linux guest
Hi Sergey, This is pretty reproducible: I pass trough a PCI device (USB controller) to a Linux guest. It works properly. Then I halt the VM, make sure that bhyve destroyed it and run Windows guest with the same PCI device passed-through. Windows device manager does show the device, however, e.g. a flash drive plugged in is not presented to Windows, instead it's being processed by FreeBSD. After that it does not work in Linux guest as well. Kernel module (vmm) unloading and reloading does not help. The flash drive being processed by FreeBSD would indicate that it has ownership of the device. Would you be able to try a 'pciconf -vl' after the Linux guest exists, and after the Windows guest exits ? later, Peter. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve PCI pass-through to Linux guest
>ppt0@pci0:0:20:0: class=0x0c0330 card=0x21f317aa chip=0x1e318086 Passthrough stub driver, part of vmm, controls the USB controller. Can you share output of /usr/sbin/devinfo from FreeBSD host, highlighting the usb mass/flash device in tree? Regards, Anish On Mon, Dec 21, 2015 at 7:44 PM, Sergey Manucharianwrote: > Excerpts from Peter Grehan's message from Tue 22-Dec-15 13:09: > > Hi Sergey, > > > > > This is pretty reproducible: > > > > > > I pass trough a PCI device (USB controller) to a Linux guest. It works > > > properly. Then I halt the VM, make sure that bhyve destroyed it and run > > > Windows guest with the same PCI device passed-through. > > > > > > Windows device manager does show the device, however, e.g. a flash > drive > > > plugged in is not presented to Windows, instead it's being processed by > > > FreeBSD. > > > > > > After that it does not work in Linux guest as well. Kernel module (vmm) > > > unloading and reloading does not help. > > > > The flash drive being processed by FreeBSD would indicate that it has > > ownership of the device. Would you be able to try a 'pciconf -vl' after > > the Linux guest exists, and after the Windows guest exits ? > > OK, I've experimented more and found that even with the same Linux VM > after halting it once, the pass-through device doesn't work when that > Linux guest is restarted again. > > It looks that both host and guest report exactly the same data before and > after restarting the guest: > > Linux - first run: > = > $ lspci -v > 00:07.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset > Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI]) > Subsystem: Lenovo Device 21f3 > Flags: bus master, medium devsel, latency 0, IRQ 24 > Memory at c001 (64-bit, prefetchable) [size=64K] > Capabilities: [70] Power Management version 2 > Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+ > Kernel driver in use: xhci_hcd > > $ lsusb > Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub > Bus 001 Device 002: ID 0781:5530 SanDisk Corp. Cruzer > Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub > > FreeBSD when Linux is running first time: > = > $ pciconf -v > ppt0@pci0:0:20:0: class=0x0c0330 card=0x21f317aa chip=0x1e318086 > rev=0x04 hdr=0x00 > vendor = 'Intel Corporation' > device = '7 Series/C210 Series Chipset Family USB xHCI Host > Controller' > class = serial bus > subclass = USB > > Linux - second run: > === > $ lspci -v > 00:07.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset > Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI]) > Subsystem: Lenovo Device 21f3 > Flags: bus master, medium devsel, latency 0, IRQ 24 > Memory at c001 (64-bit, prefetchable) [size=64K] > Capabilities: [70] Power Management version 2 > Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+ > Kernel driver in use: xhci_hcd > > FreeBSD when Linux is running second time: > == > $ pciconf -lv > ppt0@pci0:0:20:0: class=0x0c0330 card=0x21f317aa chip=0x1e318086 > rev=0x04 hdr=0x00 > vendor = 'Intel Corporation' > device = '7 Series/C210 Series Chipset Family USB xHCI Host > Controller' > class = serial bus > subclass = USB > > > -- > Thanks, > Sergey > > ___ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscr...@freebsd.org" > ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: {Spam?} Re: bhyve PCI pass-through to Linux guest
Sergey, I've seen some success passing through devices that have their driver disabled in the FreeBSD kernel. See the USB 3.0 pass through that Eresia set up in iohyve. https://github.com/pr1ntf/iohyve/wiki/USB-3.0-PCI-Controller-Pass-through When he disabled the kernel module in FreeBSD, he can then use the device in the bhyve guest. Unfortunately, I have not worked on this specific problem myself, and cannot speak for this solution. -Trent On Mon, Dec 21, 2015 at 8:09 PM, Peter Grehanwrote: > Hi Sergey, > > This is pretty reproducible: >> >> I pass trough a PCI device (USB controller) to a Linux guest. It works >> properly. Then I halt the VM, make sure that bhyve destroyed it and run >> Windows guest with the same PCI device passed-through. >> >> Windows device manager does show the device, however, e.g. a flash drive >> plugged in is not presented to Windows, instead it's being processed by >> FreeBSD. >> >> After that it does not work in Linux guest as well. Kernel module (vmm) >> unloading and reloading does not help. >> > > The flash drive being processed by FreeBSD would indicate that it has > ownership of the device. Would you be able to try a 'pciconf -vl' after the > Linux guest exists, and after the Windows guest exits ? > > later, > > Peter. > > ___ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscr...@freebsd.org" > ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Re: {Spam?} Re: bhyve PCI pass-through to Linux guest
Excerpts from Trent Thompson's message from Mon 21-Dec-15 20:48: > I've seen some success passing through devices that have their driver > disabled in the FreeBSD kernel. > See the USB 3.0 pass through that Eresia set up in iohyve. > https://github.com/pr1ntf/iohyve/wiki/USB-3.0-PCI-Controller-Pass-through > > When he disabled the kernel module in FreeBSD, he can then use the device > in the bhyve guest. Thank you for the advice, Trent. Of course, that's probably a good work-around. However if it works once there should be a way to rectify that behaviour. Sergey ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve PCI pass-through to Linux guest
Excerpts from Peter Grehan's message from Tue 22-Dec-15 13:09: > Hi Sergey, > > > This is pretty reproducible: > > > > I pass trough a PCI device (USB controller) to a Linux guest. It works > > properly. Then I halt the VM, make sure that bhyve destroyed it and run > > Windows guest with the same PCI device passed-through. > > > > Windows device manager does show the device, however, e.g. a flash drive > > plugged in is not presented to Windows, instead it's being processed by > > FreeBSD. > > > > After that it does not work in Linux guest as well. Kernel module (vmm) > > unloading and reloading does not help. > > The flash drive being processed by FreeBSD would indicate that it has > ownership of the device. Would you be able to try a 'pciconf -vl' after > the Linux guest exists, and after the Windows guest exits ? OK, I've experimented more and found that even with the same Linux VM after halting it once, the pass-through device doesn't work when that Linux guest is restarted again. It looks that both host and guest report exactly the same data before and after restarting the guest: Linux - first run: = $ lspci -v 00:07.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI]) Subsystem: Lenovo Device 21f3 Flags: bus master, medium devsel, latency 0, IRQ 24 Memory at c001 (64-bit, prefetchable) [size=64K] Capabilities: [70] Power Management version 2 Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+ Kernel driver in use: xhci_hcd $ lsusb Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 002: ID 0781:5530 SanDisk Corp. Cruzer Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub FreeBSD when Linux is running first time: = $ pciconf -v ppt0@pci0:0:20:0: class=0x0c0330 card=0x21f317aa chip=0x1e318086 rev=0x04 hdr=0x00 vendor = 'Intel Corporation' device = '7 Series/C210 Series Chipset Family USB xHCI Host Controller' class = serial bus subclass = USB Linux - second run: === $ lspci -v 00:07.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI]) Subsystem: Lenovo Device 21f3 Flags: bus master, medium devsel, latency 0, IRQ 24 Memory at c001 (64-bit, prefetchable) [size=64K] Capabilities: [70] Power Management version 2 Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+ Kernel driver in use: xhci_hcd FreeBSD when Linux is running second time: == $ pciconf -lv ppt0@pci0:0:20:0: class=0x0c0330 card=0x21f317aa chip=0x1e318086 rev=0x04 hdr=0x00 vendor = 'Intel Corporation' device = '7 Series/C210 Series Chipset Family USB xHCI Host Controller' class = serial bus subclass = USB -- Thanks, Sergey ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve PCI pass-through to Linux guest
Hi Sergey, # bhyve -S ... -s 7,passthru,0/20/0 ... Make sure you are also passing the '-S' option to grub-bhyve to force wiring of memory. Sure! I showed it in the command line above. That's the bhyve command line (if that's the one you meant). You also have to pass '-S' when invoking grub-bhyve, since that's when the memory resources are created. later, Peter. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve PCI pass-through to Linux guest
Hi Sergey, I have a working Linux guest in bhyve (Debian). Trying to pass through a PCI device (USB controller) by adding a couple of options: # bhyve -S ... -s 7,passthru,0/20/0 ... That causes bhyve termination right away with error: Unable to setup memory (17) Exactly the same option with passing the same USB controller to Windows 2012 guest works perfectly. The only difference (by my understanding) is that I load UEFI for Windows and use grub-bhyve for Linux. What should I check? Make sure you are also passing the '-S' option to grub-bhyve to force wiring of memory. later, Peter. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve PCI pass-through to Linux guest
Excerpts from Peter Grehan's message from Sun 20-Dec-15 15:18: > >>> # bhyve -S ... -s 7,passthru,0/20/0 ... > >>> > >>Make sure you are also passing the '-S' option to grub-bhyve to force > >> wiring of memory. > > > > Sure! I showed it in the command line above. > > That's the bhyve command line (if that's the one you meant). You also > have to pass '-S' when invoking grub-bhyve, since that's when the memory > resources are created. Thanks, Peter! I've missed that! What a foolish omission! Works fine now. S. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve PCI pass-through to Linux guest
Excerpts from Peter Grehan's message from Sun 20-Dec-15 15:02: > > > > # bhyve -S ... -s 7,passthru,0/20/0 ... > > > Make sure you are also passing the '-S' option to grub-bhyve to force > wiring of memory. Sure! I showed it in the command line above. Without that bhyve says: "passthru requires guest memory to be wired" S. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"