Re: bhyve PCI pass-through to Linux guest

[Sergey Manucharian]

Excerpts from Peter Grehan's message from Tue 22-Dec-15 13:09:
> > I pass trough a PCI device (USB controller) to a Linux guest. It works
> > properly. Then I halt the VM, make sure that bhyve destroyed it and run
> > Windows guest with the same PCI device passed-through.
> >
> > Windows device manager does show the device, however, e.g. a flash drive
> > plugged in is not presented to Windows, instead it's being processed by
> > FreeBSD.
> >
> > After that it does not work in Linux guest as well. Kernel module (vmm)
> > unloading and reloading does not help.
> 
>   The flash drive being processed by FreeBSD would indicate that it has 
> ownership of the device. Would you be able to try a 'pciconf -vl' after 
> the Linux guest exists, and after the Windows guest exits ?

First of all I found that I do not need to switch between guests. It is
100% reproducible with a single Linux VM. In freshly booted FreeBSD the USB
ports (ones passed through) do not sense any device plugged in. When I boot
Linux VM, it fully controls those ports, and I see the USB flash in Linux.
As soon as I halt Linux, the flash drive appears in the host!

Today I've checked out the latest sources and recompiled the whole system. 
Nothing has changed.

Here is the corresponding output of pciconf iand devinfo after Linux exits:

$ pciconf -vl

ppt0@pci0:0:20:0:   class=0x0c0330 card=0x21f317aa chip=0x1e318086
rev=0x04 hdr=0x00
vendor = 'Intel Corporation'
device = '7 Series/C210 Series Chipset Family USB xHCI Host
Controller'
class  = serial bus
subclass   = USB


$ devinfo | grep -B15 mass

pcib1
  pci1
sdhci_pci0
pcib2
  pci2
iwn0
pcib3
  pci3
ehci1
  usbus1
uhub1
  uhub3
umass0


--
S.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve PCI pass-through to Linux guest

[Sergey Manucharian]

Excerpts from Anish's message from Mon 21-Dec-15 21:48:
> >ppt0@pci0:0:20:0:   class=0x0c0330 card=0x21f317aa chip=0x1e318086
> Passthrough stub driver, part of vmm, controls the USB controller.
> 
> Can you share output of /usr/sbin/devinfo from FreeBSD host, highlighting
> the usb mass/flash device in tree?
> 

Before I start the guest VM the flash drive is not in the list. Right
after shuting down the guest OS it appears:

pcib1
  pci1
sdhci_pci0
pcib2
  pci2
iwn0
pcib3
  pci3
ehci1
  usbus1
uhub1
  uhub3
umass0

As you can see it's under "ehci", it looks that my USB3 controller is
fully USB2 backward compatible and work with "ehci" driver. I tried (as
Trent suggested in this thread) to recompile the kernel without "xhci",
and nothing changed, still behaves exactly the same way..

S.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: {Spam?} Re: bhyve PCI pass-through to Linux guest

[Peter Grehan]

Hi Sergey,


This is pretty reproducible:

I pass trough a PCI device (USB controller) to a Linux guest. It works
properly. Then I halt the VM, make sure that bhyve destroyed it and run
Windows guest with the same PCI device passed-through.

Windows device manager does show the device, however, e.g. a flash drive
plugged in is not presented to Windows, instead it's being processed by
FreeBSD.

After that it does not work in Linux guest as well. Kernel module (vmm)
unloading and reloading does not help.


 The flash drive being processed by FreeBSD would indicate that it has 
ownership of the device. Would you be able to try a 'pciconf -vl' after 
the Linux guest exists, and after the Windows guest exits ?


later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve PCI pass-through to Linux guest

[Anish]

>ppt0@pci0:0:20:0:   class=0x0c0330 card=0x21f317aa chip=0x1e318086
Passthrough stub driver, part of vmm, controls the USB controller.

Can you share output of /usr/sbin/devinfo from FreeBSD host, highlighting
the usb mass/flash device in tree?

Regards,
Anish

On Mon, Dec 21, 2015 at 7:44 PM, Sergey Manucharian  wrote:

> Excerpts from Peter Grehan's message from Tue 22-Dec-15 13:09:
> > Hi Sergey,
> >
> > > This is pretty reproducible:
> > >
> > > I pass trough a PCI device (USB controller) to a Linux guest. It works
> > > properly. Then I halt the VM, make sure that bhyve destroyed it and run
> > > Windows guest with the same PCI device passed-through.
> > >
> > > Windows device manager does show the device, however, e.g. a flash
> drive
> > > plugged in is not presented to Windows, instead it's being processed by
> > > FreeBSD.
> > >
> > > After that it does not work in Linux guest as well. Kernel module (vmm)
> > > unloading and reloading does not help.
> >
> >   The flash drive being processed by FreeBSD would indicate that it has
> > ownership of the device. Would you be able to try a 'pciconf -vl' after
> > the Linux guest exists, and after the Windows guest exits ?
>
> OK, I've experimented more and found that even with the same Linux VM
> after halting it once, the pass-through device doesn't work when that
> Linux guest is restarted again.
>
> It looks that both host and guest report exactly the same data before and
> after restarting the guest:
>
> Linux - first run:
> =
> $ lspci -v
> 00:07.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset
> Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI])
> Subsystem: Lenovo Device 21f3
> Flags: bus master, medium devsel, latency 0, IRQ 24
> Memory at c001 (64-bit, prefetchable) [size=64K]
> Capabilities: [70] Power Management version 2
> Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
> Kernel driver in use: xhci_hcd
>
> $ lsusb
> Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
> Bus 001 Device 002: ID 0781:5530 SanDisk Corp. Cruzer
> Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
>
> FreeBSD when Linux is running first time:
> =
> $ pciconf -v
> ppt0@pci0:0:20:0:   class=0x0c0330 card=0x21f317aa chip=0x1e318086
> rev=0x04 hdr=0x00
> vendor = 'Intel Corporation'
> device = '7 Series/C210 Series Chipset Family USB xHCI Host
> Controller'
> class  = serial bus
> subclass   = USB
>
> Linux - second run:
> ===
> $ lspci -v
> 00:07.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset
> Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI])
> Subsystem: Lenovo Device 21f3
> Flags: bus master, medium devsel, latency 0, IRQ 24
> Memory at c001 (64-bit, prefetchable) [size=64K]
> Capabilities: [70] Power Management version 2
> Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
> Kernel driver in use: xhci_hcd
>
> FreeBSD when Linux is running second time:
> ==
> $ pciconf -lv
> ppt0@pci0:0:20:0:   class=0x0c0330 card=0x21f317aa chip=0x1e318086
> rev=0x04 hdr=0x00
> vendor = 'Intel Corporation'
> device = '7 Series/C210 Series Chipset Family USB xHCI Host
> Controller'
> class  = serial bus
> subclass   = USB
>
>
> --
> Thanks,
> Sergey
>
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "
> freebsd-virtualization-unsubscr...@freebsd.org"
>
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: {Spam?} Re: bhyve PCI pass-through to Linux guest

[Trent Thompson]

Sergey,

I've seen some success passing through devices that have their driver
disabled in the FreeBSD kernel.
See the USB 3.0 pass through that Eresia set up in iohyve.
https://github.com/pr1ntf/iohyve/wiki/USB-3.0-PCI-Controller-Pass-through

When he disabled the kernel module in FreeBSD, he can then use the device
in the bhyve guest.

Unfortunately, I have not worked on this specific problem myself, and
cannot speak for this solution.

-Trent

On Mon, Dec 21, 2015 at 8:09 PM, Peter Grehan  wrote:

> Hi Sergey,
>
> This is pretty reproducible:
>>
>> I pass trough a PCI device (USB controller) to a Linux guest. It works
>> properly. Then I halt the VM, make sure that bhyve destroyed it and run
>> Windows guest with the same PCI device passed-through.
>>
>> Windows device manager does show the device, however, e.g. a flash drive
>> plugged in is not presented to Windows, instead it's being processed by
>> FreeBSD.
>>
>> After that it does not work in Linux guest as well. Kernel module (vmm)
>> unloading and reloading does not help.
>>
>
>  The flash drive being processed by FreeBSD would indicate that it has
> ownership of the device. Would you be able to try a 'pciconf -vl' after the
> Linux guest exists, and after the Windows guest exits ?
>
> later,
>
> Peter.
>
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "
> freebsd-virtualization-unsubscr...@freebsd.org"
>
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Re: {Spam?} Re: bhyve PCI pass-through to Linux guest

[Sergey Manucharian]

Excerpts from Trent Thompson's message from Mon 21-Dec-15 20:48:
> I've seen some success passing through devices that have their driver
> disabled in the FreeBSD kernel.
> See the USB 3.0 pass through that Eresia set up in iohyve.
> https://github.com/pr1ntf/iohyve/wiki/USB-3.0-PCI-Controller-Pass-through
> 
> When he disabled the kernel module in FreeBSD, he can then use the device
> in the bhyve guest.

Thank you for the advice, Trent. Of course, that's probably a good
work-around. However if it works once there should be a way to rectify
that behaviour.

Sergey

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve PCI pass-through to Linux guest

[Sergey Manucharian]

Excerpts from Peter Grehan's message from Tue 22-Dec-15 13:09:
> Hi Sergey,
> 
> > This is pretty reproducible:
> >
> > I pass trough a PCI device (USB controller) to a Linux guest. It works
> > properly. Then I halt the VM, make sure that bhyve destroyed it and run
> > Windows guest with the same PCI device passed-through.
> >
> > Windows device manager does show the device, however, e.g. a flash drive
> > plugged in is not presented to Windows, instead it's being processed by
> > FreeBSD.
> >
> > After that it does not work in Linux guest as well. Kernel module (vmm)
> > unloading and reloading does not help.
> 
>   The flash drive being processed by FreeBSD would indicate that it has 
> ownership of the device. Would you be able to try a 'pciconf -vl' after 
> the Linux guest exists, and after the Windows guest exits ?

OK, I've experimented more and found that even with the same Linux VM
after halting it once, the pass-through device doesn't work when that
Linux guest is restarted again.

It looks that both host and guest report exactly the same data before and
after restarting the guest:

Linux - first run:
=
$ lspci -v
00:07.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset
Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI])
Subsystem: Lenovo Device 21f3
Flags: bus master, medium devsel, latency 0, IRQ 24
Memory at c001 (64-bit, prefetchable) [size=64K]
Capabilities: [70] Power Management version 2
Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
Kernel driver in use: xhci_hcd

$ lsusb 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 0781:5530 SanDisk Corp. Cruzer
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

FreeBSD when Linux is running first time:
=
$ pciconf -v
ppt0@pci0:0:20:0:   class=0x0c0330 card=0x21f317aa chip=0x1e318086
rev=0x04 hdr=0x00
vendor = 'Intel Corporation'
device = '7 Series/C210 Series Chipset Family USB xHCI Host
Controller'
class  = serial bus
subclass   = USB

Linux - second run:
===
$ lspci -v
00:07.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset
Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI])
Subsystem: Lenovo Device 21f3
Flags: bus master, medium devsel, latency 0, IRQ 24
Memory at c001 (64-bit, prefetchable) [size=64K]
Capabilities: [70] Power Management version 2
Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
Kernel driver in use: xhci_hcd

FreeBSD when Linux is running second time:
==
$ pciconf -lv
ppt0@pci0:0:20:0:   class=0x0c0330 card=0x21f317aa chip=0x1e318086
rev=0x04 hdr=0x00
vendor = 'Intel Corporation'
device = '7 Series/C210 Series Chipset Family USB xHCI Host
Controller'
class  = serial bus
subclass   = USB


--
Thanks,
Sergey

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve PCI pass-through to Linux guest

[Peter Grehan]

Hi Sergey,


# bhyve -S ... -s 7,passthru,0/20/0 ...


   Make sure you are also passing the '-S' option to grub-bhyve to force
wiring of memory.


Sure! I showed it in the command line above.


 That's the bhyve command line (if that's the one you meant). You also 
have to pass '-S' when invoking grub-bhyve, since that's when the memory 
resources are created.


later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve PCI pass-through to Linux guest

[Peter Grehan]

Hi Sergey,

I have a working Linux guest in bhyve (Debian). Trying to pass through a
PCI device (USB controller) by adding a couple of options:

# bhyve -S ... -s 7,passthru,0/20/0 ...

That causes bhyve termination right away with error:
Unable to setup memory (17)

Exactly the same option with passing the same USB controller to Windows 2012
guest works perfectly.

The only difference (by my understanding) is that I load UEFI for Windows
and use grub-bhyve for Linux.

What should I check?


 Make sure you are also passing the '-S' option to grub-bhyve to force 
wiring of memory.


later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve PCI pass-through to Linux guest

[Sergey Manucharian]

Excerpts from Peter Grehan's message from Sun 20-Dec-15 15:18:
> >>> # bhyve -S ... -s 7,passthru,0/20/0 ...
> >>>
> >>Make sure you are also passing the '-S' option to grub-bhyve to force
> >> wiring of memory.
> >
> > Sure! I showed it in the command line above.
> 
>   That's the bhyve command line (if that's the one you meant). You also 
> have to pass '-S' when invoking grub-bhyve, since that's when the memory 
> resources are created.

Thanks, Peter! I've missed that! What a foolish omission!

Works fine now.

S.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve PCI pass-through to Linux guest

[Sergey Manucharian]

Excerpts from Peter Grehan's message from Sun 20-Dec-15 15:02:
> >
> > # bhyve -S ... -s 7,passthru,0/20/0 ...
> >
>   Make sure you are also passing the '-S' option to grub-bhyve to force 
> wiring of memory.

Sure! I showed it in the command line above. Without that bhyve says:

"passthru requires guest memory to be wired"

S.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"