|
http://wiki.xensource.com/xenwiki/VTdHowTo VT-d Pass-Through is a technique to give a domU exclusive access to a PCI function using the IOMMU provided by VT-d. It is primarily targeted at HVM (fully virtualised) guests. Supported OSes
Tested DomO OSes
VT-d Enabled Systems
Caveat on Conventional PCI Device Pass-Through
Notes on Compiling Source CodeIt is not the intention of this document to describe in detail how to compile and install Xen from source. Rather, the intention of this section is to highlight points of particular note when compiling from source for use with VT-d. Dom0 KernelA prerequisite for binding devices to pciback, which is discussed in more detail below, is that pciback is compiled statically into or as a module for the dom0 kernel. This means that in the kernel config CONFIG_XEN_BACKEND=y should be set to statically compile pciback into the kernel or CONFIG_XEN_BACKEND=m should be set to compile pciback as a module. Device ModelWhen building the device model, also known as qemu-dm and qemu-xen, often automatically downloaded during compilation and found in the tools/ioemu-dir/ subdirectory of the xen tree, it is important that the development headers and libraries for libpci are installed. Otherwise qemu-dm will be built without pass-through support. If the device model has been compiled without pass-through support, the following error will show up in xend.log at run-time when pass-through is attempted.
Binding Devices to pcibackIn order to pass-through devices using VT-d then need to be bound to pciback to ensure that they are not bound to another dom0 driver and this free for use by pass-through. It is possible to view this as hiding the device from Dom0. Binding at Boot-TimeIf pciback is statically compiled into the kernel, then perhaps the simplest to make PCI devices available for VT-d pass-through is to use the pciback.hide kernel parameter, as illustrated in the following grub configuration snippet.
The arguments to pciback.hide are PCI functions in BDFNotation, enclosed in brackets. The output of the lspci command may be useful when selecting PCI functions to hide. Binding at Run-TimeAs discussed in Assign_hardware_to_DomU_with_PCIBack_as_module, it is possible to bind a device to pciback after dom0 has booted, even if it has been bound to another device in the mean time. This method can be used both when pciback is statically compiled into the dom0 kernel and when it has been compiled as a module. Binding at Module Insertion TimeAs also discussed in Assign_hardware_to_DomU_with_PCIBack_as_module, it is possible to bind a device to pciback at the time that the pciback module is inserted. This technique is only useful if pciback has been compiled as a module. Viewing Devices that are Available for Pass-Throughxm's pci-list-assignable-devices can be used to list devices that are available for pass-through. That is, devices that have been hidden from dom0 by pciback.
The devices are shown in Extended BDF notation, for more information see BDFNotation. Implementation Detailsxm finds all devices the devices owned by pciback, checks if they have proper FLR method, checks if they have page-aligned MMIO BARs, and checks if the devices have been already assigned, finally it prints out the assignable devices. xm assumes the integrated devices (whose bus number is 0) have a proper FLR method. Availability of this Feature
Boot-Time VT-d Device Pass-ThroughAs the name suggests, boot-time pass-through occurs at HVM domU (guest) boot-time. It is also referred to as static VT-d device pass-through, however, this name is confusing as devices attached using this method may subsequently be hot-unplugged. To configure static pass-through for a domain add a "pci" line to the configuration file for the domain. For example, the following may be added to /etc/xen/hvm.conf:
After updating the configuration file, start HVM guest and use "lspci" to verify that the the pass-through device is present in the guest. "ifconfig" or "ip addr show" may be used to see if IP address has been assigned to NIC devices. Virtual Slot DesignationAs of Xen 3.4.0 it is possible to specify the virtual slot that is used for by boot-time VT-d device pass-through. This is optional and when done is specified on a per-device basis by appending @SLOT_NUMBER to the BDF. For example, the following assigns 00:02.0 to virtual slot 7 while leaving xen to choose the slot for 01:00.0:
Viewing Devices That Have Been Passed-Throughxm's pci-list command can be used to view the VT-d devices that have been passed-through to a domain. For example, below the devices attached to the HVM domain named HVMDomainVtd are shown.
Note that the domain field in the output above refers to the PCI domain not the Xen domain. PCI domains are a physical property of the host, as are PCI buses. VT-d Device Hot-PlugVT-d device hot-plug involves attaching a device to an HVM domain once it has booted. xm's pci-attach command is used to perform hot-plug The device to be attached is specified using BDFNotation and optionaly the desired virtual slot. If the virtual slot is omitted then a free one will be used. For example, the following command inserts the physical device into the HVM domain called HVMDomainVtd at virtual slot 7:
As described above, xm pci-list can be used to verify the that the device is attached. VT-d Device Hot-UnplugVT-d hot-unplug refers to detaching a pass-through device from a running HVM domain. The device may have been attached using boot-time VT-d device pass-through or VT-d device hot-plug. xm's pci-detach command is used to perform hot-unplug. The device to be detached is specified using BDFNotation. The following command removes the device 0:2:0.0 from the domain named HVMDomainVtd.
As previously shown, pci-list can be used to show the devices that are attached to a domain and thus confirm that 0:2:0.0 has been detached as requested. VT-d Device Hot-Plug/Unplug Usage Model
Virtual SlotsUp until Xen 3.4.0, slots 6 and 7 were reserved in HVM guests for VT-d pass-through. This meant that:
Xen 3.4.0 removes the reservation system. Any slot which isn't in use may be used for pass-through or IOEMU devices. Given that HMV domains have one virtual PCI bus, that a PCI bus has 32 slots for devices, and that a domain typically has a minimum of 3 IOEMU devices, 28 will be available for pass-through or extra IOEMU devices. Enabling MSI/MSI-X for Assigned Devices
Single-Function and Multi-Function Pass-ThroughA PCI device may have up to 8 functions. A device with only one function is called a single-function device and a device with more than one function is called a multi-function device. A common example of a multi-function device are USB devices. For example, the following USB device is device 1a on bus 0 and has 3 functions: 0, 1 and 7.
As of Xen 3.4.0, and in all prior versions of Xen that supported VT-d pass-through, only single-function VT-d pass-through devices are supported in HVM domains. This means that if a function of as multi-function device is passed-through to an HVM domain, it will appear as function zero of a single-function device in the HVM domain, regardless of its physical function number. Single-function devices always have function zero present (and no other functions) and thus the function number does not change when a single-function device is passed through they are passed-through. The following table shows that regardless of the physical function number, the virtual function number will be zero.
Work is being done by Simon_Horman to allow multi-function devices to be passed-through and and appear as multi-function devices in HVM domains. The time-frame for merging this work is post 3.4.0. VTdHowTo (last edited 2009-04-23 00:30:48 by Simon Horman) |
