On 24/02/2016 15:41, Daniel Pocock wrote:
Hi,
I'm trying to use PCI passthrough to give an NVIDIA GPU to a VM with
qemu / KVM. I've summarized my environment below and the error I get is
near the bottom. Any help would be appreciated.
There are a few guides I've been referring to already:
https://wiki.debian.org/VGAPassthrough
https://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/
https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
https://bbs.archlinux.org/viewtopic.php?id=162768
http://www.linux-kvm.org/page/VGA_device_assignment
Hi Daniel
I'm successfully passing through two Quadro M5000 (to two instances of
Windows 10), looking at
http://us.download.nvidia.com/Windows/Quadro_Certified/361.91/361.91-win10-quadro-grid-release-notes.pdf
(page 9) it is not obvious that this would work for either your K420 or
mine M5000. One gotcha - when starting Windows I do not see boot screen
at all, only Windows logon scren after nVidia drivers had loaded.
Explanation is in Alex's email sent to this list on 6th Feb 2016,
subject "No boot screen on Quadro M5000?" - basically passed through
Quadro cards are meant to be secondary only (I use them as only card,
though). However this small quirk aside, this works for me very well.
Here are details of my setup:
* distribution Arch , with packages : qemu 2.5.0 libvirt 1.3.1 (I do not
use qemu command line directly)
* my machine has 32 logical cores (2 x 8 core Xeon E5-26xx CPUs, with
HT) and 128GB ECC RAM, two Quadro M5000 cards and two USB 3.0 controller
cards; my motherboard is Supermicro X9DA7. I also dedicated half of
cores and about half of RAM to guests (I'm also running Linux virtual
machines with spice). Obviously you do not need this much hardware, just
a context to help you understand my configuration.
* I am passing through one GPU and one USB controller for each of two
Windows 10 guests (my Linux guests are running without passthrough at
this time, although also I experimented with such setup)
* kernel 4.1.17 (vanilla, no patches at all)
* useful bits in /etc :
bronek@gdansk /etc % cat modprobe.d/blacklist.conf
blacklist nouveau
blacklist snd_hda_intel
bronek@gdansk /etc % cat modprobe.d/kvm.conf
options kvm ignore_msrs=1
bronek@gdansk /etc % cat modprobe.d/vfio.conf
options vfio-pci ids=1912:0014,10de:13f0,10de:0fbb
bronek@gdansk /etc % cat sysctl.d/80-hugepages.conf
# Reserve this many 2MB pages for virtual machine
vm.nr_hugepages = 28000
bronek@gdansk /etc % grep -E "^MODULES" /etc/mkinitcpio.conf
MODULES="vfio vfio_iommu_type1 vfio_pci vfio_virqfd vhost_net bridge
mpt2sas nvme"
bronek@gdansk /etc % sed 's|192.168.[0-9.]*|192.168.#.#|g' netctl/br0
Description="Bridge connection"
Interface=br0
Connection=bridge
BindsToInterfaces=(enp5s0f1)
IP=static
Address=('192.168.#.#/24')
## Do not want default gateway on this interface
# Gateway='192.168.#.#'
Routes=('192.168.#.#/24 via 192.168.#.#')
DNS=('192.168.#.#' '192.168.#.#')
NETCTL_DEBUG=yes
## Ignore (R)STP and immediately activate the bridge
SkbpForwardingDelay=yes
bronek@gdansk /etc % lspci -tvnn | less
. . .
+-[0000:80]-+-01.0-[81]--
| +-02.0-[82]--+-00.0 NVIDIA Corporation GM204GL [Quadro
M5000] [10de:13f0]
| | \-00.1 NVIDIA Corporation GM204 High
Definition Audio Controller [10de:0fbb]
| +-03.0-[83]----00.0 Intel Corporation PCIe Data Center
SSD [8086:0953]
| +-03.2-[84]----00.0 Renesas Technology Corp. uPD720201
USB 3.0 Host Controller [1912:0014]
. . .
\-[0000:00]-+-00.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7
DMI2 [8086:0e00]
+-01.0-[01]----00.0 LSI Logic / Symbios Logic SAS2308
PCI-Express Fusion-MPT SAS-2 [1000:0086]
+-02.0-[02]----00.0 Renesas Technology Corp. uPD720201
USB 3.0 Host Controller [1912:0014]
+-03.0-[03]--+-00.0 NVIDIA Corporation GM204GL [Quadro
M5000] [10de:13f0]
| \-00.1 NVIDIA Corporation GM204 High
Definition Audio Controller [10de:0fbb]
* boot configuration (i.e. kernel command line parameters):
console=ttyS0,115200N8R nomodest nohz=off udev.children-max=32
edac_core.edac_mc_panic_on_ue=1 intel_iommu=on iommu=pt
isolcpus=4,5,6,7,12,13,14,15,20,21,22,23,28,29,30,31
(I removed parts relevant to ZFS only, which I also use)
* my guests are using UEFI boot in kvm (my host is starting from
"legacy" BIOS, though), using edk2.git-ovmf-x64-... from
https://www.kraxel.org/repos/jenkins/edk2/
* for Windows installation (and possibnly boot-level maintenance in the
future) I use libvirt profile like attached "-spice" profile. I start
and connect to this from an old laptop running Linux.
* also, as you will see in attached profiles, I am using virtio-scsi.
This means that at the start of Windows setup I need to install vioscsi
driver from virtio-win-0.1.112.iso (because otherwise installer won't
see disk to install to)
* first thing after Windows boot (still under spice), I install
remaining drivers and qemu guest-agent from virtio-win-0.1.112.iso
(note, use qxldod rather than qxl)
* when I'm done setting up Windows (still under spice), I issue two
commands to attach both passed-through cards and install drivers:
# attach-device --live bochnia-spice vfio2-nec.xml
# attach-device --live bochnia-spice vfio2-nvidia.xml
* after this I shutdown virtual machine running "-spice" profile and
then start it again using attached "-vfio2" profile i.e. replaced spice
with passed-through hardware
* NOTE: this line "<rom bar='off'/>" in attached vfio2 - without it my
passed through Quadro will not work at all.
When I'm done, I keep "spice" profile defined in libvirt "just in case",
but have not used it so far at all. Passed through card just works, lack
of Windows boot screen aside. FWIW, attached vfio2 is translated to this
command line by libvirt:
root@gdansk ~ # ps lfx | grep bochnia | grep -v grep
6 0 7494 1 20 0 20843916 90292 poll_s SLl ? 217:53
/usr/bin/qemu-system-x86_64 -name
bochnia-vfio2,process=qemu:bochnia-vfio2 -S -machine
pc-i440fx-2.5,accel=kvm,usb=off,mem-merge=off -cpu
host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 -drive
file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on
-drive
file=/var/lib/libvirt/qemu/nvram/bochnia-spice_VARS.fd,if=pflash,format=raw,unit=1
-m 16384 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu -realtime
mlock=off -smp 8,sockets=1,cores=4,threads=2 -uuid
dd4d4b52-0e06-41da-8357-c7792988ec4f -nographic -no-user-config
-nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-bochnia-vfio2/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet
-no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1
-boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7
-device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
-device
ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
-device
ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
-device virtio-scsi-pci,id=scsi0,num_queues=8,bus=pci.0,addr=0x4 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
file=/dev/zvol/zdata/vdis/bochnia,format=raw,if=none,id=drive-scsi0-0-0-0,cache=writeback
-device
scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
-drive
file=/dev/disk/by-partuuid/099c0f43-a887-4892-b459-89d0535739fa,format=raw,if=none,id=drive-scsi0-0-1-0,cache=none
-device
scsi-hd,bus=scsi0.0,channel=0,scsi-id=1,lun=0,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0
-drive
file=/data/vdis/isos/Windows10_x64_en_GB.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on
-device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive
file=/data/vdis/isos/virtio-win-0.1.112.iso,format=raw,if=none,id=drive-ide0-0-1,readonly=on
-device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev
tap,fds=25:27:28:29:30:31:32:33,id=hostnet0,vhost=on,vhostfds=34:35:36:37:38:39:40:41
-device
virtio-net-pci,mq=on,vectors=18,netdev=hostnet0,id=net0,mac=52:54:01:34:2e:f3,bus=pci.0,addr=0x18
-chardev
socket,id=charchannel0,path=/var/lib/libvirt/qemu/bochnia.agent,server,nowait
-device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0
-device usb-host,hostbus=4,hostaddr=4,id=hostdev0 -device
vfio-pci,host=84:00.0,id=hostdev1,bus=pci.0,addr=0x9 -device
vfio-pci,host=82:00.0,id=hostdev2,bus=pci.0,addr=0x8,rombar=0 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
B.
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x84' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x82' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</hostdev>
<domain type='kvm'>
<name>bochnia-spice</name>
<memory unit='KiB'>16777216</memory>
<currentMemory unit='KiB'>16777216</currentMemory>
<memoryBacking>
<hugepages>
<page size='2048' unit='KiB'/>
</hugepages>
<nosharepages/>
</memoryBacking>
<vcpu placement='static'>8</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='12'/>
<vcpupin vcpu='1' cpuset='28'/>
<vcpupin vcpu='2' cpuset='13'/>
<vcpupin vcpu='3' cpuset='29'/>
<vcpupin vcpu='4' cpuset='14'/>
<vcpupin vcpu='5' cpuset='30'/>
<vcpupin vcpu='6' cpuset='15'/>
<vcpupin vcpu='7' cpuset='31'/>
</cputune>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-i440fx-2.5'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
<nvram template='/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='4096'/>
</hyperv>
</features>
<cpu mode='host-passthrough'>
<topology sockets='1' cores='4' threads='2'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='writeback'/>
<source dev='/dev/zvol/zdata/vdis/bochnia'/>
<target dev='sda' bus='scsi'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/data/vdis/isos/Windows10_x64_en_GB.iso'/>
<target dev='hda' bus='ide'/>
<readonly/>
<boot order='1'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/data/vdis/isos/virtio-win-0.1.112.iso'/>
<target dev='hdb' bus='ide'/>
<readonly/>
<address type='drive' controller='0' bus='0' target='0' unit='1'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</controller>
<controller type='scsi' index='0' model='virtio-scsi'>
<driver queues='8'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='52:54:01:34:2e:f3'/>
<source bridge='br0'/>
<model type='virtio'/>
<driver queues='8'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x18' function='0x0'/>
</interface>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/bochnia.agent'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0'/>
<address type='virtio-serial' controller='0' bus='0' port='2'/>
</channel>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='spice' port='5907' autoport='no' listen='0.0.0.0'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<sound model='ich6'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0'/>
</sound>
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x11' function='0x0'/>
</video>
<redirdev bus='usb' type='spicevmc'>
</redirdev>
<redirdev bus='usb' type='spicevmc'>
</redirdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</memballoon>
</devices>
</domain>
<domain type='kvm' id='8'>
<name>bochnia-vfio2</name>
<memory unit='KiB'>16777216</memory>
<currentMemory unit='KiB'>16777216</currentMemory>
<memoryBacking>
<hugepages>
<page size='2048' unit='KiB'/>
</hugepages>
<nosharepages/>
</memoryBacking>
<vcpu placement='static'>8</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='12'/>
<vcpupin vcpu='1' cpuset='28'/>
<vcpupin vcpu='2' cpuset='13'/>
<vcpupin vcpu='3' cpuset='29'/>
<vcpupin vcpu='4' cpuset='14'/>
<vcpupin vcpu='5' cpuset='30'/>
<vcpupin vcpu='6' cpuset='15'/>
<vcpupin vcpu='7' cpuset='31'/>
</cputune>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-i440fx-2.5'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
<nvram template='/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd'>/var/lib/libvirt/qemu/nvram/bochnia-spice_VARS.fd</nvram>
</os>
<features>
<acpi/>
<apic/>
<pae/>
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='4096'/>
</hyperv>
</features>
<cpu mode='host-passthrough'>
<topology sockets='1' cores='4' threads='2'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='writeback'/>
<source dev='/dev/zvol/zdata/vdis/bochnia'/>
<target dev='sda' bus='scsi'/>
<boot order='1'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source dev='/dev/disk/by-partuuid/099c0f43-a887-4892-b459-89d0535739fa'/>
<target dev='sdb' bus='scsi'/>
<address type='drive' controller='0' bus='0' target='1' unit='0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/data/vdis/isos/Windows10_x64_en_GB.iso'/>
<target dev='hda' bus='ide'/>
<readonly/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/data/vdis/isos/virtio-win-0.1.112.iso'/>
<target dev='hdb' bus='ide'/>
<readonly/>
<address type='drive' controller='0' bus='0' target='0' unit='1'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</controller>
<controller type='scsi' index='0' model='virtio-scsi'>
<driver queues='8'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='52:54:01:34:2e:f3'/>
<source bridge='br0'/>
<model type='virtio'/>
<driver queues='8'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x18' function='0x0'/>
</interface>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/bochnia.agent'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<hostdev mode='subsystem' type='usb' managed='yes'>
<source>
<vendor id='0x046a'/>
<product id='0xc090'/>
</source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x84' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x82' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</hostdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</memballoon>
</devices>
</domain>
_______________________________________________
vfio-users mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/vfio-users
