I'm trying this on Proxmox VE, which is Debian 8 based.
I'm awaiting a UEFI bios from EVGA, but for now I don't have it available.
I'm following the "tips and tricks" series, but we differ at part 4, as I'd
like this to eventually work with my distro's web frontend.
With the goal of finding an invocation of the KVM command that will work, I did
a ps ax|grep kvm and modified what wasn't working in my distro to come up with
something that I think should work, which is:
=================
/usr/bin/kvm -id 102 -chardev
socket,id=qmp,path=/var/run/qemu-server/102.qmp,server,nowait -mon
chardev=qmp,mode=control -pidfile /var/run/qemu-server/102.pid -daemonize
-smbios type=1,uuid=6a51d2b1-c63f-4eef-9c58-30743a0252f6 -name MSW10E-1 -smp
8,sockets=1,cores=8,maxcpus=8 -nodefaults -boot
menu=on,strict=on,reboot-timeout=1000 -vga std -vnc
unix:/var/run/qemu-server/102.vnc,x509,password -no-hpet -cpu
host,kvm=off,hv_vendor_id=Nvidia43FIX,+kvm_pv_unhalt,+kvm_pv_eoi -m 12288 -k
en-us -readconfig /usr/share/qemu-server/pve-q35.cfg -device
usb-tablet,id=tablet,bus=ehci.0,port=1 -device
vfio-pci,host=04:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi
initiator-name=iqn.1993-08.org.debian:01:3f1e9afe6fdb -drive
file=/var/lib/vz/template/iso/MSW106in1.iso,if=none,id=drive-ide2,media=cdrom,aio=threads
-device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=100 -drive
file=/dev/zvol/rpool/data/vm-102-dis!
k-1,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on
-device
virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=200
-netdev
type=tap,id=net0,ifname=tap102i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on
-device
virtio-net-pci,mac=32:36:39:65:33:31,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
-rtc driftfix=slew,base=localtime -machine type=q35 -global
kvm-pit.lost_tick_policy=discard
=================
Specifically, I added the ,kvm=off,hv_vendor_id=Nvidia43FIX and removed all of
the hypervisor hv_ stuff which might flag the nvidia driver.
My understanding is that I'm still running up against the "problem" of the
nvidia driver detecting that it is in a VM, and going into stupid mode.
I did take a shortcut in that I don't load the vfio module in the initrd, but
instead I assign driver overrides to GPUs respectively for both the nouveau and
vfio-pci, with a slightly modified version of the script from the howto series.
I don't think it's causing a problem in my setup, but I realize it isn't
ideal.. Nouveau seems to work well on the boot GPU, though I haven't tested it
much.
Here is some additional info which may be helpful:
==========================================
dmesg |grep -e DMAR -e IOMMU
[ 0.000000] ACPI: DMAR 0x000000009F7980C0 000130 (v01 AMI OEMDMAR
00000001 MSFT 00000097)
[ 0.000000] DMAR: IOMMU enabled
[ 0.102094] DMAR-IR: This system BIOS has enabled interrupt remapping
[ 1.109864] DMAR: Host address width 39
[ 1.109915] DMAR: DRHD base: 0x000000fbffe000 flags: 0x1
[ 1.109984] DMAR: dmar0: reg_base_addr fbffe000 ver 1:0 cap c90780106f0462
ecap f020fe
[ 1.110054] DMAR: RMRR base: 0x000000000ec000 end: 0x000000000effff
[ 1.110108] DMAR: RMRR base: 0x0000009f7da000 end: 0x0000009f7d9fff
[ 1.110163] DMAR: ATSR flags: 0x0
[ 1.110410] DMAR: dmar0: Using Queued invalidation
[ 1.110475] DMAR: Setting RMRR:
[ 1.110545] DMAR: Setting identity map for device 0000:00:1a.0 [0x9f7da000 -
0x9f7d9fff]
[ 1.118284] DMAR: Mapping reserved region failed
[ 1.118350] DMAR: Setting identity map for device 0000:00:1a.1 [0x9f7da000 -
0x9f7d9fff]
[ 1.120550] DMAR: Mapping reserved region failed
[ 1.120611] DMAR: Setting identity map for device 0000:00:1a.2 [0x9f7da000 -
0x9f7d9fff]
[ 1.122815] DMAR: Mapping reserved region failed
[ 1.122876] DMAR: Setting identity map for device 0000:00:1a.7 [0x9f7da000 -
0x9f7d9fff]
[ 1.127756] DMAR: Mapping reserved region failed
[ 1.127818] DMAR: Setting identity map for device 0000:00:1d.0 [0x9f7da000 -
0x9f7d9fff]
[ 1.130015] DMAR: Mapping reserved region failed
[ 1.130076] DMAR: Setting identity map for device 0000:00:1d.1 [0x9f7da000 -
0x9f7d9fff]
[ 1.137610] DMAR: Mapping reserved region failed
[ 1.137676] DMAR: Setting identity map for device 0000:00:1d.2 [0x9f7da000 -
0x9f7d9fff]
[ 1.139892] DMAR: Mapping reserved region failed
[ 1.139953] DMAR: Setting identity map for device 0000:00:1d.7 [0x9f7da000 -
0x9f7d9fff]
[ 1.142154] DMAR: Mapping reserved region failed
[ 1.142215] DMAR: Setting identity map for device 0000:00:1a.0 [0xec000 -
0xeffff]
[ 1.142312] DMAR: Setting identity map for device 0000:00:1a.1 [0xec000 -
0xeffff]
[ 1.142408] DMAR: Setting identity map for device 0000:00:1a.2 [0xec000 -
0xeffff]
[ 1.142503] DMAR: Setting identity map for device 0000:00:1a.7 [0xec000 -
0xeffff]
[ 1.142598] DMAR: Setting identity map for device 0000:00:1d.0 [0xec000 -
0xeffff]
[ 1.142693] DMAR: Setting identity map for device 0000:00:1d.1 [0xec000 -
0xeffff]
[ 1.145483] DMAR: Setting identity map for device 0000:00:1d.2 [0xec000 -
0xeffff]
[ 1.145579] DMAR: Setting identity map for device 0000:00:1d.7 [0xec000 -
0xeffff]
[ 1.145668] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 1.145730] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 -
0xffffff]
[ 1.145958] DMAR: Intel(R) Virtualization Technology for Directed I/O
===================================
# cat /etc/modprobe.d/vfio_pci.conf
options vfio_pci disable_vga=1
install vfio_pci /root/sbin/vfio-pci-override-vga.sh
====================================
# cat /root/sbin/vfio-pci-override-vga.sh
#!/bin/sh
for i in $(find /sys/devices/pci* -name boot_vga); do
if [ $(cat $i) -eq 0 ]; then
GPU=$(dirname $i)
AUDIO=$(echo $GPU | sed -e "s/0$/1/")
echo "vfio-pci" > $GPU/driver_override
if [ -d $AUDIO ]; then
echo "vfio-pci" > $AUDIO/driver_override
fi
fi
if [ $(cat $i) -eq 1 ]; then
GPU=$(dirname $i)
AUDIO=$(echo $GPU | sed -e "s/0$/1/")
echo "nouveau" > $GPU/driver_override
if [ -d $AUDIO ]; then
echo "nouveau" > $AUDIO/driver_override
fi
fi
done
# All rights reserved :p
=======================================
# lspci -v |grep -i VGA
03:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 660]
(rev a1) (prog-if 00 [VGA controller])
Subsystem: eVga.com. Corp. Device 2662
Subsystem: eVga.com. Corp. Device 2662
04:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 660]
(rev a1) (prog-if 00 [VGA controller])
Subsystem: eVga.com. Corp. Device 2662
Subsystem: eVga.com. Corp. Device 2662
05:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 660]
(rev a1) (prog-if 00 [VGA controller])
Subsystem: eVga.com. Corp. Device 2662
Subsystem: eVga.com. Corp. Device 2662
========================================
# cat /sys/devices/pci0000:00/0000:00:03.0/0000:03:00.0/driver_override
nouveau
# cat /sys/devices/pci0000:00/0000:00:07.0/0000:04:00.0/driver_override
vfio-pci
=========================================
Thanks in advance, and I appreciate all the help to date,
Brian
_______________________________________________
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users
