Re: [vfio-users] Any advice how to minimize vfio device latency *without* virtual interrupt controller?

[Bronek Kozicki]

On Sat, 4 Jun 2022, at 4:37 PM, Roman Mamedov wrote:
> On Sat, 4 Jun 2022 16:25:32 +0100
> James Dutton  wrote:
>
>> I ran into problems some time ago, with UDP packets being dropped during
>> sustained disk accesses, so the sound device might also be affected by disk
>> accesses.
>
> Did you check to ensure that you're using iothreads?
> http://kvmonz.blogspot.com/p/knowledge-disk-performance-hints-tips.html
> And native aio as the article also suggests.

Thank you, my disks are configured for native IO.

My problem is different, how to shorten the interrupt handling path from PCI 
USB card to VM, without virtual interrupt handling. I suspect that my isolcpus 
and having all vCPU on the same node where USB card is attached might not be 
helping. It would be probably appropriate if I had virtual interrupt 
controller, but nope, there's no AVIC feature in EPYC 7003. I guess this means 
that interrupts must be handled by the host kernel and then routed to VM. Which 
host core is doing this handling?


B.

-- 
  Bronek Kozicki
  b...@incorrekt.com

___
vfio-users mailing list
vfio-users@redhat.com
https://listman.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] crackling audio in guest, after upgrade from Ivy Bridge to Epyc Milan

[Bronek Kozicki]

On Tue, 8 Mar 2022, at 7:35 PM, Bronek Kozicki wrote:
> On Tue, 8 Mar 2022, at 12:13 PM, Bronek Kozicki wrote:
>> On Mon, 7 Mar 2022, at 10:33 PM, Alex Williamson wrote:
. . .
>> Hello Alex - thanks a lot for your help!
>>
>> I had no idea about AVIC. I went through all the steps in the guide you 
>> pointed me to, but it seems my kernels does not want to enable it. I 
>> think I need your help figuring out why.
>>
>> The effective kvm_amd parameters are:
>>
>> root@gdansk /sys/module/kvm_amd/parameters # grep -E '*' *
>> avic:N
>> dump_invalid_vmcb:N
>> intercept_smi:Y
>> nested:0
>> npt:Y
>> nrips:1
>> pause_filter_count:3000
>> pause_filter_count_grow:2
>> pause_filter_count_max:65535
>> pause_filter_count_shrink:0
>> pause_filter_thresh:128
>> sev:N
>> sev_es:N
>> vgif:1
>> vls:1
>>
>
> FWIW My CPU is
>
> root@gdansk ~ # lscpu
> Architecture:x86_64
> CPU op-mode(s):  32-bit, 64-bit
> Address sizes:   48 bits physical, 48 bits virtual
> Byte Order:  Little Endian
> CPU(s):  96
> On-line CPU(s) list: 0-95
> Vendor ID:   AuthenticAMD
> BIOS Vendor ID:  Advanced Micro Devices, Inc.
> Model name:  AMD EPYC 7413 24-Core Processor
> BIOS Model name: AMD EPYC 7413 24-Core Processor
> 
> CPU family:  25
> Model:   1
> Thread(s) per core:  2
> Core(s) per socket:  24
> Socket(s):   2
> Stepping:1
> Frequency boost: enabled
> CPU max MHz: 3630.8101
> CPU min MHz: 1500.
> BogoMIPS:5300.12
> Flags:   fpu vme de pse tsc msr pae mce cx8 
> apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht 
> syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl 
> nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor 
> ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c 
> rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 
> 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb 
> bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate 
> ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid 
> cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec 
> xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero 
> irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock 
> nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter 
> pfthreshold v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes 
> vpclmulqdq rdpid overflow_recov succor smca
> Virtualization:  AMD-V
> L1d cache:   1.5 MiB (48 instances)
> L1i cache:   1.5 MiB (48 instances)
> L2 cache:24 MiB (48 instances)
> L3 cache:256 MiB (8 instances)
> NUMA node(s):2
> NUMA node0 CPU(s):   0-23,48-71
> NUMA node1 CPU(s):   24-47,72-95


OK so it seems avic is not listed in the CPU Flags which is puzzling, since it 
is almost 10 years old CPU feature and supported in kernels since 5.6 if I read 
that right. I have send a query to the motherboard manufacturer (Supermicro) in 
case it needs to be enabled in BIOS, or maybe conflicts with existing settings.


B.

-- 
  Bronek Kozicki
  b...@incorrekt.com

___
vfio-users mailing list
vfio-users@redhat.com
https://listman.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] crackling audio in guest, after upgrade from Ivy Bridge to Epyc Milan

[Bronek Kozicki]

On Tue, 8 Mar 2022, at 12:13 PM, Bronek Kozicki wrote:
> On Mon, 7 Mar 2022, at 10:33 PM, Alex Williamson wrote:
>> On Mon, 07 Mar 2022 22:13:01 +
>> "Bronek Kozicki"  wrote:
>>
>>> I know, this is such an old topic ...
>>> 
>>> I have today upgraded my hypervisor from intel Ivy-Bridge to AMD Epyc
>>> Milan and, after making the necessary adjustments in vfio
>>> configuration to make my virtual machines work again, I found that
>>> there's immense amount of crackling (like, many times per second) in
>>> audio, both in Linux and in Windows guests. The audio is USB DAC
>>> connected through USB, and USB is on passed-through PCI (dedicated
>>> for guest).
>>> 
>>> The host has two AMD 7413 CPUs, with lots of cores dedicated to
>>> guests:
>>> 
>>> video=efifb:off iommu=pt amd_iommu=on add_efi_memmap
>>> nohz_full=6-35,54-83 rcu_nocbs=6-35,54-83 isolcpus=6-35,54-83
>>> nvme.poll_queues=12 
>>>
>>
>> Getting interrupt bypass to work on AMD (AVIC) is rather more finicky
>> than the equivalent on Intel (APICv).  Have a look at this pointer and
>> see if you can get it working:
>>
>> https://forum.level1techs.com/t/svm-avic-iommu-avic-improves-interrupt-performance-on-zen-2-etc-based-processors/155226
>>
>> When working correctly, you should essentially stop seeing interrupt
>> counts increase in /proc/interrupts on the host for any vfio MSI
>> vectors.  Thanks,
>
>
> Hello Alex - thanks a lot for your help!
>
> I had no idea about AVIC. I went through all the steps in the guide you 
> pointed me to, but it seems my kernels does not want to enable it. I 
> think I need your help figuring out why.
>
> The effective kvm_amd parameters are:
>
> root@gdansk /sys/module/kvm_amd/parameters # grep -E '*' *
> avic:N
> dump_invalid_vmcb:N
> intercept_smi:Y
> nested:0
> npt:Y
> nrips:1
> pause_filter_count:3000
> pause_filter_count_grow:2
> pause_filter_count_max:65535
> pause_filter_count_shrink:0
> pause_filter_thresh:128
> sev:N
> sev_es:N
> vgif:1
> vls:1
>

FWIW My CPU is

root@gdansk ~ # lscpu
Architecture:x86_64
CPU op-mode(s):  32-bit, 64-bit
Address sizes:   48 bits physical, 48 bits virtual
Byte Order:  Little Endian
CPU(s):  96
On-line CPU(s) list: 0-95
Vendor ID:   AuthenticAMD
BIOS Vendor ID:  Advanced Micro Devices, Inc.
Model name:  AMD EPYC 7413 24-Core Processor
BIOS Model name: AMD EPYC 7413 24-Core Processor
CPU family:  25
Model:   1
Thread(s) per core:  2
Core(s) per socket:  24
Socket(s):   2
Stepping:1
Frequency boost: enabled
CPU max MHz: 3630.8101
CPU min MHz: 1500.
BogoMIPS:5300.12
Flags:   fpu vme de pse tsc msr pae mce cx8 apic sep 
mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext 
fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid 
extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 
sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic 
cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext 
perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 
invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 
smep bmi2 invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt 
xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero 
irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save 
tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold 
v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid 
overflow_recov succor smca
Virtualization:  AMD-V
L1d cache:   1.5 MiB (48 instances)
L1i cache:   1.5 MiB (48 instances)
L2 cache:24 MiB (48 instances)
L3 cache:256 MiB (8 instances)
NUMA node(s):2
NUMA node0 CPU(s):   0-23,48-71
NUMA node1 CPU(s):   24-47,72-95
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf:  Not affected
Vulnerability Mds:   Not affected
Vulnerability Meltdown:  Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled 
via prctl and seccomp
Vulnerability Spectre v1:Mitiga

Re: [vfio-users] crackling audio in guest, after upgrade from Ivy Bridge to Epyc Milan

[Bronek Kozicki]

On Mon, 7 Mar 2022, at 10:33 PM, Alex Williamson wrote:
> On Mon, 07 Mar 2022 22:13:01 +
> "Bronek Kozicki"  wrote:
>
>> I know, this is such an old topic ...
>> 
>> I have today upgraded my hypervisor from intel Ivy-Bridge to AMD Epyc
>> Milan and, after making the necessary adjustments in vfio
>> configuration to make my virtual machines work again, I found that
>> there's immense amount of crackling (like, many times per second) in
>> audio, both in Linux and in Windows guests. The audio is USB DAC
>> connected through USB, and USB is on passed-through PCI (dedicated
>> for guest).
>> 
>> The host has two AMD 7413 CPUs, with lots of cores dedicated to
>> guests:
>> 
>> video=efifb:off iommu=pt amd_iommu=on add_efi_memmap
>> nohz_full=6-35,54-83 rcu_nocbs=6-35,54-83 isolcpus=6-35,54-83
>> nvme.poll_queues=12 
>>
>
> Getting interrupt bypass to work on AMD (AVIC) is rather more finicky
> than the equivalent on Intel (APICv).  Have a look at this pointer and
> see if you can get it working:
>
> https://forum.level1techs.com/t/svm-avic-iommu-avic-improves-interrupt-performance-on-zen-2-etc-based-processors/155226
>
> When working correctly, you should essentially stop seeing interrupt
> counts increase in /proc/interrupts on the host for any vfio MSI
> vectors.  Thanks,


Hello Alex - thanks a lot for your help!

I had no idea about AVIC. I went through all the steps in the guide you pointed 
me to, but it seems my kernels does not want to enable it. I think I need your 
help figuring out why.

The effective kvm_amd parameters are:

root@gdansk /sys/module/kvm_amd/parameters # grep -E '*' *
avic:N
dump_invalid_vmcb:N
intercept_smi:Y
nested:0
npt:Y
nrips:1
pause_filter_count:3000
pause_filter_count_grow:2
pause_filter_count_max:65535
pause_filter_count_shrink:0
pause_filter_thresh:128
sev:N
sev_es:N
vgif:1
vls:1

Perhaps related, kvm effective parameters are:

root@gdansk /sys/module/kvm/parameters # grep -E '*' *  
enable_vmware_backdoor:N
flush_on_reuse:N
force_emulation_prefix:N
halt_poll_ns:20
halt_poll_ns_grow:2
halt_poll_ns_grow_start:1
halt_poll_ns_shrink:0
ignore_msrs:N
kvmclock_periodic_sync:Y
lapic_timer_advance_ns:-1
min_timer_period_us:200
mmio_caching:Y
mmu_audit:N
nx_huge_pages:N
nx_huge_pages_recovery_ratio:60
pi_inject_timer:1
report_ignored_msrs:Y
tdp_mmu:Y
tsc_tolerance_ppm:250
vector_hashing:Y

The parameters I wanted to apply are:

root@gdansk /etc/modprobe.d (git)-[master] # cat kvm.conf 
options kvm ignore_msrs=0
options kvm_amd nested=0 avic=1 npt=1

I cannot see any errors in dmesg, but this is logged when I look for SVM

root@gdansk /etc/modprobe.d (git)-[master] # dmesg -H -k --time-format=iso 
-L=always | grep SVM
2022-03-08T11:49:24,171231+00:00 SVM: kvm: Nested Paging enabled
2022-03-08T11:49:24,171257+00:00 SVM: Virtual VMLOAD VMSAVE supported
2022-03-08T11:49:24,171259+00:00 SVM: Virtual GIF supported

My kernel is 5.15.26 vanilla ; my kernel config is:

root@gdansk /etc (git)-[master] # zcat /proc/config.gz | grep -E "_KVM|_AMD"
CONFIG_X86_AMD_PLATFORM_DEVICE=y
CONFIG_KVM_GUEST=y
CONFIG_CPU_SUP_AMD=y
CONFIG_X86_MCE_AMD=y
CONFIG_PERF_EVENTS_AMD_POWER=m
CONFIG_PERF_EVENTS_AMD_UNCORE=y
CONFIG_MICROCODE_AMD=y
CONFIG_AMD_MEM_ENCRYPT=y
# CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT is not set
# CONFIG_AMD_NUMA is not set
# CONFIG_X86_AMD_FREQ_SENSITIVITY is not set
CONFIG_AMD_NB=y
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_IRQFD=y
CONFIG_HAVE_KVM_IRQ_ROUTING=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_HAVE_KVM_MSI=y
CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y
CONFIG_KVM_VFIO=y
CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT=y
CONFIG_KVM_COMPAT=y
CONFIG_HAVE_KVM_IRQ_BYPASS=y
CONFIG_HAVE_KVM_NO_POLL=y
CONFIG_KVM_XFER_TO_GUEST_WORK=y
CONFIG_HAVE_KVM_PM_NOTIFIER=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
# CONFIG_X86_SGX_KVM is not set
CONFIG_KVM_AMD=m
CONFIG_KVM_AMD_SEV=y
CONFIG_KVM_XEN=y
CONFIG_KVM_MMU_AUDIT=y
CONFIG_PATA_AMD=m
CONFIG_NET_VENDOR_AMD=y
CONFIG_AMD8111_ETH=m
CONFIG_AMD_XGBE=m
CONFIG_AMD_XGBE_DCB=y
CONFIG_AMD_XGBE_HAVE_ECC=y
CONFIG_AMD_PHY=m
CONFIG_HW_RANDOM_AMD=m
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD756_S4882=m
CONFIG_I2C_AMD8111=m
CONFIG_I2C_AMD_MP2=m
CONFIG_SPI_AMD=m
CONFIG_PTP_1588_CLOCK_KVM=m
CONFIG_PINCTRL_AMD=m
CONFIG_GPIO_AMDPT=m
CONFIG_GPIO_AMD_FCH=m
CONFIG_GPIO_AMD8111=m
CONFIG_AGP_AMD64=m
CONFIG_DRM_AMDGPU=m
CONFIG_DRM_AMDGPU_SI=y
CONFIG_DRM_AMDGPU_CIK=y
CONFIG_DRM_AMDGPU_USERPTR=y
CONFIG_DRM_AMD_ACP=y
CONFIG_DRM_AMD_DC=y
CONFIG_DRM_AMD_DC_DCN=y
CONFIG_DRM_AMD_DC_HDCP=y
CONFIG_DRM_AMD_DC_SI=y
# CONFIG_DRM_AMD_SECURE_DISPLAY is not set
CONFIG_HSA_AMD=y
CONFIG_HSA_AMD_SVM=y
CONFIG_DRM_I915_GVT_KVMGT=m
CONFIG_SND_SOC_AMD_ACP=m
CONFIG_SND_SOC_AMD_CZ_DA7219MX98357_MACH=m
CONFIG_SND_SOC_AMD_CZ_RT5645_MACH=m
CONFIG_SND_SOC_AMD_ACP3x=m
CONFIG_SND_SOC_AMD_RV_RT5682_MACH=m

[vfio-users] crackling audio in guest, after upgrade from Ivy Bridge to Epyc Milan

[Bronek Kozicki]

I know, this is such an old topic ...

I have today upgraded my hypervisor from intel Ivy-Bridge to AMD Epyc Milan 
and, after making the necessary adjustments in vfio configuration to make my 
virtual machines work again, I found that there's immense amount of crackling 
(like, many times per second) in audio, both in Linux and in Windows guests. 
The audio is USB DAC connected through USB, and USB is on passed-through PCI 
(dedicated for guest).

The host has two AMD 7413 CPUs, with lots of cores dedicated to guests:

video=efifb:off iommu=pt amd_iommu=on add_efi_memmap nohz_full=6-35,54-83 
rcu_nocbs=6-35,54-83 isolcpus=6-35,54-83 nvme.poll_queues=12 

Example guest:

 12
  1
  














  
  

  
  
/machine
  
  
hvm
/usr/share/ovmf/x64/OVMF_CODE.fd
/var/lib/libvirt/qemu/nvram/lipowa_VARS.fd
  
  




  
  
  

  
  

  







  

.
.
.

  



  
  
  


  
  
  


  
  
  


  
  
  


  
  
  


  
  
  


  
  
  


  
  
  


  
  


  


  

.
.
.

   
  
  

  
  
  


  
  

  
  
  


  
  

  
  
  


  
  

  
  
  


  
  

  
  
  


  


  
  /dev/urandom
  

  


Sound is passed through USB card, which is dedicated to guest. From guest this 
card looks like this:

05:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host 
Controller [1912:0014] (rev 03) (prog-if 30 [XHCI])
Physical Slot: 0-4
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- https://listman.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] NVIDIA error: Failed to initialize DMA. Failed to allocate push buffer

[Bronek Kozicki]

Yes, this was it!

Before making may more changes, I looked in /proc/iomem and found this:

1802100-20020ff : PCI Bus :80
  200-2000fff
  200-200120f : PCI Bus :81
200-2000fff : :81:00.0
  200-2257fff : efifb <<<< HERE, wrong module!
2001000-20011ff : :81:00.0
  2001000-20011ff : vfio-pci
2001200-2001203 : :81:00.2
  2001200-2001203 : vfio-pci
2001204-2001204 : :81:00.2
  2001204-2001204 : vfio-pci

I added "video=efifb:off"  to kernel options, also added "blacklist efifb" to 
modprobe.d and restarted; 

After restart (but before starting virtual machine), /proc/iomem (for this mem. 
range) was:

1802100-20020ff : PCI Bus :80
  200-200120f : PCI Bus :81
200-2000fff : :81:00.0
2001000-20011ff : :81:00.0
2001200-2001203 : :81:00.2
2001204-2001204 : :81:00.2

i.e. no modules listed.

After starting virtual machine (with GPU working - finally!), it is:

1802100-20020ff : PCI Bus :80
  200-200120f : PCI Bus :81
200-2000fff : :81:00.0
  200-2000fff : vfio-pci <<< HERE, correct module!
2001000-20011ff : :81:00.0
  2001000-20011ff : vfio-pci
2001200-2001203 : :81:00.2
  2001200-2001203 : vfio-pci
2001204-2001204 : :81:00.2
  2001204-2001204 : vfio-pci

Thanks and Happy New Year to you!


B.


On Thu, 30 Dec 2021, at 11:04 AM, Arjen wrote:
> On Thursday, December 30th, 2021 at 11:48, Bronek Kozicki 
>  wrote:
>
>> I think here is the strongest hint; the host dmesg is floodeed with messages 
>> "BAR 1: can't reserve"
>
> This sounds like you need to add video=efifb:off to the kernel 
> parameters.
> (Don't use a combined video=efifb:off,vesafb:off, but separate them if 
> you need them both.)
> Or release the EFI framebuffer before doing passthrough.
> Or make sure the system does not use this GPU during POST and boot.
> Or maybe some other work-around that is needed for passthrough of the 
> single GPU of the system?

-- 
  Bronek Kozicki
  b...@incorrekt.com

___
vfio-users mailing list
vfio-users@redhat.com
https://listman.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] NVIDIA error: Failed to initialize DMA. Failed to allocate push buffer

[Bronek Kozicki]

I think here is the strongest hint; the host dmesg is floodeed with messages 
"BAR 1: can't reserve"


2021-12-30T10:01:59.456992+ gauss.lan.incorrekt.net kernel: vfio-pci 
:81:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258
2021-12-30T10:01:59.457413+ gauss.lan.incorrekt.net kernel: vfio-pci 
:81:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
2021-12-30T10:01:59.457675+ gauss.lan.incorrekt.net kernel: vfio-pci 
:81:00.0: BAR 1: can't reserve [mem 0x200-0x2000fff 64bit pref]
2021-12-30T10:01:59.486586+ gauss.lan.incorrekt.net kernel: vfio-pci 
:81:00.1: enabling device ( -> 0002)
2021-12-30T10:01:59.546592+ gauss.lan.incorrekt.net kernel: vfio-pci 
:81:00.3: enabling device ( -> 0002)

. . .

2021-12-30T10:09:58.164738+ gauss.lan.incorrekt.net kernel: vfio-pci 
:81:00.0: BAR 1: can't reserve [mem 0x200-0x2000fff 64bit pref]
2021-12-30T10:09:58.164811+ gauss.lan.incorrekt.net kernel: vfio-pci 
:81:00.0: BAR 1: can't reserve [mem 0x200-0x2000fff 64bit pref]


I will try adjusting host BIOS options.


B.


On Thu, 30 Dec 2021, at 10:25 AM, Bronek Kozicki wrote:
> Some more information:
>
>
> 1. driver seem to be loading fine in guest
> bronekk@euclid:~$ sudo dmesg | grep -E "nvidia|0d:00"
> [0.810066] pci :0d:00.0: [10de:1eb1] type 00 class 0x03
> [0.814518] pci :0d:00.0: reg 0x10: [mem 0xc000-0xc0ff]
> [0.818518] pci :0d:00.0: reg 0x14: [mem 
> 0x10-0x100fff 64bit pref]
> [0.825110] pci :0d:00.0: reg 0x1c: [mem 
> 0x101000-0x1011ff 64bit pref]
> [0.829048] pci :0d:00.0: reg 0x24: [io  0x9000-0x907f]
> [0.834899] pci :0d:00.0: PME# supported from D0 D3hot D3cold
> [0.836042] pci :0d:00.1: [10de:10f8] type 00 class 0x040300
> [0.837841] pci :0d:00.1: reg 0x10: [mem 0xc100-0xc1003fff]
> [0.845020] pci :0d:00.2: [10de:1ad8] type 00 class 0x0c0330
> [0.847351] pci :0d:00.2: reg 0x10: [mem 
> 0x101200-0x101203 64bit pref]
> [0.854518] pci :0d:00.2: reg 0x1c: [mem 
> 0x101204-0x101204 64bit pref]
> [0.858820] pci :0d:00.2: PME# supported from D0 D3hot D3cold
> [0.862836] pci :0d:00.3: [10de:1ad9] type 00 class 0x0c8000
> [0.864838] pci :0d:00.3: reg 0x10: [mem 0xc1004000-0xc1004fff]
> [0.873964] pci :0d:00.3: PME# supported from D0 D3hot D3cold
> [0.932598] pci :0d:00.0: vgaarb: VGA device added: 
> decodes=io+mem,owns=none,locks=none 
>[0.934523] pci 
> :0d:00.0: vgaarb: bridge control possible   
> 
>[0.936134] pci :0d:00.0: vgaarb: setting as boot 
> device (VGA legacy resources not available) 
>  [1.440190] pci 
> :0d:00.1: D0 power state depends on :0d:00.0
> 
>[1.441170] pci :0d:00.2: D0 power state depends 
> on :0d:00.0 
>   [1.443582] pci 
> :0d:00.3: D0 power state depends on :0d:00.0
> [2.619525] xhci_hcd :0d:00.2: xHCI Host Controller
> [2.620624] xhci_hcd :0d:00.2: new USB bus registered, assigned 
> bus number 11
> [2.622792] xhci_hcd :0d:00.2: hcc params 0x0180ff05 hci version 
> 0x110 quirks 0x0010
> [2.672211] usb usb11: SerialNumber: :0d:00.2
> [2.676422] xhci_hcd :0d:00.2: xHCI Host Controller
> [2.677944] xhci_hcd :0d:00.2: new USB bus registered, assigned 
> bus number 12
> [2.681209] xhci_hcd :0d:00.2: Host supports USB 3.1 Enhanced 
> SuperSpeed
> [2.705956] usb usb12: SerialNumber: :0d:00.2
> [3.926249] nvidia: loading out-of-tree module taints kernel.
> [3.927118] nvidia: module license 'NVIDIA' taints kernel.
> [3.938804] nvidia: module verification failed: signature and/or 
> required key missing - tainting kernel
> [3.966693] nvidia-nvlink: Nvlink Core is being initialized, major 
> device number 249
> [3.971181] nvidia :0d:00.0: vgaarb: changed VGA decodes: 
> olddecodes=io+mem,decodes=none:owns=none
> [4.070078] nvidia-modeset: Loading NVIDIA Kernel Mode Setting 
> Driver for UNIX platforms  460.91.03  Fri Jul  2 05:43:38 UTC 2021
> [4.349705] [drm] [nvidia-drm] [GPU ID 0x0d00] Loading driver
> [4.352647] [drm] Initialized nvidia-drm 0.0.0 20160202 for 
> :0d

Re: [vfio-users] NVIDIA error: Failed to initialize DMA. Failed to allocate push buffer

[Bronek Kozicki]

[Quadro 
RTX 4000] [10de:1eb1] (rev a1)
81:00.1 Audio device [0403]: NVIDIA Corporation TU104 HD Audio Controller 
[10de:10f8] (rev a1)
81:00.2 USB controller [0c03]: NVIDIA Corporation TU104 USB 3.1 Host Controller 
[10de:1ad8] (rev a1)
81:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI 
Controller [10de:1ad9] (rev a1)

3. device mapping in libvirt:


  
  

  
  
  


  
  

  
  
  


  
  

  
  
  


  
  

  
  
  



4. something is definitely wrong inside the guest, since I am getting these:

[ 1236.179163] watchdog: BUG: soft lockup - CPU#12 stuck for 23s! [Xorg:2982]
[ 1236.179961] Modules linked in: hid_generic usbhid hid rfkill 
snd_hda_codec_hdmi snd_hda_intel snd_intel_dspcfg soundwire_intel 
soundwire_generic_allocation snd_soc_core ghash_clmulni_intel snd_compress 
soundwire_cadence nls_ascii snd_hda_codec nls_cp437 vfat fat aesni_intel 
snd_hda_core libaes snd_hwdep crypto_simd soundwire_bus cryptd nvidia_drm(POE) 
snd_pcm glue_helper snd_timer drm_kms_helper snd iTCO_wdt intel_pmc_bxt joydev 
iTCO_vendor_support sg serio_raw cec watchdog soundcore virtio_console 
virtio_balloon pcspkr evdev efi_pstore qemu_fw_cfg nvidia_modeset(POE) 
nvidia(POE) drm fuse configfs efivarfs virtio_rng rng_core ip_tables x_tables 
autofs4 ext4 crc16 mbcache jbd2 crc32c_generic sd_mod t10_pi sr_mod crc_t10dif 
cdrom crct10dif_generic ahci libahci xhci_pci libata xhci_hcd virtio_scsi 
virtio_net net_failover failover scsi_mod usbcore crct10dif_pclmul psmouse 
crct10dif_common crc32_pclmul crc32c_intel i2c_i801 virtio_pci lpc_ich 
i2c_smbus virtio_ring usb_common virtio but
 ton
[ 1236.189681] CPU: 12 PID: 2982 Comm: Xorg Tainted: P   OEL
5.10.0-10-amd64 #1 Debian 5.10.84-1
[ 1236.190725] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 
02/06/2015
[ 1236.191711] RIP: 0010:_nv032887rm+0x12/0x40 [nvidia]
[ 1236.192286] Code: d2 0e 31 c0 e8 af 7d 78 ff e8 ca 3c eb ff 31 c0 48 83 c4 
08 c3 0f 1f 00 48 83 ec 08 39 4a 10 76 17 48 8b 02 c1 e9 02 8b 04 88 <48> 83 c4 
08 c3 66 0f 1f 84 00 00 00 00 00 be 00 00 d5 09 bf 0a ad
[ 1236.194379] RSP: 0018:a9b840f6ba98 EFLAGS: 0256
[ 1236.194977] RAX: 164000a1 RBX: 0020 RCX: 
[ 1236.195804] RDX: 9995889fd0a0 RSI: 9995889fc008 RDI: 99958b67d008
[ 1236.196617] RBP: 999586b02a00 R08: 0020 R09: 
[ 1236.197425] R10: 9995889fc008 R11: 9995889fd0a0 R12: 
[ 1236.198217] R13:  R14:  R15: 9995889fc008
[ 1236.199011] FS:  7f7bcbbd6a40() GS:999cdfb0() 
knlGS:
[ 1236.199931] CS:  0010 DS:  ES:  CR0: 80050033
[ 1236.200587] CR2: 564dd482a3a8 CR3: 000102d80005 CR4: 00770ee0
[ 1236.201392] PKRU: 5554
[ 1236.201699] Call Trace:
[ 1236.202118]  ? _nv009235rm+0x1f1/0x230 [nvidia]
[ 1236.202763]  ? _nv036126rm+0x62/0x70 [nvidia]
[ 1236.203393]  ? _nv028825rm+0x46/0x4a0 [nvidia]
[ 1236.204041]  ? _nv009323rm+0x7b/0x90 [nvidia]
[ 1236.204667]  ? _nv009319rm+0xfb/0x4f0 [nvidia]
[ 1236.205302]  ? _nv037231rm+0xfd/0x180 [nvidia]
[ 1236.205939]  ? _nv034489rm+0x248/0x370 [nvidia]
[ 1236.206528]  ? _nv009448rm+0x3d/0x90 [nvidia]
[ 1236.207153]  ? _nv029075rm+0x14c/0x670 [nvidia]
[ 1236.207759]  ? _nv028910rm+0x520/0x900 [nvidia]
[ 1236.208378]  ? _nv002525rm+0x9/0x20 [nvidia]
[ 1236.208966]  ? _nv003517rm+0x1b/0x80 [nvidia]
[ 1236.209551]  ? _nv013021rm+0x6fe/0x770 [nvidia]
[ 1236.210149]  ? _nv038021rm+0xb3/0x150 [nvidia]
[ 1236.210736]  ? _nv038020rm+0x388/0x4e0 [nvidia]
[ 1236.211336]  ? _nv036312rm+0xbe/0x140 [nvidia]
[ 1236.211939]  ? _nv036313rm+0x42/0x70 [nvidia]
[ 1236.212525]  ? _nv008273rm+0x4b/0x90 [nvidia]
[ 1236.213117]  ? _nv000709rm+0x4ef/0x880 [nvidia]
[ 1236.213709]  ? rm_ioctl+0x54/0xb0 [nvidia]
[ 1236.214228]  ? nvidia_ioctl+0x66c/0x880 [nvidia]
[ 1236.214816]  ? nvidia_frontend_unlocked_ioctl+0x37/0x50 [nvidia]
[ 1236.215516]  ? __x64_sys_ioctl+0x83/0xb0
[ 1236.215972]  ? do_syscall_64+0x33/0x80
[ 1236.216405]  ? entry_SYSCALL_64_after_hwframe+0x44/0xa9

On Wed, 29 Dec 2021, at 11:16 PM, Bronek Kozicki wrote:
> Hi
>
> Hoping someone solved this one before.
>
> My host if Epyc Milan, running on Asrock ROMED8-2T, GPU is NVIDIA 
> Quadro RTX 4000, running on fresh Arch Linux install. The guest is 
> Debian 11 , with NVIDIA-460 drivers . I can see the drivers are 
> correctly loaded in the guest (with nvidia-smi), but Xorg fails to 
> initialize. The /var/log/Xorg.0.log tail is:
>
>
> [   254.714] (II) NVIDIA: Using 24576.00 MB of virtual memory for 
> indirect memory
> [   254.714] (II) NVIDIA: access.
> [   257.719] (EE) NVIDIA(GPU-0): Failed to initialize DMA.
> [   257.720] (EE) NVIDIA(0): Failed to allocate 

[vfio-users] NVIDIA error: Failed to initialize DMA. Failed to allocate push buffer

[Bronek Kozicki]

Hi

Hoping someone solved this one before.

My host if Epyc Milan, running on Asrock ROMED8-2T, GPU is NVIDIA Quadro RTX 
4000, running on fresh Arch Linux install. The guest is Debian 11 , with 
NVIDIA-460 drivers . I can see the drivers are correctly loaded in the guest 
(with nvidia-smi), but Xorg fails to initialize. The /var/log/Xorg.0.log tail 
is:


[   254.714] (II) NVIDIA: Using 24576.00 MB of virtual memory for indirect 
memory
[   254.714] (II) NVIDIA: access.
[   257.719] (EE) NVIDIA(GPU-0): Failed to initialize DMA.
[   257.720] (EE) NVIDIA(0): Failed to allocate push buffer
[   257.829] (EE) 
Fatal server error:
[   257.829] (EE) AddScreen/ScreenInit failed for driver 0
[   257.829] (EE) 
[   257.829] (EE) 
Please consult the The X.Org Foundation support 
 at http://wiki.x.org
 for help. 
[   257.829] (EE) Please also check the log file at "/var/log/Xorg.0.log" for 
additional information.
[   257.829] (EE) 
[   257.829] (EE) Server terminated with error (1). Closing log file.

I am running similar configuration (same card, also Debian 11 and nvidia-460 
drivers) on a different host, with an older Intel Xeon CPU. No problems there.

Any hints?


B.

-- 
  Bronek Kozicki
  b...@incorrekt.com

___
vfio-users mailing list
vfio-users@redhat.com
https://listman.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] NVIDIA has enabled GPU passthrough beta support for a Windows virtual machine on GeForce GPUs

[Bronek Kozicki]

On Wed, 31 Mar 2021, at 12:51 PM, Patrick O'Callaghan wrote:
> On Wed, 2021-03-31 at 10:38 +0100, Bronek Kozicki wrote:
> > Some good news, straight from the source:
> > 
> > https://nvidia.custhelp.com/app/answers/detail/a_id/5173/~/geforce-gpu-passthrough-for-windows-virtual-machine-%28beta%29
> 
> How is this different from what we've been doing up to now?

1. commitment from NVidia
2. no need to use  in features section

B.

-- 
  Bronek Kozicki
  b...@incorrekt.com

___
vfio-users mailing list
vfio-users@redhat.com
https://listman.redhat.com/mailman/listinfo/vfio-users

[vfio-users] NVIDIA has enabled GPU passthrough beta support for a Windows virtual machine on GeForce GPUs

[Bronek Kozicki]

Some good news, straight from the source:

https://nvidia.custhelp.com/app/answers/detail/a_id/5173/~/geforce-gpu-passthrough-for-windows-virtual-machine-%28beta%29

Now, if only I was able to buy a GeForce card ...


-- 
  Bronek Kozicki
  b...@incorrekt.com

___
vfio-users mailing list
vfio-users@redhat.com
https://listman.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Stuttering in VM and "ignored rdmsr: 0x122" on new kernels?

[Bronek Kozicki]

Thank you, that helped!

I checked that 0x122 corresponds with a new MSR introduced by Intel to disable 
TSX, called MSR_IA32_TSX_CTRL . I have also found that in kernel 5.5 there's 
been a series of changes to hard-disable TSX inside KVM guests with 
MSR_IA32_TSX_CTRL (merged to mainline in 
752272f16dd18f2cac58a583a8673c8e2fb93abb )

The funny thing is that my CPU is Ivy Bridge, i.e. the last generation *before* 
addition of TSX. Perhaps KVM does not check it, or perhaps it's my fault for 
using a relatively new microcode patch which somehow fools the kernel. Or 
perhaps the problem is elsewhere. Anyways, switching this option seem to have 
fixed the problem!


B.

On Sat, 31 Oct 2020, at 6:32 PM, sL1pKn07 SpinFlo wrote:
> El vie., 30 oct. 2020 a las 23:56, Bronek Kozicki
> () escribió:
> >
> > I recently tried upgrading from vanilla LTS kernel 5.4 to vanilla 5.8 and 
> > found that my virtual machines "stutter" when they did not previously. The 
> > hypervisor is configured with tickless kernel (NO_HZ_FULL) and physical 
> > cores are exclusively reserved for vcpus by means of isolcpus , nohz_full 
> > and rcu_nocbs. The user experience under kernel 5.4 is good, it's only 5.8 
> > when stuttering happens. This could be either just a brief "pop" when 
> > playing music, every few seconds, or actual short screen-freeze when 
> > playing games under Windows VM (I use GPU pass-through with Nvidia Quadro 
> > cards).
> >
> > This seem to correspond with messages appearing on hypervisor console which 
> > look like:
> >
> > [41426.188902] kvm [176116]: vcpu10, guest rIP: 0x7ff87a6a2aba ignored 
> > rdmsr: 0x122
> > [41426.202067] kvm [176116]: vcpu10, guest rIP: 0x7ff87a6a2aba ignored 
> > rdmsr: 0x122
> > [41426.218588] kvm [176116]: vcpu10, guest rIP: 0x7ff87b6d1d3a ignored 
> > rdmsr: 0x122
> > [41451.754213] kvm [1644425]: vcpu3, guest rIP: 0x7f5d4b442335 ignored 
> > rdmsr: 0x122
> > [41482.036754] kvm [176116]: vcpu2, guest rIP: 0x76f320ea ignored rdmsr: 
> > 0x122
> > [41482.181285] kvm [176116]: vcpu2, guest rIP: 0x76f320ea ignored rdmsr: 
> > 0x122
> > [41482.188947] kvm [176116]: vcpu2, guest rIP: 0x76d27f55 ignored rdmsr: 
> > 0x122
> >
> > There is quite a lot of it, but only logged by kernel 5.8 and not by 5.4 (I 
> > skipped 5.5 , 5.6 and 5.7).
[ cut ]

> 
> try with
> 
> └───╼  cat /etc/modprobe.d/kvm.conf
> options kvm ignore_msrs=0
> 
> greetings
>

-- 
  Bronek Kozicki
  b...@incorrekt.com


___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

[vfio-users] Stuttering in VM and "ignored rdmsr: 0x122" on new kernels?

[Bronek Kozicki]

I recently tried upgrading from vanilla LTS kernel 5.4 to vanilla 5.8 and found 
that my virtual machines "stutter" when they did not previously. The hypervisor 
is configured with tickless kernel (NO_HZ_FULL) and physical cores are 
exclusively reserved for vcpus by means of isolcpus , nohz_full and rcu_nocbs. 
The user experience under kernel 5.4 is good, it's only 5.8 when stuttering 
happens. This could be either just a brief "pop" when playing music, every few 
seconds, or actual short screen-freeze when playing games under Windows VM (I 
use GPU pass-through with Nvidia Quadro cards).

This seem to correspond with messages appearing on hypervisor console which 
look like:

[41426.188902] kvm [176116]: vcpu10, guest rIP: 0x7ff87a6a2aba ignored rdmsr: 
0x122
[41426.202067] kvm [176116]: vcpu10, guest rIP: 0x7ff87a6a2aba ignored rdmsr: 
0x122
[41426.218588] kvm [176116]: vcpu10, guest rIP: 0x7ff87b6d1d3a ignored rdmsr: 
0x122
[41451.754213] kvm [1644425]: vcpu3, guest rIP: 0x7f5d4b442335 ignored rdmsr: 
0x122
[41482.036754] kvm [176116]: vcpu2, guest rIP: 0x76f320ea ignored rdmsr: 0x122
[41482.181285] kvm [176116]: vcpu2, guest rIP: 0x76f320ea ignored rdmsr: 0x122
[41482.188947] kvm [176116]: vcpu2, guest rIP: 0x76d27f55 ignored rdmsr: 0x122

There is quite a lot of it, but only logged by kernel 5.8 and not by 5.4 (I 
skipped 5.5 , 5.6 and 5.7).

I build my own kernel roughly based on ArchLinux, but taking unaltered sources 
directly from kernel.org and my own configuration tweaks (such as 
CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y , MCORE2=y , PREEMPT_VOLUNTARY=y , 
NO_HZ_FULL=y , HZ_100=y , disable RCU_FAST_NO_HZ , disable X86_IOPL_IOPERM , 
disable USERFAULTFD , disable XEN, disable PAGE_POISONING , disable 
RAID6_PQ_BENCHMARK)

The hypervisor box has 2x Xeon 2697v2 (12 cores per CPU, two sockets, 48 
physical threads) so there is plenty of CPU power left for I/O etc, despite 12 
cores (24 threads) reserved for vcpus only: nohz_full=6-17,30-41 
rcu_nocbs=6-17,30-41 isolcpus=6-17,30-41 . There are two VMs using these 
reserved cores:

  12
  4
  

















  
  

  

... and:

  12
  4
  

















  
  

  https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] new video card. vm wont boot

[Bronek Kozicki]

; >>>>> vfio_iommu_type1, vfio_pci, vfio_virqfd to /etc/modules and 
> >>>>> update-initramfs -u
> >>>> Where did you get your kernel? Did you enable VFIO before building it?
> >>>>
> >>>>> On 5/29/20 8:41 PM, Roger Lawhorn wrote:
> >>>>>
> >>>>>> This appears to be the navi reset bug.
> >>>>>> Supposedly kernel 5.4 fixes this.
> >>>>>> However, I am not able to upgrade beyond kernel 5.3.
> >>>>>> Kernel 5.4 gets a raid error when trying to read my encryption 
> >>>>>> key and
> >>>>>> I cannot type my passphrase in order to get booted.
> >>>>>> Same with higher kernels.
> >>>>>> Alas.
> >>>>>> I can sit and watch qemu reset endlessly while trying to boot the 
> >>>>>> vm.
> >>>>>> On 5/28/20 1:02 PM, Roger Lawhorn wrote:
> >>>>>>
> >>>>>>> ok,
> >>>>>>> I removed all cards but the new one and turned off vfio.
> >>>>>>> Linux boots but finds no driver and uses fbdev instead:
> >>>>>>> $ inxi -G
> >>>>>>> Graphics:
> >>>>>>>    Device-1: AMD Navi 14 [Radeon RX 5500/5500M / Pro 5500M] 
> >>>>>>> driver: N/A
> >>>>>>>    Display: x11 server: X.Org 1.19.6 driver: ati,fbdev
> >>>>>>>    unloaded: modesetting,radeon,vesa resolution: 1024x768~76Hz
> >>>>>>>    OpenGL: renderer: llvmpipe (LLVM 9.0 128 bits) v: 3.3 Mesa 
> >>>>>>> 19.2.8
> >>>>>>> So at least its a working card.
> >>>>>>> It still wont work with either vm.
> >>>>>>> I am wondering if the nvidia gtx 980 ti oc (card for linux side) 
> >>>>>>> has
> >>>>>>> a hardware issue with the newer card.
> >>>>>>> Not compatible.
> >>>>>>> On 5/28/20 11:58 AM, Roger Lawhorn wrote:
> >>>>>>>
> >>>>>>>> I upgraded from a radeon duo pro 8gb hbm to a radeon rx 5500.
> >>>>>>>> Neither the win10 vm nor the win7 vm will boot with this card
> >>>>>>>> installed.
> >>>>>>>> The card is owned by vfio-pci on boot.
> >>>>>>>> The script didnt need any changes as it shows up at the same
> >>>>>>>> hardware address as the last card.
> >>>>>>>> Very frustrated.
> >>>>>>>> This card is so new it is listed as not supported till kernel 5.4.
> >>>>>>>> I am on kernel 5.3 and cannot get 5.4 to work.
> >>>>>>>> But that is for just trying to get linux to use it.
> >>>>>>>> Windows should just boot in vga mode and then look for a windows
> >>>>>>>> update.
> >>>>>>>> I did try removing -vga none to force qemu to emulate a vga driver
> >>>>>>>> long enough to boot.
> >>>>>>>> No dice. qemu wont pull up the normal vga window on the linux 
> >>>>>>>> side.
> >>>>>>>> Something has got to give.
> >>>>>>>> kind regards, Arjen
> >>
> >
> > ___
> > vfio-users mailing list
> > vfio-users@redhat.com
> > https://www.redhat.com/mailman/listinfo/vfio-users
> 
> ___
> vfio-users mailing list
> vfio-users@redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
>

-- 
  Bronek Kozicki
  b...@incorrekt.com


___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] new video card. vm wont boot

[Bronek Kozicki]

What is your distribution? It's all working for me:

root@gdansk ~ # zcat /proc/config.gz | grep VFIO 
CONFIG_KVM_VFIO=y
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO_VIRQFD=m
CONFIG_VFIO=m
# CONFIG_VFIO_NOIOMMU is not set
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y
CONFIG_VFIO_PCI_MMAP=y
CONFIG_VFIO_PCI_INTX=y
CONFIG_VFIO_PCI_IGD=y
CONFIG_VFIO_MDEV=m
CONFIG_VFIO_MDEV_DEVICE=m
root@gdansk ~ # uname -r
5.4.43-1-lts


B.

On Sun, 31 May 2020, at 4:45 AM, Roger Lawhorn wrote:
> ok, I got kernel 5.4 booted.
> They changed how raid0 works and I am one of the few that use raid0.
> 
> Anyway, vfio-pci is missing in kernel 5.4.
> Anyone know why?
> 
> 
> On 5/29/20 8:41 PM, Roger Lawhorn wrote:
> > This appears to be the navi reset bug.
> > Supposedly kernel 5.4 fixes this.
> > However, I am not able to upgrade beyond kernel 5.3.
> > Kernel 5.4 gets a raid error when trying to read my encryption key and 
> > I cannot type my passphrase in order to get booted.
> > Same with higher kernels.
> > Alas.
> >
> > I can sit and watch qemu reset endlessly while trying to boot the vm.
> >
> >
> >
> > On 5/28/20 1:02 PM, Roger Lawhorn wrote:
> >>
> >> ok,
> >>
> >> I removed all cards but the new one and turned off vfio.
> >> Linux boots but finds no driver and uses fbdev instead:
> >>
> >> $ inxi -G
> >> Graphics:
> >>   Device-1: AMD Navi 14 [Radeon RX 5500/5500M / Pro 5500M] driver: N/A
> >>   Display: x11 server: X.Org 1.19.6 driver: ati,fbdev
> >>   unloaded: modesetting,radeon,vesa resolution: 1024x768~76Hz
> >>   OpenGL: renderer: llvmpipe (LLVM 9.0 128 bits) v: 3.3 Mesa 19.2.8
> >>
> >>
> >> So at least its a working card.
> >>
> >> It still wont work with either vm.
> >>
> >> I am wondering if the nvidia gtx 980 ti oc (card for linux side) has 
> >> a hardware issue with the newer card.
> >> Not compatible.
> >>
> >>
> >> On 5/28/20 11:58 AM, Roger Lawhorn wrote:
> >>>
> >>>
> >>> I upgraded from a radeon duo pro 8gb hbm to a radeon rx 5500.
> >>>
> >>> Neither the win10 vm nor the win7 vm will boot with this card 
> >>> installed.
> >>> The card is owned by vfio-pci on boot.
> >>> The script didnt need any changes as it shows up at the same 
> >>> hardware address as the last card.
> >>>
> >>> Very frustrated.
> >>>
> >>> This card is so new it is listed as not supported till kernel 5.4.
> >>> I am on kernel 5.3 and cannot get 5.4 to work.
> >>> But that is for just trying to get linux to use it.
> >>>
> >>> Windows should just boot in vga mode and then look for a windows 
> >>> update.
> >>>
> >>> I did try removing -vga none to force qemu to emulate a vga driver 
> >>> long enough to boot.
> >>> No dice. qemu wont pull up the normal vga window on the linux side.
> >>>
> >>> Something has got to give.
> >>>
> >>> _______
> >>> vfio-users mailing list
> >>> vfio-users@redhat.com
> >>> https://www.redhat.com/mailman/listinfo/vfio-users
> >>>
> >>
> >> ___
> >> vfio-users mailing list
> >> vfio-users@redhat.com
> >> https://www.redhat.com/mailman/listinfo/vfio-users
> >
> > ___
> > vfio-users mailing list
> > vfio-users@redhat.com
> > https://www.redhat.com/mailman/listinfo/vfio-users
> 
> ___
> vfio-users mailing list
> vfio-users@redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
>

-- 
  Bronek Kozicki
  b...@incorrekt.com


___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] vfio not working with vanilla kernel 5.4.22

[Bronek Kozicki]

On Thu, 27 Feb 2020, at 11:35 AM, Bronek Kozicki wrote:
> On Mon, 24 Feb 2020, at 6:58 PM, Bronek Kozicki wrote:
> > On Mon, 24 Feb 2020, at 5:23 PM, Alex Williamson wrote:
> > > On Mon, 24 Feb 2020 10:40:39 +0000
> > > "Bronek Kozicki"  wrote:
> > > 
> > > > Heads up to anyone running the latest vanilla kernels - after upgrade
> > > > from 5.4.21 to 5.4.22 one of my VMs lost access to a vfio1
> > > > passed-through GPU. This was restored when I downgraded to 5.4.21 so
> > > > the problem seems related to some patch in version 5.4.22
> > > > 
> > > > Also, when starting the VM, I noticed the hypervisor log flooded with
> > > > messages "BAR 3: can't reserve" like:
> > > > 
> > > > Feb 24 09:49:38 gdansk.lan.incorrekt.net kernel: vfio-pci
> > > > :03:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258 Feb 24 09:49:38
> > > > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0:
> > > > vfio_ecap_init: hiding ecap 0x19@0x900 Feb 24 09:49:38
> > > > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: can't
> > > > reserve [mem 0xc000-0xc1ff 64bit pref] Feb 24 09:49:38
> > > > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: No more image
> > > > in the PCI ROM Feb 24 09:51:43 gdansk.lan.incorrekt.net kernel:
> > > > vfio-pci :03:00.0: BAR 3: can't reserve [mem
> > > > 0xc000-0xc1ff 64bit pref] Feb 24 09:51:43
> > > > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: can't
> > > > reserve [mem 0xc000-0xc1ff 64bit pref] Feb 24 09:51:43
> > > > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: can't
> > > > reserve [mem 0xc000-0xc1ff 64bit pref] Feb 24 09:51:43
> > > > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: can't
> > > > reserve [mem 0xc000-0xc1ff 64bit pref] Feb 24 09:51:43
> > > > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: can't
> > > > reserve [mem 0xc000-0xc1ff 64bit pref]
> > > > 
> > > > journalctl -b-2 | grep "vfio-pci :03:00.0: BAR 3: can't reserve"
> > > > | wc -l 2609
> > > > 
> > > > Finally, when shutting down the VM I observed kernel panic on the
> > > > hypervisor:
> > > > 
> > > > [  873.831301] Kernel panic - not syncing: Timeout: Not all CPUs
> > > > entered broadcast exception handler [  874.874008] Shutting down cpus
> > > > with NMI [  874.888189] Kernel Offset: 0x0 from 0x8100
> > > > (relocation range: 0x8000-0xbfff) [
> > > > 875.074319] Rebooting in 30 seconds..
> > > 
> > > Tried v5.4.22, not getting anything similar.  Potentially there's a
> > > driver activated in this kernel that wasn't previously on your system
> > > and it's attached itself to part of your device.  Look in /proc/iomem
> > > to see what it might be and disable it.  Thanks,
> > > 
> > > Alex 
> > 
> > 
> > Thank you Alex. One more thing which might be relevant: my system has 
> > two identical GPUs (Quadro  M5000), each in its own IOMMU group, and 
> > two VMs each using one of these GPUs. One of the VMs is Windows 10 and 
> > I think it is configured for MSI-X, the other is Ubuntu Biopic with 
> > stable nvidia drivers.
> > 
> > I will try to find more debugging information when I get home, but 
> > perhaps above will allow you to reproduce. 
> 
> Some more information
> 
> My system has 2 Xeon CPUs  E5-2667 v2, each with 8 cores and 16 threads 
> (total 32 threads over 2 sockets). The motherboard is Supermicro X9DA7. 
> Despite 2 GPUs attached, the machine is headless with ttyS0 for control 
> - both GPUs are dedicated for virtual machines.
> 
> There is 128GB of ECC RAM, shared between small number of VMs and ZFS 
> filesystems. 80GB is reserved in hugepages for the VMs, 20GB is 
> reserved for ZFS cache. The kernel options are my own and unlikely to 
> be very good (happy to take feedback); I use the same kernel package 
> for both hypervisor and for one of the virtual machines, so some kernel 
> options enabled only make sense in a VM. I need 
> CONFIG_PREEMPT_VOLUNTARY and CONFIG_TREE_RCU for ZFS, I do not care 
> about Xen, legacy hardware or some kernel debugging options (although 
> perhaps I should).
> 
> I did get more of that on my main computer (including dmesg logs, pcie 
> topology etc), but because of a kernel panic (same as seen

Re: [vfio-users] vfio not working with vanilla kernel 5.4.22

[Bronek Kozicki]

On Mon, 24 Feb 2020, at 6:58 PM, Bronek Kozicki wrote:
> On Mon, 24 Feb 2020, at 5:23 PM, Alex Williamson wrote:
> > On Mon, 24 Feb 2020 10:40:39 +0000
> > "Bronek Kozicki"  wrote:
> > 
> > > Heads up to anyone running the latest vanilla kernels - after upgrade
> > > from 5.4.21 to 5.4.22 one of my VMs lost access to a vfio1
> > > passed-through GPU. This was restored when I downgraded to 5.4.21 so
> > > the problem seems related to some patch in version 5.4.22
> > > 
> > > Also, when starting the VM, I noticed the hypervisor log flooded with
> > > messages "BAR 3: can't reserve" like:
> > > 
> > > Feb 24 09:49:38 gdansk.lan.incorrekt.net kernel: vfio-pci
> > > :03:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258 Feb 24 09:49:38
> > > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0:
> > > vfio_ecap_init: hiding ecap 0x19@0x900 Feb 24 09:49:38
> > > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: can't
> > > reserve [mem 0xc000-0xc1ff 64bit pref] Feb 24 09:49:38
> > > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: No more image
> > > in the PCI ROM Feb 24 09:51:43 gdansk.lan.incorrekt.net kernel:
> > > vfio-pci :03:00.0: BAR 3: can't reserve [mem
> > > 0xc000-0xc1ff 64bit pref] Feb 24 09:51:43
> > > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: can't
> > > reserve [mem 0xc000-0xc1ff 64bit pref] Feb 24 09:51:43
> > > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: can't
> > > reserve [mem 0xc000-0xc1ff 64bit pref] Feb 24 09:51:43
> > > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: can't
> > > reserve [mem 0xc000-0xc1ff 64bit pref] Feb 24 09:51:43
> > > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: can't
> > > reserve [mem 0xc000-0xc1ff 64bit pref]
> > > 
> > > journalctl -b-2 | grep "vfio-pci :03:00.0: BAR 3: can't reserve"
> > > | wc -l 2609
> > > 
> > > Finally, when shutting down the VM I observed kernel panic on the
> > > hypervisor:
> > > 
> > > [  873.831301] Kernel panic - not syncing: Timeout: Not all CPUs
> > > entered broadcast exception handler [  874.874008] Shutting down cpus
> > > with NMI [  874.888189] Kernel Offset: 0x0 from 0x8100
> > > (relocation range: 0x8000-0xbfff) [
> > > 875.074319] Rebooting in 30 seconds..
> > 
> > Tried v5.4.22, not getting anything similar.  Potentially there's a
> > driver activated in this kernel that wasn't previously on your system
> > and it's attached itself to part of your device.  Look in /proc/iomem
> > to see what it might be and disable it.  Thanks,
> > 
> > Alex 
> 
> 
> Thank you Alex. One more thing which might be relevant: my system has 
> two identical GPUs (Quadro  M5000), each in its own IOMMU group, and 
> two VMs each using one of these GPUs. One of the VMs is Windows 10 and 
> I think it is configured for MSI-X, the other is Ubuntu Biopic with 
> stable nvidia drivers.
> 
> I will try to find more debugging information when I get home, but 
> perhaps above will allow you to reproduce. 

Some more information

My system has 2 Xeon CPUs  E5-2667 v2, each with 8 cores and 16 threads (total 
32 threads over 2 sockets). The motherboard is Supermicro X9DA7. Despite 2 GPUs 
attached, the machine is headless with ttyS0 for control - both GPUs are 
dedicated for virtual machines.

There is 128GB of ECC RAM, shared between small number of VMs and ZFS 
filesystems. 80GB is reserved in hugepages for the VMs, 20GB is reserved for 
ZFS cache. The kernel options are my own and unlikely to be very good (happy to 
take feedback); I use the same kernel package for both hypervisor and for one 
of the virtual machines, so some kernel options enabled only make sense in a 
VM. I need CONFIG_PREEMPT_VOLUNTARY and CONFIG_TREE_RCU for ZFS, I do not care 
about Xen, legacy hardware or some kernel debugging options (although perhaps I 
should).

I did get more of that on my main computer (including dmesg logs, pcie topology 
etc), but because of a kernel panic (same as seen earlier, while trying to 
reproduce the bug) its root filesystem is currently not in a good state and I 
am unfortunately too busy at the moment to fix it and access this data. Will 
send more over the weekend, assuming that fixing my computer wont take very 
long.


B.

-- 
  Bronek Kozicki
  b...@spamcop.net


___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] vfio not working with vanilla kernel 5.4.22

[Bronek Kozicki]

On Mon, 24 Feb 2020, at 5:23 PM, Alex Williamson wrote:
> On Mon, 24 Feb 2020 10:40:39 +
> "Bronek Kozicki"  wrote:
> 
> > Heads up to anyone running the latest vanilla kernels - after upgrade
> > from 5.4.21 to 5.4.22 one of my VMs lost access to a vfio1
> > passed-through GPU. This was restored when I downgraded to 5.4.21 so
> > the problem seems related to some patch in version 5.4.22
> > 
> > Also, when starting the VM, I noticed the hypervisor log flooded with
> > messages "BAR 3: can't reserve" like:
> > 
> > Feb 24 09:49:38 gdansk.lan.incorrekt.net kernel: vfio-pci
> > :03:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258 Feb 24 09:49:38
> > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0:
> > vfio_ecap_init: hiding ecap 0x19@0x900 Feb 24 09:49:38
> > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: can't
> > reserve [mem 0xc000-0xc1ff 64bit pref] Feb 24 09:49:38
> > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: No more image
> > in the PCI ROM Feb 24 09:51:43 gdansk.lan.incorrekt.net kernel:
> > vfio-pci :03:00.0: BAR 3: can't reserve [mem
> > 0xc000-0xc1ff 64bit pref] Feb 24 09:51:43
> > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: can't
> > reserve [mem 0xc000-0xc1ff 64bit pref] Feb 24 09:51:43
> > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: can't
> > reserve [mem 0xc000-0xc1ff 64bit pref] Feb 24 09:51:43
> > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: can't
> > reserve [mem 0xc000-0xc1ff 64bit pref] Feb 24 09:51:43
> > gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: can't
> > reserve [mem 0xc000-0xc1ff 64bit pref]
> > 
> > journalctl -b-2 | grep "vfio-pci :03:00.0: BAR 3: can't reserve"
> > | wc -l 2609
> > 
> > Finally, when shutting down the VM I observed kernel panic on the
> > hypervisor:
> > 
> > [  873.831301] Kernel panic - not syncing: Timeout: Not all CPUs
> > entered broadcast exception handler [  874.874008] Shutting down cpus
> > with NMI [  874.888189] Kernel Offset: 0x0 from 0x8100
> > (relocation range: 0x8000-0xbfff) [
> > 875.074319] Rebooting in 30 seconds..
> 
> Tried v5.4.22, not getting anything similar.  Potentially there's a
> driver activated in this kernel that wasn't previously on your system
> and it's attached itself to part of your device.  Look in /proc/iomem
> to see what it might be and disable it.  Thanks,
> 
> Alex 


Thank you Alex. One more thing which might be relevant: my system has two 
identical GPUs (Quadro  M5000), each in its own IOMMU group, and two VMs each 
using one of these GPUs. One of the VMs is Windows 10 and I think it is 
configured for MSI-X, the other is Ubuntu Biopic with stable nvidia drivers.

I will try to find more debugging information when I get home, but perhaps 
above will allow you to reproduce. 


B.

-- 
  Bronek Kozicki
  b...@spamcop.net


___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

[vfio-users] vfio not working with vanilla kernel 5.4.22

[Bronek Kozicki]

Heads up to anyone running the latest vanilla kernels - after upgrade from 
5.4.21 to 5.4.22 one of my VMs lost access to a vfio1 passed-through GPU. This 
was restored when I downgraded to 5.4.21 so the problem seems related to some 
patch in version 5.4.22

Also, when starting the VM, I noticed the hypervisor log flooded with messages 
"BAR 3: can't reserve" like:

Feb 24 09:49:38 gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: 
vfio_ecap_init: hiding ecap 0x1e@0x258
Feb 24 09:49:38 gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: 
vfio_ecap_init: hiding ecap 0x19@0x900
Feb 24 09:49:38 gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: 
can't reserve [mem 0xc000-0xc1ff 64bit pref]
Feb 24 09:49:38 gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: No more 
image in the PCI ROM
Feb 24 09:51:43 gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: 
can't reserve [mem 0xc000-0xc1ff 64bit pref]
Feb 24 09:51:43 gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: 
can't reserve [mem 0xc000-0xc1ff 64bit pref]
Feb 24 09:51:43 gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: 
can't reserve [mem 0xc000-0xc1ff 64bit pref]
Feb 24 09:51:43 gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: 
can't reserve [mem 0xc000-0xc1ff 64bit pref]
Feb 24 09:51:43 gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: 
can't reserve [mem 0xc000-0xc1ff 64bit pref]

journalctl -b-2 | grep "vfio-pci :03:00.0: BAR 3: can't reserve" | wc -l
2609

Finally, when shutting down the VM I observed kernel panic on the hypervisor:

[  873.831301] Kernel panic - not syncing: Timeout: Not all CPUs entered 
broadcast exception handler
[  874.874008] Shutting down cpus with NMI
[  874.888189] Kernel Offset: 0x0 from 0x8100 (relocation range: 
0x8000-0xbfff)
[  875.074319] Rebooting in 30 seconds..


B.

-- 
  Bronek Kozicki
  b...@incorrekt.com


___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

[vfio-users] vfio not working with vanilla kernel 5.4.22

[Bronek Kozicki]

Heads up to anyone running the latest vanilla kernels - after upgrade from 
5.4.21 to 5.4.22 one of my VMs lost access to a vfio1 passed-through GPU. This 
was restored when I downgraded to 5.4.21 so the problem seems related to some 
patch in version 5.4.22

Also, when starting the VM, I noticed the hypervisor log flooded with messages 
"BAR 3: can't reserve" like:

Feb 24 09:49:38 gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: 
vfio_ecap_init: hiding ecap 0x1e@0x258
Feb 24 09:49:38 gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: 
vfio_ecap_init: hiding ecap 0x19@0x900
Feb 24 09:49:38 gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: 
can't reserve [mem 0xc000-0xc1ff 64bit pref]
Feb 24 09:49:38 gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: No more 
image in the PCI ROM
Feb 24 09:51:43 gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: 
can't reserve [mem 0xc000-0xc1ff 64bit pref]
Feb 24 09:51:43 gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: 
can't reserve [mem 0xc000-0xc1ff 64bit pref]
Feb 24 09:51:43 gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: 
can't reserve [mem 0xc000-0xc1ff 64bit pref]
Feb 24 09:51:43 gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: 
can't reserve [mem 0xc000-0xc1ff 64bit pref]
Feb 24 09:51:43 gdansk.lan.incorrekt.net kernel: vfio-pci :03:00.0: BAR 3: 
can't reserve [mem 0xc000-0xc1ff 64bit pref]

journalctl -b-2 | grep "vfio-pci :03:00.0: BAR 3: can't reserve" | wc -l
2609

Finally, when shutting down the VM I observed kernel panic on the hypervisor:

[  873.831301] Kernel panic - not syncing: Timeout: Not all CPUs entered 
broadcast exception handler
[  874.874008] Shutting down cpus with NMI
[  874.888189] Kernel Offset: 0x0 from 0x8100 (relocation range: 
0x8000-0xbfff)
[  875.074319] Rebooting in 30 seconds..


B.

-- 
  Bronek Kozicki
  b...@spamcop.net


___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] vga passthrough works but no grub screen for linux guest

[Bronek Kozicki]

On Sun, 30 Jun 2019, at 1:41 PM, Christoph Willing wrote:
> I have vga passthrough working with a linux guest running in qemu-3.1.0
> on a host with two Nvidia Quadro K600 cards and the nvidia binary driver
> 430.26 installed. A dedicated keyboard & mouse is also passed through
> successfully so, in general, I'm pretty happy with the setup but ...
> 
> when I use OVMF and run the guest without passthrough (just a regular
> qemu vm running on the host display) I see a grub screen to select a
> particular kernel, followed by the usual console output during bootup
> before X starts. However, when running with passthrough enabled, the
> guest vm's monitor (Samsung U28D590) shows nothing until X starts -
> there is no grub screen and no console output.
> 
> Before I had the nvidia binary driver installed in the guest, I had some
> console output while booting but still no grub screen.
> 
> Is there some way I can see everything including the grub screen, or is
> it normal not to see that in the guest?

It is normal on my Quadro M5000 - these cards are not supported as primary 
display with passthrough. This might be something to do with ROM, but I was not 
successful trying to investigate further. To compensate for this all my virtual 
machines have serial console, so I can do "virsh console"


B.

-- 
  Bronek Kozicki
  b...@spamcop.net

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Question about integrated GPU passthrough and initialization

[Bronek Kozicki]

On Fri, 24 May 2019, at 7:17 PM, Micah Morton wrote:
> I’ve been working with an Intel Chrome OS device to see if integrated
> GPU passthrough works. The device is a 7th Generation (Kaby Lake)
> Intel Core i5-7Y57 with HD Graphics 615. So no discrete GPU. I think
> iGPU passthrough should work on this device.
> 
> Initializing the graphics hardware has proven to be the trickiest part
> in all of this, and I have a question I thought someone on this list
> might be able to answer:
> 
> Why does vfio enforce that I _need_ a VGA rom to be available in order
> to boot the guest in legacy passthrough mode
> (https://github.com/qemu/qemu/blob/master/hw/vfio/pci-quirks.c#L1616)?
> The Intel device I’m working with normally doesn’t initialize the GPU
> at all until the kernel is running, at which point the i915 driver
> does the initialization. So there is never actually any VGA rom
> anywhere on the system at any time for me to grab. I suppose I maybe
> could get one for this hardware from Intel or the web somewhere, but
> seems like if the device can normally initialize the GPU hardware from
> the kernel driver then it wouldn’t be too unreasonable to pass through
> the GPU and let the guest (with the same i915 driver) initialize the
> GPU without using a VGA rom. Do you think it would be reasonable for
> me to find a way to patch qemu/vfio/SeaBIOS to allow for this
> scenario?
> 
> I’ve tried commenting out some blocks in this function
> (https://github.com/qemu/qemu/blob/master/hw/vfio/pci-quirks.c#L1557)
> and patching SeaBIOS to avoid doing any kind of graphics init stuff in
> the guest, and am still working through the debugging, but was
> wondering if this is a reasonable approach? Or something that has been
> done before?
> 
> Thanks,
> Micah
> 
> ___
> vfio-users mailing list
> vfio-users@redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
>

I don't know about iGPU but for nVidia Quadro cards which I am passing through, 
I do not have a Rom. I use "" element in the device definition, 
and it just works, with an obvious limitation that guest doesn't show any image 
onscreen until the system is fully started. Perhaps it also helps that all my 
virtual machines are UEFI, not sure.


B.


-- 
  Bronek Kozicki
  b...@incorrekt.com

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Proper way for systemd service to wait mdev gvt device initialization

[Bronek Kozicki]

On Sun, 19 May 2019, at 7:45 PM, Alex Ivanov wrote:
> Hello.
> What is the proper way to do that? I have a unit that creates gvt 
> device in the system
> 
> ExecStart = "sh -c 'echo a297db4a-f4c2-11e6-90f6-d3b88d6c9525 > 
> /sys/bus/pci/devices/:00:02.0/mdev_supported_types/i915-GVTg_V5_8/create'";
> ExecStop = "sh -c 'echo 1 > 
> /sys/bus/pci/devices/:00:02.0/a297db4a-f4c2-11e6-90f6-d3b88d6c9525/remove'";
> 
> Ideally I would to like to start this service when :00:02.0 device 
> appears in the system, but the problem is that 
> /sys/bus/pci/devices/:00:02.0/mdev_supported_types/ tree is 
> populated later, so my service will fail.


You might want to investigate writing udev rules. Here is example rule I wrote 
to run a systemd service whenever a USB is plugged in:

$ cat /etc/udev/rules.d/90-libvirt-usb.rules 
ACTION=="add", \
  SUBSYSTEM=="usb", \
  RUN+="/usr/bin/systemctl --no-block --runtime start 
attach-usb@$attr{busnum}:$attr{devnum}:$attr{idVendor}.service"

This is actually one line, as you can see with "\" being used to glue lines 
together


B.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] (good) working GPU for passthrough?

[Bronek Kozicki]

On Wed, 13 Feb 2019, at 8:32 PM, Bronek Kozicki wrote:
> On Wed, 13 Feb 2019, at 8:29 PM, Kyle Marek wrote:
. . .
> > 
> > Successfully resets on EFI VM start with the EFI BIOS update applied.
> > 
> 
> Niiice, where did you get it from?
> 

nvm, just spotted your email on the other threead!


B.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] (good) working GPU for passthrough?

[Bronek Kozicki]

On Wed, 13 Feb 2019, at 8:29 PM, Kyle Marek wrote:
> On 2/12/19 9:55 PM, Kyle Marek wrote:
> > On 2/11/19 9:18 PM, Alex Williamson wrote:
> >> On Mon, 11 Feb 2019 20:39:07 -0500
> >> Kyle Marek  wrote:
> >>
> >>> On 2/10/19 8:59 PM, Kash Pande wrote:
>  On 2019-02-10 5:25 p.m., Kyle Marek wrote:  
> > When I quit in the QEMU monitor, the image stays on the screen, and no
> > further host dmesg output is produced.
> >  
>  You must do a full reset in the guest.  
> >>> Hmmm... other cards (GTX 780, GTX 1060), are reset when quitting from
> >>> the monitor, so I don't have to rely on a graceful guest shutdown to
> >>> avoid rebooting my host.
> >> NVIDIA cards reset properly from a bus reset.
> >>  
> >>> Do I need to wait for a reset quirk to be implemented for my card to
> >>> reset on quit/boot? It shouldn't be a condition to usage that the guest
> >>> needs to gracefully shutdown for the card to not get stuck.
> >> Don't hold your breath.
> > Good news! Radeon VII resets on guest start, but quitting QEMU from the
> > monitor leaves the card rendering its BIOS character buffer (test case
> > is quitting during SeaBIOS).
> >
> > When do PCI bus resets happen?
> >
> > Hopefully it also resets on VM start if I ever get a BIOS update with an
> > EFI ROM.
> 
> Successfully resets on EFI VM start with the EFI BIOS update applied.
> 

Niiice, where did you get it from?


B.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

[vfio-users] anyone else experiencing problems with Xubuntu + lock screen under nvidia card?

[Bronek Kozicki]

I documented the problem in detail at 
https://bugs.launchpad.net/ubuntu/+source/xorg/+bug/1815429 where you can also 
see VM definition.  If you are able to reproduce this bug, please vote for the 
bugfix.

TL,DR:

if you are running GPU passthrough to nvidia card and running Xubuntu in the 
VM, then after installing nvidia drivers you will not be able to lock the 
screen. The workaround is to install xfwm4 version 4.13 from Mint repository.


B.



-- 
  Bronek Kozicki
  b...@spamcop.net

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] (good) working GPU for passthrough?

[Bronek Kozicki]

On Tue, 22 Jan 2019, at 10:51 PM, Laszlo Ersek wrote:
> On 01/18/19 21:28, Kash Pande wrote:
> > On 2019-01-18 12:20 p.m., Bronek Kozicki wrote:
> >> This is starting to make sense, going to try passthrough with q35 and
> >> UEFI now.
> >
> >
> > Good luck! Looks like you're on the right track.
> 
> Sorry, I've had zero time to follow this list in recent weeks (months?)
> 
> If it's of any help, let me dump a domain XML on you that works fine on
> my end, for assigning my GTX750.
> 
> I vaguely recall that even S3 suspend/resume worked with it. (I haven't
> booted this guest in a good while now; I set it up because I wanted to
> see if Q35 would work.)
> 
> I'll make some comments on it, below.

. . . 


this is great stuff, thank you for sharing!


B.


___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] (good) working GPU for passthrough?

[Bronek Kozicki]

On Fri, 18 Jan 2019, at 1:39 PM, Bronek Kozicki wrote:
> On Thu, 17 Jan 2019, at 10:39 PM, Kash Pande wrote:
> > On 2019-01-17 2:03 p.m., Tobias Geiger wrote:
> > > I tried a VEGA 64 - i was able to live with the ACS patch needed here
> > > (Z170 Chipset... not needed with the old HD7800, but whatever...) -
> > > but i couldn't stand the reset/FLR/Bug which forces you at least to
> > > suspend/resume the host when you want to reboot only the guest... 
> > 
> > 
> > AIUI, this does not happen using the Q35 machine type with proper PCIe
> > root complex heirarchy.
> > 
> 
> 
> I do not have great experience with "Q35 machine type with proper PCIe 
> root complex hierarchy". Selection of the chipset type Q35 in the virt-
> manager upon creation does not seem to work with UEFI type of firmware 
> (as recommended for vfio passthrough). Or perhaps I should have 
> downloaded a more recent Tianocore UEFI firmware for my virtual 
> machines?


So I did some more experiments and learned what I suppose everyone else knew 
already: if switching to SCSI for CDROM, make sure to change the type of the 
controller to VirtIO from the default LSI.

Also, the host machine must have EFI firmware for guest to boot from. For 
ArchLinux the appropriate package is ovmf (or ovmf-git from AUR, for 
adventurous). Setting nvram in /etc/libvirt/qemu.conf must point to the EFI 
binaries installed by this package. Other distributions also have this package 
https://pkgs.org/download/ovmf , but that's not news.

Initially, my attempts were failing because OVMF EFI is unable to boot from LSI 
SCSI controller. This is obviously unrelated to GPU passthrough, except for me 
changing the incorrect CDROM type from IDE to SCSI (I would not have that 
problem if I changed to SATA).

Comparing the .xml it is obvious that virt-manager actually constructs the PCI 
topology with PCIe root for q35, here is appropriate diff excerpt from two 
nearly identical machines, one created as i440fx and another as q35 (with the 
above extra steps).

-
+
+
+  
+  
+  
+
+
+  
+  
+  
+
+
+  
+  
+
+
+  
+  
+  
+
+
+  
+  
+  
+
+
+  
+  
+  
+ 

This is starting to make sense, going to try passthrough with q35 and UEFI now.


B.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] (good) working GPU for passthrough?

[Bronek Kozicki]

On Thu, 17 Jan 2019, at 10:39 PM, Kash Pande wrote:
> On 2019-01-17 2:03 p.m., Tobias Geiger wrote:
> > I tried a VEGA 64 - i was able to live with the ACS patch needed here
> > (Z170 Chipset... not needed with the old HD7800, but whatever...) -
> > but i couldn't stand the reset/FLR/Bug which forces you at least to
> > suspend/resume the host when you want to reboot only the guest... 
> 
> 
> AIUI, this does not happen using the Q35 machine type with proper PCIe
> root complex heirarchy.
> 


I do not have great experience with "Q35 machine type with proper PCIe root 
complex hierarchy". Selection of the chipset type Q35 in the virt-manager upon 
creation does not seem to work with UEFI type of firmware (as recommended for 
vfio passthroug). Or perhaps I should have downloaded a more recent Tianocore 
UEFI firmware for my virtual machines?


B.


-- 
  Bronek Kozicki
  b...@spamcop.net

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Installing Catalyst prevents VM from booting.

[Bronek Kozicki]

Huh. Good point. Admittedly you might want to reuse very similar VM for both 
Windows and Linux.

-- 
  Bronek Kozicki
  b...@spamcop.net

On Thu, 27 Dec 2018, at 8:25 PM, Brett Foster wrote:
> Wait, we're talking about Linux +ati drivers? Alright, then I have no 
> idea... but if it's windows, try the 440 and see if that helps.
> 
> Brett
> 
> Sent from my BlackBerry - the most secure mobile device
> 
>   Original Message  
> From: b...@spamcop.net
> Sent: December 27, 2018 12:21 PM
> To: fost...@edgeandvertex.org; erra...@yourstruly.sx; vfio-users@redhat.com
> Subject: Re: [vfio-users] Installing Catalyst prevents VM from booting.
> 
> Some Linuxes do not boot on 440FX (at least, not for me with booting 
> with UEFI, which is recommended for GPU passthrough).
> 
> 
> B.
> 
> 
> On Thu, 27 Dec 2018, at 8:18 PM, Brett Foster wrote:
> > Why use q35? I always thought that wasn't recommended. At least a couple 
> > years ago, that was my impression.
> > 
> > 
> > Sent from my BlackBerry - the most secure mobile device
> > 
> >   Original Message  
> > From: erra...@yourstruly.sx
> > Sent: December 27, 2018 12:12 PM
> > To: fost...@edgeandvertex.org; b...@spamcop.net
> > Cc: vfio-users@redhat.com
> > Subject: Re: [vfio-users] Installing Catalyst prevents VM from booting.
> > 
> > 
> > BTW I think the specific reason you want to separate your pcie endpoints 
> > on q35 from the root complex using a switch or up/downstream ports is 
> > due to the fact that the root complex has direct access to memory. I 
> > can't find anything more specific regarding why that might be 
> > problematic but I know that pcie addressing is different for devices 
> > downstream as opposed to being connected directly to the root complex. I 
> > arrived at the conclusion that the amd driver works now with q35 because 
> > changed the topology as per the advice I read somewhere regarding how it 
> > should be done. I'd like to know for sure but I don't really understand 
> > PCIe well enough and I think as far as the driver is concerned its up to 
> > a guess as to why one configuration works and the other doesn't (unless 
> > you have better luck getting the windows remote kernel debugging to work 
> > than I have.)
> > 
> > From: vfio-users-boun...@redhat.com  on 
> > behalf of Brett Foster 
> > Sent: Wednesday, December 26, 2018 11:24:48 AM
> > To: Bronek Kozicki
> > Cc: vfio-users
> > Subject: Re: [vfio-users] Installing Catalyst prevents VM from booting.
> > 
> > I have been using AMD via i440fx for like 5 years or so (across 
> > different hardware and graphics cards). The drivers that work are the 
> > ones that come via Windows Update. The drivers distributed by AMD do 
> > not. Not sure why since presumably they come from the same place. lol.
> > 
> > On Tue, Dec 25, 2018 at 9:51 AM Bronek Kozicki 
> > mailto:b...@spamcop.net>> wrote:
> > On Tue, 25 Dec 2018, at 4:21 PM, Paige Thompson wrote:
> > > Hey,
> > >
> > > From what I understand in general connecting devices straight to the
> > > root complex is problematic and they should be connected to a bridge. I
> > > was kind of wondering if anybody had any knowledge of this. I find the
> > > configuration really confusing. I still cant really tell how the
> > > topology works.
> > 
> > 
> > Hope Alex will shed some light on it when he's back from the holiday break.
> > 
> > 
> > --
> >   Bronek Kozicki
> >   b...@spamcop.net<mailto:b...@spamcop.net>
> > 
> > ___
> > vfio-users mailing list
> > vfio-users@redhat.com<mailto:vfio-users@redhat.com>
> > https://www.redhat.com/mailman/listinfo/vfio-users

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Installing Catalyst prevents VM from booting.

[Bronek Kozicki]

Some Linuxes do not boot on 440FX (at least, not for me with booting with UEFI, 
which is recommended for GPU passthrough).


B.


On Thu, 27 Dec 2018, at 8:18 PM, Brett Foster wrote:
> Why use q35? I always thought that wasn't recommended. At least a couple 
> years ago, that was my impression.
> 
> 
> Sent from my BlackBerry - the most secure mobile device
> 
>   Original Message  
> From: erra...@yourstruly.sx
> Sent: December 27, 2018 12:12 PM
> To: fost...@edgeandvertex.org; b...@spamcop.net
> Cc: vfio-users@redhat.com
> Subject: Re: [vfio-users] Installing Catalyst prevents VM from booting.
> 
> 
> BTW I think the specific reason you want to separate your pcie endpoints 
> on q35 from the root complex using a switch or up/downstream ports is 
> due to the fact that the root complex has direct access to memory. I 
> can't find anything more specific regarding why that might be 
> problematic but I know that pcie addressing is different for devices 
> downstream as opposed to being connected directly to the root complex. I 
> arrived at the conclusion that the amd driver works now with q35 because 
> changed the topology as per the advice I read somewhere regarding how it 
> should be done. I'd like to know for sure but I don't really understand 
> PCIe well enough and I think as far as the driver is concerned its up to 
> a guess as to why one configuration works and the other doesn't (unless 
> you have better luck getting the windows remote kernel debugging to work 
> than I have.)
> 
> From: vfio-users-boun...@redhat.com  on 
> behalf of Brett Foster 
> Sent: Wednesday, December 26, 2018 11:24:48 AM
> To: Bronek Kozicki
> Cc: vfio-users
> Subject: Re: [vfio-users] Installing Catalyst prevents VM from booting.
> 
> I have been using AMD via i440fx for like 5 years or so (across 
> different hardware and graphics cards). The drivers that work are the 
> ones that come via Windows Update. The drivers distributed by AMD do 
> not. Not sure why since presumably they come from the same place. lol.
> 
> On Tue, Dec 25, 2018 at 9:51 AM Bronek Kozicki 
> mailto:b...@spamcop.net>> wrote:
> On Tue, 25 Dec 2018, at 4:21 PM, Paige Thompson wrote:
> > Hey,
> >
> > From what I understand in general connecting devices straight to the
> > root complex is problematic and they should be connected to a bridge. I
> > was kind of wondering if anybody had any knowledge of this. I find the
> > configuration really confusing. I still cant really tell how the
> > topology works.
> 
> 
> Hope Alex will shed some light on it when he's back from the holiday break.
> 
> 
> --
>   Bronek Kozicki
>   b...@spamcop.net<mailto:b...@spamcop.net>
> 
> ___
> vfio-users mailing list
> vfio-users@redhat.com<mailto:vfio-users@redhat.com>
> https://www.redhat.com/mailman/listinfo/vfio-users

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] USB ASIO DAC buffer underruns on Windows guest

[Bronek Kozicki]

Have you tried DPC latency checker from 
https://www.thesycon.de/eng/latency_check.shtml ? It might be some "rogue" 
driver causing issues, on my computer I had to tweak nVidia a little bit to 
stop polling power all the time. Perhaps AMD is doing something similar?


B.

-- 
  Bronek Kozicki
  b...@incorrekt.com

On Tue, 25 Dec 2018, at 4:52 PM, Paige Thompson wrote:
> Hi, 
> 
> I have a very long standing problem that I've recently been trying to 
> root cause and document. As near as I can tell the problem is rooted 
> somewhere in Windows itself and/or the virtio drivers for Windows. I 
> believe it must be one of the windows virtio drivers because: 
> 
> - Both of my DACs work perfectly fine connected to a Windows laptop 
> running Windows on metal 
> - A Linux guest with the same VM configuration doesn't have this problem 
> when using either of my DACs to play back audio 
> - I've been able to eliminate the problem intermittently on Windows (It 
> was working fine yesterday until a large batch of Windows Updates were 
> installed.) 
> 
> Even so, I want to know who out there has tried getting a USB audio DAC 
> to work with a Windows Guest on KVM/QEMU? I'm using a USB3.0 pcie card 
> that is passed through to the guest, I haven't been able to get it to 
> work any other way really. 
> 
> I've sifted through everything I can possibly for KVM/QEMU to make this 
> work. In the process I even figured out how to get Q35 to work with my 
> AMD r9 390 card by virtue of wanting to see if it would solve my problem 
> however it did not. If I had some other USB audio devices to try I would 
> but I don't. It'd be nice to find out what others have done to get one 
> to work as well as it does on bare metal. I think I'm close but I'm not 
> sure. 
> 
> Here is my work log, with a description of what I've tried starting from 
> the top and towards the end in comments things I have tried changing 
> along the way: https://gitlab.com/snippets/1789874
> 
> Most recently I tried reinstalling Windows and using virtio-scsi as the 
> issue seemed to be related to disk I/O on the host. It was working 
> really well without any problems at all until windows installed more 
> updates. In previous attempts I addeed things like isolcpus, hugepages, 
> increasing the global, emulator, and iothread periods, forcing the timer 
> to tsc native, etc. I think all of these things in combination has 
> helped but I'm trying to elliminate the problem completely. Today I 
> tested it under OpenSuse on linux and that didn't suck at all; works 
> perfectly on a linux guest. 
> 
> 
> ___
> vfio-users mailing list
> vfio-users@redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Installing Catalyst prevents VM from booting.

[Bronek Kozicki]

On Tue, 25 Dec 2018, at 4:21 PM, Paige Thompson wrote:
> Hey, 
> 
> From what I understand in general connecting devices straight to the 
> root complex is problematic and they should be connected to a bridge. I 
> was kind of wondering if anybody had any knowledge of this. I find the 
> configuration really confusing. I still cant really tell how the 
> topology works. 


Hope Alex will shed some light on it when he's back from the holiday break.


-- 
  Bronek Kozicki
  b...@spamcop.net

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Installing Catalyst prevents VM from booting.

[Bronek Kozicki]

On Tue, 25 Dec 2018, at 12:57 PM, Paige Thompson wrote:
> Hi,
> 
> As you probably know there is a workaround to use i440fx instead of q35 
> to get the R9 390 working with VFIO. However, I have managed to get it 
> working with Q35 by adding a PCIe bridge as well as upstream and 
> downstream ports to the configuration. You've probably noticed that the 
> R9 390 card works fine with Q35 up until the point in which you install 
> the AMD drivers and then it starts crashing. As near as I can tell the 
> driver simply doesn't like it if you attach the video card directly to 
> the root complex.
> 
> Here is a link to the configuration that I am using: 
> https://gitlab.com/snippets/1788426#note_127191397
> qemu command-line rough translation: 
> https://gitlab.com/snippets/1788426#note_127194064
> For the entire work log :  https://gitlab.com/snippets/1788426
> 
> I'd also like to point out that this also works with a Linux guest as I 
> confirmed a moment ago. I never could get Linux to work with either 
> i440fx or q35 before I tried using a pcie bridge in q35. My 
> understanding of why Windows would work with i440fx is unclear but I 
> know that pcie devices essentially speak the same as pci devices and 
> there are no pcie devices on i440fx platforms; still whatever the amd 
> drivers check for is somehow circumvented in that particular 
> configuration.
> 
> I'm surprised nobody knew to suggest this, I sort of figured this out 
> based on something I read somewhere about how it's a bad idea to allow 
> devices to talk directly as discrete devices to the root complex anyway. 
> So, I gave this a try and low and behold it worked. It won't give you 
> any performance improvements, but q35 is a more accurate emulation of a 
> modern system and may help with other aspects of troubleshooting, it 
> just requires more topology complexity to be configured correctly.
> 
> Hope that helps,


HI Paige


thanks for sharing, this is interesting. FWIW I am currently using Ubuntu 
Bionic with vfio GPU passthrough but I have given up on AMD cards years ago and 
am using nVidia Quadro now, which does not have problems with 440FX.


B.

-- 
  Bronek Kozicki
  b...@spamcop.net

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Spectre, meltdown what's the performance implication for vga-passthrough gaming setups?

[Bronek Kozicki]

On Sun, 7 Jan 2018, at 12:00 PM, Patrick O'Callaghan wrote:
> On Sun, 2018-01-07 at 11:28 +, Ivan Volosyuk wrote:
> > From what I heard virtualization setups suffer the most from the 
> > vulnerabilities. After virtualized windows is patched as well as host linux 
> > what is the expected slowdown for gaming? I have pretty much zero 
> > understanding how much syscall/interrupt heavy qemu/kvm with vfio. 
> > Is it time to urgently switch to AMD CPU?
> 
> I have patched Linux host and patched Windows guest and haven't noticed
> anything so far. Games are unlikely to be severely slowed down as much
> of the work is done by the GPU with relatively few system calls.

I patched my host (kernel 4.14.12) , and I also have Windows 10 which 
self-updated  to https://support.microsoft.com/en-gb/help/4056892/ . While I 
can see that the PTI has been activated in the host:

% dmesg | grep isolation
[0.00] Kernel/User page tables isolation: enabled

... but I am not so sure about the guest systems. Anyway, with the patch active 
in the host the guest machines are just fine for gaming.


B.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] NVidia code 43 with Winddows 10 guest - does NVidia have another trick up their sleeve?

[Bronek Kozicki]

Perhaps disabling ROM and adding spice as "boot" video card will help. That's 
how I use my Quadro M5000 , apparently the only supported way of doing it. 
Turns out spice is useful for other purposes, even if its Windows display 
output is disabled.


B.


Please excuse brevity, typing on a phone.

  Original Message  
From: alex.william...@redhat.com
Sent: 4 January 2018 4:32 pm
To: robert.j.dahlst...@gmail.com
Cc: vfio-users@redhat.com
Subject: Re: [vfio-users] NVidia code 43 with Winddows 10 guest - does NVidia 
have another trick up their sleeve?

On Wed, 3 Jan 2018 17:28:39 -0600
Robert Dahlstrom  wrote:

> Hello, I have been trying to setup a Windows 10 guest on my Linux host,
> passing through an nvidia 1060. I've been having a very difficult time,
> getting the dreaded code 43. I haven't followed an individual guide, but
> have taken bits from https://wiki.archlinux.org/ind
> ex.php/PCI_passthrough_via_OVMF and Alex Williamson's when relevant among
> others.
> 
> I have tried just about every possible solution I've found, but have come
> up short.
> 
> One thing I'll point out right away: in my VM, I put two virtual disks. On
> one I installed windows, on the other I installed Debian 9. When booting
> into the linux guest OS, with the proprietary nvidia driver, **everything
> works**. When I boot into windows: code 43. Based on that, I think I have
> most of my stuff correctly setup, and I am just at the receiving end of
> nvidia treachery. I hope I'm wrong about that though.

Nobody else is reporting new treachery.

> Since both cards are nvidia cards, I use vfio.conf to prevent the nvidia
> driver from loading on the guest. The guest GPU is in an IOMMU group that
> consists solely of it, its HDMI audio device, and the Host Bridge/PCI
> Bridge devices. Both the video and audio devices are forwarded to the guest.
> 
> What I have tried:
> 
>  1) Seemingly every permutation of hidden kvm, vendor_id settings, hyperv
> settings, both i440fx and q35, BIOS and OVMF,

My suggestion for GeForce remains 440fx + OVMF.

>  2) My current situation has my guest GPU in the first PCIe slot, with my
> host GPU in the third. I think the first 2 slots share an IOMMU group,
> presumably for SLI or some such. This puts my guest GPU as the boot card,
> which is what the system BIOS and bootloader comes up on. I read that may
> cause trouble, and so I've tried swapping the cards.  This I don't believe
> works because there is a boatload of other stuff in the IOMMU group of the
> third slot (Group 0), such as my host SATA and USB controllers, that I
> can't forward.

Look for a BIOS update for your system and/or BIOS settings to enable
ACS, AMD added ACS support in one of the AGESA releases for Ryzen7
systems.  Using the host boot graphics device is very likely
contributing to your problems.  I would at a very minimum swap cards in
order to get a ROM dump from your GPU and use that rather than one from
techpowerup.

>  3) Passing disable_vga in vfio.conf, doesn't seem to do anything
> 
>  4) Setting MSI interrupt signaling in windows for both the video and audio
> devices.
> 
> What I have not tried:
> 
>  1) Switching which GPU I use for host and guest. If I can't use the 1060
> the whole thing is pointless. It may tell me if the boot GPU issue is the
> cause of my woe, however.

Yes, it very well could be.

>  2) I have read sometimes booting with the virtual graphics device (Spice +
> QXL) causes problems. Trying that, I have not been able to boot up at all
> without it. Presence of -nographics on qemu commandline seems to make no
> difference. When I blast the display from the config, the qemu process
> starts up and sits at 0% and never does anything, and I can't later remote
> desktop or ssh into the guest.

If you can't get a BIOS/UEFI splash screen out of the assigned
graphics, it's likely not going to work in the OS either.  You're
pretty much guaranteed a Code 43 so long as the emulated graphics are
present.

>  3) I have not yet successfully gone through the process of personally
> downloading the ROM from the card and using it. I have instead grabbed a
> ROM file from techpowerup.com, and verified that it has both BIOS and EFI
> portions. I can passthrough in linux regardless whether I use the ROM or
> not, and in Windows it makes no difference, code 43 for both. When I try to
> download the ROM myself I get the Invalid PCI ROM header signature error,
> and an input/output error.

That's because you're trying to dump the ROM from the boot graphics...

Thanks,
Alex

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Switch setup for a 1440p@96Hz DVI monitor

[Bronek Kozicki]

On Tue, 7 Nov 2017, at 06:27 PM, Tmplt wrote:
> My monitor only has a single dual-link DVI input, so instead of crawling
> behind the system and fibble with the cables I've been looking for a
> switch setup. Thus far, I've been very unsuccessful. No DVI-switches
> appears to support this bandwidth and my searches for a simple 2-to-1
> DisplayPort switch (which I could probably use with an adapter) have
> yielded no fruit.

I am using this switch
https://www.amazon.co.uk/StarTech-com-Port-Resolution-Switch-Audio/dp/B004NNYILM/ref=sr_1_1?ie=UTF8=1510083758=8-1=startech+SV231DVIUAHR
(
https://www.startech.com/uk/Server-Management/KVM-Switches/2-Port-High-Resolution-USB-DVI-Dual-Link-KVM-Switch-with-Audio~SV231DVIUAHR
, my display resolution is 2560x1600 ). It is expensive, I think
displayport one would have been cheaper. But it works.


B.


-- 
  Bronek Kozicki
  b...@spamcop.net

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Using a physical disk in QEMU

[Bronek Kozicki]

Note: only SAS drives should be used with expanders. SATA drivers are
unlikely to work well with expanders. For those, this card counts as "8
drives only". Which is IMO still good.

-- 
  Bronek Kozicki
  b...@spamcop.net

On Sun, 22 Oct 2017, at 01:18 AM, taii...@gmx.com wrote:
> Another option that would provide the best performance is attaching the 
> drive to a SR-IOV capable HBA's VF and then using VFIO to attach the VF 
> to your VM.
> 
> The LSI SAS 2308 series is the most affordable choice for this at around 
> $50-70 for an 8 internal port adapter (which can support up to 256 
> drives via expander cards)
> 
> ___
> vfio-users mailing list
> vfio-users@redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] setting up virtual disks

[Bronek Kozicki]

Here are relevant parts of my VM definition:


  
  



  
  
  
  
  


  
  
  
  

  
  
  
  


As you can see, I've setup the controller with 4 queues and 3 devices,
of which 2 are ZFS ZVOL and 1 is raw block device. I also have plenty of
spare CPUs (2 sockets, 16 cores, 32 threads), so I'm expecting that
having 4 queues will aid IO performance.


B.


-- 
  Bronek Kozicki
  b...@spamcop.net

On Tue, 28 Mar 2017, at 01:21 PM, Patrick O'Callaghan wrote:
> On Tue, 2017-03-28 at 11:47 +0200, Laszlo Ersek wrote:
> > I recommend the following setup:
> > 
> > - hard disk(s):  virtio-blk or virtio-scsi disk, as you prefer
> 
> I'm interested in why someone would prefer one over the other. Can you
> explain?
> 
> poc
> 
> ___
> vfio-users mailing list
> vfio-users@redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Speed tips requested

[Bronek Kozicki]

On 26/03/2017 15:31, Alex Williamson wrote:

On Sun, Mar 26, 2017 at 4:33 AM, Patrick O'Callaghan <p...@usb.ve
<mailto:p...@usb.ve>> wrote:

On Sun, 2017-03-26 at 10:58 +0100, Bronek Kozicki wrote:
> Assuming you use libvirt, make sure to use vCPU pinning. For disk access, 
try cache='writeback' io='threads'. If you switch to scsio-vfio, this will give 
you the ability to define queue length which might additionally improve IO. Also, 
try QCOW2 format for guest disk, it might enable some additional optimizations. 
However given you host seem to have little spare capacity, YMMV

Thanks. I'm already using CPU pinning as I said. The disk options are
both set to "hypervisor default" so I'll try changing them. I'd
configured the guest disk as 'raw' assuming that would be faster than
QCOW2 but I'll look into it.



Generally the recommendation is to use raw (not sparse), LVM, or a block
device for the best performance.  QCOW is never going to be as fast as
these at writing unused blocks since it needs to go out and allocate new
blocks from the underlying file system when this happens.


I am not going to argue with your experience here, only wanted to note 
that QCOW2 can be created with preallcation=falloc (or full, which is 
not very useful) which means there will no extra allocations in the 
rutime. Everything will be allocated at the moment of disk creation with 
qemu-img create -f qcow2 -o preallcation=falloc 




B.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Speed tips requested

[Bronek Kozicki]

Assuming you use libvirt, make sure to use vCPU pinning. For disk access, try 
cache='writeback' io='threads'. If you switch to scsio-vfio, this will give you 
the ability to define queue length which might additionally improve IO. Also, 
try QCOW2 format for guest disk, it might enable some additional optimizations. 
However given you host seem to have little spare capacity, YMMV


B.

  Original Message  
From: Patrick O'Callaghan
Sent: Sunday, 26 March 2017 09:25
To: vfio-users@redhat.com
Subject: [vfio-users] Speed tips requested

I'm running Windows 10 in a KVM/QEMU VM under Fedora 25. The host is a
16GB i3770 and I have 4 threads (2 cores) dedicated to the VM with CPU
pinning and 8GB hugepages. The passthrough GPU is an Nvidia Geforce GTX
1050.

The guest disk is a 100GB raw file under vfio. The host disk is a 1TB
Toshiba SATA 2.

This generally works quite well, except for two things:

1) KB events seem to overrun in games, i.e. there is stuttering and a
buzz when holding down a key for too long. Keyboard and mouse are
wireless Logitech hardware on a USB-2 port.

Initially I had configured USB-2 on the VM and in an effort to fix this
changed it to USB-3, which seemed to make it slightly faster but not go
away (nb: this is still on the same physical USB-2 port).

I then attempted to enable MSI as outlined in https://vfio.blogspot.co.
uk/2014/09/vfio-interrupts-and-how-to-coax-windows.html. After the USB-
3 change (see above) Windows now shows the USB-3 controller as using
MSI, however the Windows Nvidia drivers don't seem to support it, even
after registry editing. Note that the GPU hardware does support it
according to Linux ('lspci -v -s ').

2) There is quite frequent (every few seconds) visual stuttering --
pause then continue -- when moving rapidly through a rendered scene,
possibly caused by disk I/O.

I'm already using vfio. Would vfio-scsi make a difference? Should I be
thinking about a dedicated disk?

Thanks for any hints.

poc

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] I need your help!

[Bronek Kozicki]

Hi Paige





I noticed your -cpu option does not disable hypervisor; you might want
to try it, please see
http://vfio.blogspot.co.uk/2016/10/how-to-improve-performance-in-windows-7.html
for reference. On the example of my virtual machines, this translates to
qemu options:


-machine pc-i440fx-2.5,accel=kvm,usb=off,dump-guest-core=off,mem-
merge=off -cpu host,-
hypervisor,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000


Another thing, I make use of vCPU pinning in my libvirt configuration
and I think this would help you as well. The CPUs I use for virtual
machines are removed from Linux kernel scheduler, with "isolcpus"
option. FWIW my motherboard is SuperMicro X9DA7. And one more thing,
when setting up my VMs I took particular care to ensure that the
physical cores I pinned my vCPUs to, are all on the same socket and
directly attached to PCIe slots that I mapped to my VMs with vfio.




B.



--

  Bronek Kozicki

  b...@spamcop.net







On Thu, 23 Feb 2017, at 11:22 AM, Paige Thompson wrote:

> I have been running this setup since October and I have tried since
> this time to fix the same issues without any success either in terms
> of isolating the problem or fixing it. Here’s a lot of details/specs
> about the setup that I could think of off-hand to include:
>  



> https://gist.github.com/cloudkitsch/eefec59233269ace0c9707d0a57e0d5f
>  



> However before diving into that I want to merely point out that the
> problem I’m having is not specific to my graphics configuration which
> works just fine except that HDMI audio doesn’t work. Not really sure
> why, maybe because I’m running Windows 10 N? Don’t know. Seems like I
> remember HDMI audio working on my Radeon 5770 / Opteron setup that I
> had VFIO setup on quite a while ago.
>  



> I don’t really want HDMI audio either. I have a Yamaha MG10-XU mixer
> connected to a USB 3 PCIe card that is also passed through to the
> guest. It is a USB audio DAC. The problem I’m having is very
> specifically with this mixer however the mixer works just fine on
> other computers. It doesn’t matter what buffer size I’m using @
> 44.1khz (lowest sampling rate) I still have the same problem which is
> hard to explain because while I can reproduce it consistently I can’t
> understand the behavior. Asides from theories, I’m not really sure
> where I would start to test. I think that other people around me are
> more frustrated with it than I am but I would really like to first
> isolate the problem and provide my time (while available) to do what I
> can to eliminate the behavior or fix the problem.
>  



> I would like to suggest a few reasons why it might be happening,
> starting with the fact that the audio will start to skip when the load
> on the bare metal host increases or is saturated. This is not always
> the case, but I have not found a way to dedicate specific cores to the
> gust without having to do something really backwards with numactl to
> ensure nothing ever touches the cores that the guest is using. Maybe
> this is not worth trying anyway? It seems to suggest that the USB ASIO
> driver for the MG10-XU has issues with timing that are impacted by not
> enough CPU cycles.
>  



> It must be more than that though, because it seems to also cut out and
> skip relentlessly with mpeg (x264 specifically.) In a lot of games the
> sound tends to skip during cinematics and  some high bandwidth movies
> consistently cause this problem.
>  



> However, I can get audio playback up to 192khz FLAC working but the
> driver usually crashes after a while and usually requires the whole
> system to be pretty idle. The mixer driver doesn’t crash like that on
> other computers. Both the VGA card and USB card are situated in PCIe
> slots adjacent to the same CPU in the NUMA topology however I’m not
> making any changes to the NUMA configuration at runtime, I tried
> without finding that it made any difference.
>  



> I used to have another USB card but that one did the same thing.
> Everything else that I could possibly care about works fantastic,
> including the mouse responsiveness—I have a whole powered hub of USB
> devices including an xBox controller connected to the USB card all of
> which work great. Games work really well in 1080p@120hz. I would try
> to pass the mixer through VIA USB redirection (same way I’m doing the
> corsair PSU) but it wouldn’t work and I don’t imagine it would be much
> better. I’ve also tried getting my Scarlett 2i2 with it but I have the
> same kind of problems and it typically doesn’t work for more than 20
> seconds at a time.
>  



> Any thoughts or information you have to share would be appreciated.
> I’d be happy to try and debug this some more but I’m completely out of
> ideas of where to go. It’s an 

Re: [vfio-users] Stuttering audio when passing through USB controller

[Bronek Kozicki]

On 03/02/2017 19:53, Tmplt wrote:

I saw on an older thread here that passing a USB controller through and
using a USB soundcard through it solved that users audio issues. I tried
that myself by adding


  

  
  


to the .xml-file, but I can still hear frequent audio stutters akin to
my quoted thread. Am I passing the controller incorrectly? The device
I'm trying to play through is a FiiO e10k DAC which works without issue
on the host and on bare-metal Windows 8.1.



More likely your Windows guest is suffering from large latency, since 
most USB DACs have small buffers and require (relatively) precise timing 
of audio stream sent from the USB port. You can check this by comparing 
http://www.thesycon.de/eng/latency_check.shtml and 
http://www.resplendence.com/latencymon when running on guest vs. bare 
metal .


If you are running nvidia card, then disabling PowerMizer might help , 
see this thread 
https://www.native-instruments.com/forum/threads/solved-dropouts-cracks-pops-on-windows-7-and-nvidia-gfx-card.126080/ 



You may also want to disable hypervisor CPU policy, as documented by 
Alex here 
http://vfio.blogspot.co.uk/2016/10/how-to-improve-performance-in-windows-7.html 
. This setting worked on my Windows 10 guest to significantly reduce its 
latency, its applicability is obviously not limited to Windows 7 only



B.


___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] AMD and NVIDIA cards for QEMU PCI passthrough

[Bronek Kozicki]

  Actually with OVMF (rather than legacy BIOS and VGA) this should "just" work, assuming the card reset is working properly . It is not for most AMD cards, but for nVidia it should. The small complication here is that some guest OS-es do not allow straightforward installation on OVMF (eg Windows 7 and some Linux distributions, from my experience)B.From: Ivan DimitrovSent: Monday, 7 November 2016 09:34To: matt.jk3Cc: vfio-users@redhat.comSubject: Re: [vfio-users] AMD and NVIDIA cards for QEMU PCI passthroughHi, It might work but in my experience the device release from the VM is very iffy. Also most of the guides I have checked recommend to blacklist the guest VGA from the host. So practically speaking you can not use the guest card.  With Best Regards,Ivan Dimitrov
On 5 November 2016 at 16:34, matt.jk3  wrote:Hello

I'm preparing to setup my first Linux system running a VM with near native performance.
My CPU doesn't have an integrated GPU so I bought AMD for use with the host system and I'd like to use NVIDIA card on the guest. My question is: can I use NVIDIA on the "host" when I'm not running the VM?
NVIDIA is my main card and I'd prefer if AMD took over only when I'm passing the other to the VM.

Best Regards
Matt

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users



___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] vga passthrough on server boards

[Bronek Kozicki]

On 12/10/2016 18:04, Alex Williamson wrote:
> On Wed, Oct 12, 2016 at 10:51 AM, Ethan Thomas  > wrote:
>
> I've been using a SuperMicro X8DTH-iF for quite some time with no
> problems. However it's worth noting that with some generations of
> multi-cpu boards the PCI-E lanes and ram may be associated with a
> specific CPU, so you may need to adjust which slots and cores you
> associate with a particular VM for best performance.
>
>
> I would go so far as to say this is true of any modern multi-socket
> system, it's called NUMA, Non-Uniform Memory Access. You can use tools
> like lstopo to identify the locality of memory, devices, and
> processors. Using memory and CPU from the correct node is important for
> an VM, and an assigned device should be an extra pull towards the I/O
> local node.
‎
Hi Alex

Memory locality is one thing I totally forgot about when setting up my 
VMs. Do you have any example how to reserve huge pages on a specific 
node via sysctl, and refer to it later in libvirt configuration? 
Currently I only use this:

~ # cat /etc/sysctl.d/80-hugepages.conf
# Reserve this many 2MB pages for virtual machines
vm.nr_hugepages = 28000

~ # grep hugepages /etc/mtab
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0

~ # grep hugepages /etc/libvirt/qemu.conf | grep -vE "^#"
hugetlbfs_mount = "/dev/hugepages"

~ # virsh dumpxml lublin-vfio1 | head -11

lublin-vfio1
bc578734-6a43-4fda-9b19-e43225007a83
16777216
16777216







... which entirely ignores memory locality. TIA!


B.



___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] vga passthrough on server boards

[Bronek Kozicki]

On 12/10/2016 13:41, Jiri 'Ghormoon' Novak wrote:

Hi,

I'm considering rebuilding my setup from server components (mainly due
to ECC ram), but I can't find much information on vga passtrough on
server boards (supermicro, hp ...) to check if there are not issues eg.
with stupid IRQ mapping like desktop boards sometimes have.

is there any list of tested server boards? or can anyone recommend some
(preferably few years old generations, I'm gonna buy secondhand HW, new
servers are not in my reach)?


FWIW, SuperMicro X9DA7 works very nicely for me.


B.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Win10 Guest, stability is horrible and freeze quickly

[Bronek Kozicki]

  Perhaps you have something eating into RAM of the host. I had my host freezing when I forgot to limit memory utilization in the ZFS. It does not matter what eats into your host RAM - since qemu is just a regular userspace process, it's memory needs (ie those of the guest system) are not "prioritized" as more essential than any other userspace process. If your host does not have enough memory, then the more you allocate in qemu (or libvirt) to the guest system, the more probability that qemu will run out of RAM, effectively freezing the guest OS. At least, this is what it would appear to be (it might be simply swapping, but that slows down guest execution speed by orders of magnitude, so it is effectively frozen)B. From: BerillionsSent: Friday, 7 October 2016 20:00To: Jayme HowardCc: mar...@schrodt.org; vfio-usersSubject: Re: [vfio-users] Win10 Guest,	stability is horrible and freeze quicklyVery very very things. If i set more than 2GB Memory Ram in Virt-Manager for the Guest, i have artefacts and freeze. If I set 2GB like actually, the guest works perfectly.So i don't know if it will be good for gaming even if i set Windows swap to 10GB ...2016-10-07 19:43 GMT+02:00 Jayme Howard :Put the 960 into the one that goes 4x if you're that worried about it, but it shouldn't be a huge deal.  I don't know if the 960 can even saturate a 4x.  It might be able to, but since it's a lower performance card anyway, it shouldn't really matter.On Fri, Oct 7, 2016 at 12:36 PM, Berillions  wrote:I have two PCI-E 3.0 16x slot,The first with the 960 and the second for the 970. if the second slot is used, his bandwidth will not be 16x but 4x.The others slots are only PCI-E 3.0 1x ...
___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users




___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Host hard lockups

[Bronek Kozicki]

On 05/08/2016 22:26, Bronek Kozicki wrote:

I had something similar too, it was happening few times per month. And
then it stopped, but I do not remember what changed back then :( Could
be hardware change since I switched around that time from AMD to nVidia
Quadro, or could be software change as I upgraded to qemu 2.5 (but I see
you are running 4.6, so probably not this).


I meant to say I also upgraded my kernel to 4.1, then removed this part 
but stupidly left parentheses, obviously irrelevant



B.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Host hard lockups

[Bronek Kozicki]

On 05/08/2016 22:11, vfio wrote:

Hello everyone,

I've been running VGA passthrough with a Debian unstable host and a
Windows 10 guest for months now. Everything works perfectly, except that
the entire machine randomly freezes when the guest is running.

When a freeze happens, the guest immediately locks up. Sometimes, if
audio was playing, it goes into a short loop. Strangely, the host does
not usually freeze immediately; it takes a few seconds after the guest
has frozen. For example, my CPU monitor on the host will usually perform
a few more measurements before completely freezing, and the mouse cursor
on the host machine will continue working for a few seconds as well.

When the host freezes, not even the physical reset button on the machine
works. It requires a hard reset by holding the power button. There have
been a few times where the reset button worked. However, in one of these
instances, the host refused to boot after a reset, claiming to be unable
to initialize one of the USB buses. Sometimes the issue does not happen
for several days with multiple-hour sessions. Sometimes it happens
multiple times per day, possibly a few minutes after booting the guest.

No freeze leaves any traces in syslog.

These issues are very similar to those reported by Colin Godsey on this
list in May. While the conclusion of that thread seemed to be BIOS
firmware problems or "the Skylake freeze", I am using a Core i7-4960X
(Ivy Bridge-E) and an ASUS Rampage IV Extreme with the final BIOS
revision (4901 from 2014-06-18).



I had something similar too, it was happening few times per month. And 
then it stopped, but I do not remember what changed back then :( Could 
be hardware change since I switched around that time from AMD to nVidia 
Quadro, or could be software change as I upgraded to qemu 2.5 (but I see 
you are running 4.6, so probably not this). I also made changes in my 
system configuration to ensure it does not run out of RAM (it is also 
running ZFS, which is very greedy).



B.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] vfio fails Guest FreeBSD9.3 host Fedora 23

[Bronek Kozicki]

On 27/05/2016 19:29, Alex Williamson wrote:

I am using 1G huge pages and as i have 2 numa node configuration, i am
ensuring that VM memory is pinned to numazone 1( coz my bus is in numazone


but your VM is configured for 2MB-sized hugepages.


B.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Recommendation for LGA2011 board

[Bronek Kozicki]

On 23/05/2016 19:20, Brian Yglesias wrote:


I have another thread whereby I try and fail to get vfio/VT-d working 
on two separate lga1366 boards.  (Asus Rampage II Gene and 
GA-EX58-UD5; both due to broken BIOS).


I was considering a rampage III, also 1366/X58, which is reported to 
work, or an OEM system, bug given some of the issues with X58, I'm 
starting to feel like I should just pony up for a new platform.


However, since the GB board was also reported to work with Linux 
(albeit different kernel version from ), I'm worried about pulling the 
trigger and getting stuck with a 3rd incompatible board.


As such, I was hoping someone here could recommend something that is 
working for them right now in the 2011 size.


I'd like 3 x16 ports and dual lan, but those aren't deal breakers.  
I'd rather stay away from brand new flagship stuff, for reasons of 
cost.  that said, my main concern is that it can be made to work, 
without crafting glue and rubber bands. ;)




When buying a motherboard I recommend reading the manual, CPU and memory 
compatibility list, rather than relying on someone's opinion. Checking 
the FAQ for mentions of Linux (and in the context of this thread, VT-d) 
is also helpful. Having said that, I have SuperMicro X9DA7 which I can 
recommend, but this is dual-socket and EATX (i.e. rather large), also it 
does not support the current generation of Intel Xeons. You may want to 
look at any of those instead 
https://www.supermicro.nl/products/motherboard/Xeon3000/#2011 . Other 
trusted brands are http://www.asrockrack.com/general/products.asp#Server 
, http://b2b.gigabyte.com/products/list.aspx?cg=11=189=35=101 . I 
also have good experience with Tyan, but they do not seem to have a 
motherboard which would suit you.


Asus is typical consumer brand, they do not pay attention to enterprise 
features or even compatibility.



B.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Sound Options for Guest

[Bronek Kozicki]

On 02/05/2016 18:52, Brett Peckinpaugh wrote:
I was wondering what are some options for audio?  Currently using the 
monitor, but want to use my speakers for better quality.  The options 
I can think of are a PCIE soundcard or USB sound device.


Are there any better options?  Would a USB which is cheaper and not 
needed in the case blocking fans work well? Currently I am passing 
through a PCIE USB 3.0 card which is where I would connect it.



In this case external USB DAC will be your best bet - e.g. like SMSL M3, 
Audioengine D1 (note, I had clicking sound with this one), Focusrite 
Scarlett, FiiO E10K, Topping VX1, Audioquest Dragonfly, Shiit Modi 2 
etc. Some of these use "standard" drivers and USB protocols, hence do 
not need extra drivers (e.g. SMSL M3), some need extra drivers (e.g. 
Audioengine). Some have builtin headphones amplifier, some have volume 
regulation, some require additional power (more cables), just do your 
research.


More expensive devices utilise asynchronous USB protocol, which makes 
them more resilient to jitter (and also requires extra drivers). Also 
some support DSD (1bit format used for SACD records) e.g Teac UD-301. 
This does not mean that cheaper devices sound any worse - unless you 
have $1000 amplifier and headphones already (in which case I doubt you 
would ask here for advice).



B.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] [FEEDBACK NEEDED] Rewriting the Arch wiki article

[Bronek Kozicki]

  For me setting up networking with an existing bridge "just works", I wrote few days ago on this lis how I've set it up on my machine. Hint: I do not use virsh "networks" capabilities at all - none defined (undefined the default one) and none started. Just my, manually crafted bridge, explicitly used in VM definitions.B.From: Garland KeySent: Monday, 18 April 2016 22:21To: Nicolas Roy-Renaud; vfio-usersSubject: Re: [vfio-users] [FEEDBACK NEEDED] Rewriting the Arch wiki articleI'm an intermediate Linux user, so this this stuff can be complicated to me sometimes.  Right now I'm having trouble setting up a network bridge that virt-manager will recognize.  I've arrived at the conclusion that this simply isn't possible on Arch.  That said, I can't find any documentation on how to convince qemu to use an existing network bridge.  If you're willing, please add this information as well.  If you already know how, any pointers would be greatly appreciated.Best,GarlandOn Mon, Apr 18, 2016 at 5:14 PM, Garland Key  wrote:Please add what to do if you have two identical GPUs.  Here is exactly what is needed to make it work.- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  /etc/modprobe.d/vfio.conf    install vfio-pci /sbin/vfio-pci-override-vga.sh
    options vfio-pci disable_vga=1 allow_unsafe_interrupts=1- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - /sbin/vfio-pci-override-vga.sh    #!/bin/sh

        for i in $(find /sys/devices/pci* -name boot_vga); do
            if [ $(cat $i) -eq 0 ]; then
                    GPU=$(dirname $i)
                    AUDIO=$(echo $GPU | sed -e "s/0$/1/")
                   echo "vfio-pci" > $GPU/driver_override
                    if [ -d $AUDIO ]; then
                            echo "vfio-pci" > $AUDIO/driver_override
                    fi
            fi
    done

        modprobe -i vfio-pci- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
Add the following to /etc/mkinitcpio.conf and then run mkinitcpio -p linux    BINARIES="/usr/bin/find /usr/bin/dirname"    FILES="/sbin/vfio-pci-override-vga.sh"- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Tested Hardware:Motherboard: Asus Sabertooth X99CPU: Intel Core i7-5930KGPU 1: GIGABYTE GeForce GTX 970 4GB G1 Gaming OC EditionGPU 2: GIGABYTE GeForce GTX 970 4GB G1 Gaming OC EditionRAM: 32GB Corsair Dominator Platinum DDR4 (CMD16GX4M2A2666C15)On Tue, Apr 12, 2016 at 3:36 PM, Nicolas Roy-Renaud  wrote:
  

  
  
I'm currently planning a full rewrite of the
  article on Arch wiki about PCI passthroughs and, as per Arch
wiki guidelines, I'm supposed get the approval of other users before
undergoing such comlex edits. If anyone on this mailing list is an
Arch wiki collaborator or frequent user, I would appreciate if you
could give
  me some feedback on the planned structure and propose
additional sections or potential user mistakes to highlight. My
primary objective here is to make most of what's on Alex
Williamson's blog more straightforward and concise.

I've already rewritten the first two sections ("Prerequisites" and
"Setting up IOMMU"), and the rest of the article should essentially
follow the same basic structure and style. Replies here or on the
wiki's discussion page would be much appreciated.

-Nicolas
  

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users




___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Arch, Virt-manager and Network Manager [help needed]

[Bronek Kozicki]

On 16/04/2016 14:00, Garland Key wrote:

Thanks.  Does this have to be listed in a particular part of the virsh XML?

On Sat, Apr 16, 2016 at 8:34 AM, Philip Abernethy > wrote:

I've done the same. Bridge created with NM and used by the VM. You
should be able to just select the bridge device. That's the
resulting XML for me:


  default
  278badf4-d33e-493e-b604-083b3dcc2804
  
  





that was network definition, not virtual machine. I do not know why this 
is needed, I do not use those at all.



B.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Arch, Virt-manager and Network Manager [help needed]

[Bronek Kozicki]

On 16/04/2016 13:34, Philip Abernethy wrote:

I've done the same. Bridge created with NM and used by the VM. You
should be able to just select the bridge device. That's the resulting
XML for me:


  default
  278badf4-d33e-493e-b604-083b3dcc2804
  
  





Here is xml of my VM, using "br0" bridge I defined with netctl:


  
  
  
  
  function='0x0'/>




I expect it does not matter how a bridge is defined, as long as it 
works. In /etc/libvirt/qemu.conf I have the below line:

bridge_helper = "/usr/lib/qemu/qemu-bridge-helper"


B.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] [FEEDBACK NEEDED] Rewriting the Arch wiki article

[Bronek Kozicki]

On 12/04/2016 20:36, Nicolas Roy-Renaud wrote:

I've already rewritten the first two sections ("Prerequisites" and
"Setting up IOMMU"), and the rest of the article should essentially
follow the same basic structure and style. Replies here or on the wiki's
discussion page would be much appreciated.


Hope Alex can clear any misconception I am about to present below


1. I am using option iommu=pt, under impression this is expected to 
improve performance (my CPU is Xeon IvyBridge)


2. does PCI bridge have to be in a separate IOMMU group than 
passed-through device?


3. would be nice to provide hints for headless host. FWIW, I use 
combination of

3.1. kernel options:
console=ttyS0,115200N8R nomodest video=vesa:off video=efifb:off vga=normal
3.2.following line in /etc/modprobe.d/vfio.conf:
options vfio-pci disable_vga=1
3.3. large list of blacklisted modules (all framebuffers and nvidia and 
AMD drivers) in /etc/modprobe.d/blacklist.conf:
# This host is headless, prevent any modules from attaching to video 
hardware

# NVIDIA
blacklist nouveau
blacklist nvidia
# AMD
blacklist radeon
blacklist amdgpu
blacklist amdkfd
blacklist fglrx
# HDMI sound on a GPU
blacklist snd_hda_intel
# Framebuffers (ALL of them)
blacklist vesafb
blacklist aty128fb
blacklist atyfb
blacklist radeonfb
blacklist cirrusfb
blacklist cyber2000fb
blacklist cyblafb
blacklist gx1fb
blacklist hgafb
blacklist i810fb
blacklist intelfb
blacklist kyrofb
blacklist lxfb
blacklist matroxfb_base
blacklist neofb
blacklist nvidiafb
blacklist pm2fb
blacklist rivafb
blacklist s1d13xxxfb
blacklist savagefb
blacklist sisfb
blacklist sstfb
blacklist tdfxfb
blacklist tridentfb
blacklist vfb
blacklist viafb
blacklist vt8623fb
blacklist udlfb

4. ignore_msrs=1 also helps running Linux guests

5. do not use qemu:arg for binding host device to guest, here is example 
how to do it properly:


  
  

  
  function='0x0'/>


5.1. for nVidia Quadro, add just below 


6. if guest is started from BIOS rather than UEFI, keep the above 
 but replace emulator with a script, e.g.

# virsh dumpxml gdynia-vfio1 | grep emulator
/usr/bin/qemu-system-x86_64.xvga.sh
# cat /usr/bin/qemu-system-x86_64.xvga.sh
#!/bin/sh
exec nice --adjustment=-5 /usr/bin/qemu-system-x86_64 `echo "$@" | \
sed 's/-device vfio-pci,host=82:00.0/-device 
vfio-pci,host=82:00.0,x-vga=on/g' | \
sed 's/-device vfio-pci,host=03:00.0/-device 
vfio-pci,host=03:00.0,x-vga=on/g'`



7. performance optimizations
7.1. use huge pages
7.2. use isolcpus
7.3. use vCPU pinnig
7.4. use virtio-scsi with multiple queues (depending on number of 
available CPUs, after removing these dedicated to guest)

7.5. use multiple queues for virtio-net
7.6. for Linux  guests, use P9 for mounting host filesystems in guest




B.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] vfio PCI passthrough NVIDIA K420 GPU problems

[Bronek Kozicki]

On 24/02/2016 20:26, Bronek Kozicki wrote:

On 24/02/2016 15:41, Daniel Pocock wrote:



Hi,

I'm trying to use PCI passthrough to give an NVIDIA GPU to a VM with
qemu / KVM.  I've summarized my environment below and the error I get is
near the bottom.  Any help would be appreciated.

There are a few guides I've been referring to already:
https://wiki.debian.org/VGAPassthrough
https://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
https://bbs.archlinux.org/viewtopic.php?id=162768
http://www.linux-kvm.org/page/VGA_device_assignment



Hi Daniel


I'm successfully passing through two Quadro M5000 (to two instances of
Windows 10), looking at
http://us.download.nvidia.com/Windows/Quadro_Certified/361.91/361.91-win10-quadro-grid-release-notes.pdf
(page 9) it is not obvious that this would work for either your K420 or
mine M5000. One gotcha - when starting Windows I do not see boot screen
at all, only Windows logon scren after nVidia drivers had loaded.
Explanation is in Alex's email sent to this list on 6th Feb 2016,
subject "No boot screen on Quadro M5000?" - basically passed through
Quadro cards are meant to be secondary  only (I use them as only card,
though). However this small quirk aside, this works for me very well.
Here are details of my setup:

* distribution Arch , with packages : qemu 2.5.0 libvirt 1.3.1 (I do not
use qemu command line directly)

* my machine has 32 logical cores (2 x 8 core Xeon E5-26xx CPUs, with
HT) and 128GB ECC RAM, two Quadro M5000 cards and two USB 3.0 controller
cards; my motherboard is Supermicro X9DA7. I also dedicated half of
cores and about half of RAM to guests (I'm also running Linux virtual
machines with spice). Obviously you do not need this much hardware, just
a context to help you understand my configuration.

* I am passing through one GPU and one USB controller for each of two
Windows 10 guests (my Linux guests are running without passthrough at
this time, although also I experimented with such setup)

* kernel 4.1.17 (vanilla, no patches at all)

* useful bits in /etc :
   bronek@gdansk /etc % cat modprobe.d/blacklist.conf
   blacklist nouveau
   blacklist snd_hda_intel

   bronek@gdansk /etc % cat modprobe.d/kvm.conf
   options kvm ignore_msrs=1

   bronek@gdansk /etc % cat modprobe.d/vfio.conf
   options vfio-pci ids=1912:0014,10de:13f0,10de:0fbb

   bronek@gdansk /etc % cat sysctl.d/80-hugepages.conf
   # Reserve this many 2MB pages for virtual machine
   vm.nr_hugepages = 28000

   bronek@gdansk /etc % grep -E "^MODULES" /etc/mkinitcpio.conf
   MODULES="vfio vfio_iommu_type1 vfio_pci vfio_virqfd vhost_net bridge
mpt2sas nvme"

   bronek@gdansk /etc % sed 's|192.168.[0-9.]*|192.168.#.#|g' netctl/br0
   Description="Bridge connection"
   Interface=br0
   Connection=bridge
   BindsToInterfaces=(enp5s0f1)
   IP=static
   Address=('192.168.#.#/24')
   ## Do not want default gateway on this interface
   # Gateway='192.168.#.#'
   Routes=('192.168.#.#/24 via 192.168.#.#')
   DNS=('192.168.#.#' '192.168.#.#')
   NETCTL_DEBUG=yes
   ## Ignore (R)STP and immediately activate the bridge
   SkbpForwardingDelay=yes

   bronek@gdansk /etc % lspci -tvnn | less
   . . .
  +-[:80]-+-01.0-[81]--
  |   +-02.0-[82]--+-00.0  NVIDIA Corporation GM204GL [Quadro
M5000] [10de:13f0]
  |   |\-00.1  NVIDIA Corporation GM204 High
Definition Audio Controller [10de:0fbb]
  |   +-03.0-[83]00.0  Intel Corporation PCIe Data Center
SSD [8086:0953]
  |   +-03.2-[84]00.0  Renesas Technology Corp. uPD720201
USB 3.0 Host Controller [1912:0014]
   . . .
  \-[:00]-+-00.0  Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7
DMI2 [8086:0e00]
  +-01.0-[01]00.0  LSI Logic / Symbios Logic SAS2308
PCI-Express Fusion-MPT SAS-2 [1000:0086]
  +-02.0-[02]00.0  Renesas Technology Corp. uPD720201
USB 3.0 Host Controller [1912:0014]
  +-03.0-[03]--+-00.0  NVIDIA Corporation GM204GL [Quadro
M5000] [10de:13f0]
  |\-00.1  NVIDIA Corporation GM204 High
Definition Audio Controller [10de:0fbb]

* boot configuration (i.e. kernel command line parameters):
console=ttyS0,115200N8R nomodest nohz=off udev.children-max=32
edac_core.edac_mc_panic_on_ue=1 intel_iommu=on iommu=pt
isolcpus=4,5,6,7,12,13,14,15,20,21,22,23,28,29,30,31
(I removed parts relevant to ZFS only, which I also use)

* my guests are using UEFI boot in kvm (my host is starting from
"legacy" BIOS, though), using edk2.git-ovmf-x64-... from
https://www.kraxel.org/repos/jenkins/edk2/

* for Windows installation (and possibnly boot-level maintenance in the
future) I use libvirt profile like attached "-spice" profile. I start
and connect to this from an old laptop running Linux.

* also, as you will see in attached profiles, I am using virtio-scsi.
This means that at the 

Re: [vfio-users] vfio PCI passthrough NVIDIA K420 GPU problems

[Bronek Kozicki]

On 24/02/2016 15:41, Daniel Pocock wrote:



Hi,

I'm trying to use PCI passthrough to give an NVIDIA GPU to a VM with
qemu / KVM.  I've summarized my environment below and the error I get is
near the bottom.  Any help would be appreciated.

There are a few guides I've been referring to already:
https://wiki.debian.org/VGAPassthrough
https://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/
https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
https://bbs.archlinux.org/viewtopic.php?id=162768
http://www.linux-kvm.org/page/VGA_device_assignment



Hi Daniel


I'm successfully passing through two Quadro M5000 (to two instances of 
Windows 10), looking at 
http://us.download.nvidia.com/Windows/Quadro_Certified/361.91/361.91-win10-quadro-grid-release-notes.pdf 
(page 9) it is not obvious that this would work for either your K420 or 
mine M5000. One gotcha - when starting Windows I do not see boot screen 
at all, only Windows logon scren after nVidia drivers had loaded. 
Explanation is in Alex's email sent to this list on 6th Feb 2016, 
subject "No boot screen on Quadro M5000?" - basically passed through 
Quadro cards are meant to be secondary  only (I use them as only card, 
though). However this small quirk aside, this works for me very well.


Here are details of my setup:

* distribution Arch , with packages : qemu 2.5.0 libvirt 1.3.1 (I do not 
use qemu command line directly)


* my machine has 32 logical cores (2 x 8 core Xeon E5-26xx CPUs, with 
HT) and 128GB ECC RAM, two Quadro M5000 cards and two USB 3.0 controller 
cards; my motherboard is Supermicro X9DA7. I also dedicated half of 
cores and about half of RAM to guests (I'm also running Linux virtual 
machines with spice). Obviously you do not need this much hardware, just 
a context to help you understand my configuration.


* I am passing through one GPU and one USB controller for each of two 
Windows 10 guests (my Linux guests are running without passthrough at 
this time, although also I experimented with such setup)


* kernel 4.1.17 (vanilla, no patches at all)

* useful bits in /etc :
  bronek@gdansk /etc % cat modprobe.d/blacklist.conf
  blacklist nouveau
  blacklist snd_hda_intel

  bronek@gdansk /etc % cat modprobe.d/kvm.conf
  options kvm ignore_msrs=1

  bronek@gdansk /etc % cat modprobe.d/vfio.conf
  options vfio-pci ids=1912:0014,10de:13f0,10de:0fbb

  bronek@gdansk /etc % cat sysctl.d/80-hugepages.conf
  # Reserve this many 2MB pages for virtual machine
  vm.nr_hugepages = 28000

  bronek@gdansk /etc % grep -E "^MODULES" /etc/mkinitcpio.conf
  MODULES="vfio vfio_iommu_type1 vfio_pci vfio_virqfd vhost_net bridge 
mpt2sas nvme"


  bronek@gdansk /etc % sed 's|192.168.[0-9.]*|192.168.#.#|g' netctl/br0
  Description="Bridge connection"
  Interface=br0
  Connection=bridge
  BindsToInterfaces=(enp5s0f1)
  IP=static
  Address=('192.168.#.#/24')
  ## Do not want default gateway on this interface
  # Gateway='192.168.#.#'
  Routes=('192.168.#.#/24 via 192.168.#.#')
  DNS=('192.168.#.#' '192.168.#.#')
  NETCTL_DEBUG=yes
  ## Ignore (R)STP and immediately activate the bridge
  SkbpForwardingDelay=yes

  bronek@gdansk /etc % lspci -tvnn | less
  . . .
 +-[:80]-+-01.0-[81]--
 |   +-02.0-[82]--+-00.0  NVIDIA Corporation GM204GL [Quadro 
M5000] [10de:13f0]
 |   |\-00.1  NVIDIA Corporation GM204 High 
Definition Audio Controller [10de:0fbb]
 |   +-03.0-[83]00.0  Intel Corporation PCIe Data Center 
SSD [8086:0953]
 |   +-03.2-[84]00.0  Renesas Technology Corp. uPD720201 
USB 3.0 Host Controller [1912:0014]

  . . .
 \-[:00]-+-00.0  Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 
DMI2 [8086:0e00]
 +-01.0-[01]00.0  LSI Logic / Symbios Logic SAS2308 
PCI-Express Fusion-MPT SAS-2 [1000:0086]
 +-02.0-[02]00.0  Renesas Technology Corp. uPD720201 
USB 3.0 Host Controller [1912:0014]
 +-03.0-[03]--+-00.0  NVIDIA Corporation GM204GL [Quadro 
M5000] [10de:13f0]
 |\-00.1  NVIDIA Corporation GM204 High 
Definition Audio Controller [10de:0fbb]


* boot configuration (i.e. kernel command line parameters):
console=ttyS0,115200N8R nomodest nohz=off udev.children-max=32 
edac_core.edac_mc_panic_on_ue=1 intel_iommu=on iommu=pt 
isolcpus=4,5,6,7,12,13,14,15,20,21,22,23,28,29,30,31

(I removed parts relevant to ZFS only, which I also use)

* my guests are using UEFI boot in kvm (my host is starting from 
"legacy" BIOS, though), using edk2.git-ovmf-x64-... from 
https://www.kraxel.org/repos/jenkins/edk2/


* for Windows installation (and possibnly boot-level maintenance in the 
future) I use libvirt profile like attached "-spice" profile. I start 
and connect to this from an old laptop running Linux.


* also, as you will see in attached profiles, I am using virtio-scsi. 
This means that at the start of Windows setup I need to install vioscsi 
driver from virtio-win-0.1.112.iso (because otherwise 

[vfio-users] help with choice of GPU for passthrough

[Bronek Kozicki]

I'm upgrading my home machine, it is running two Windows 7 VMs and they 
need identical hardware configuration. The host is 2x Xeon E5-2667V2 + 
128GB RAM - obviously oversized for only two Windows ;-P . I'm 
considering choice between Radeon Fury Nano and more expensive nVidia. I 
know that Radeons have problem with PCIe reset (in kernel 4.1.15 + qemu 
2.4.1), is this likely to have been solved in a newer version I am not 
using yet/not written yet?


If not, then I will consider switching to nVidia but the problem here is 
the price, obviously. For the price only "little" higher than Nano I can 
buy Quadro M4000 but Id rather have more performance. I wonder whether 
nVidia played silly games disabling its drivers under vfio with Titan X 
- that would be a very nice option. I do not want to spend more than 
$1300 per card.


Any advice?


B.


PS Feel free to cite me to AMD support, they may possibly lose my brand 
loyalty due to broken PCIe reset - I'm just too tired of not being able 
to reliably reset my VMs. Even if I need to pay more for worse performance.


___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users