I recently tried upgrading from vanilla LTS kernel 5.4 to vanilla 5.8 and found
that my virtual machines "stutter" when they did not previously. The hypervisor
is configured with tickless kernel (NO_HZ_FULL) and physical cores are
exclusively reserved for vcpus by means of isolcpus , nohz_full and rcu_nocbs.
The user experience under kernel 5.4 is good, it's only 5.8 when stuttering
happens. This could be either just a brief "pop" when playing music, every few
seconds, or actual short screen-freeze when playing games under Windows VM (I
use GPU pass-through with Nvidia Quadro cards).
This seem to correspond with messages appearing on hypervisor console which
look like:
[41426.188902] kvm [176116]: vcpu10, guest rIP: 0x7ff87a6a2aba ignored rdmsr:
0x122
[41426.202067] kvm [176116]: vcpu10, guest rIP: 0x7ff87a6a2aba ignored rdmsr:
0x122
[41426.218588] kvm [176116]: vcpu10, guest rIP: 0x7ff87b6d1d3a ignored rdmsr:
0x122
[41451.754213] kvm [1644425]: vcpu3, guest rIP: 0x7f5d4b442335 ignored rdmsr:
0x122
[41482.036754] kvm [176116]: vcpu2, guest rIP: 0x76f320ea ignored rdmsr: 0x122
[41482.181285] kvm [176116]: vcpu2, guest rIP: 0x76f320ea ignored rdmsr: 0x122
[41482.188947] kvm [176116]: vcpu2, guest rIP: 0x76d27f55 ignored rdmsr: 0x122
There is quite a lot of it, but only logged by kernel 5.8 and not by 5.4 (I
skipped 5.5 , 5.6 and 5.7).
I build my own kernel roughly based on ArchLinux, but taking unaltered sources
directly from kernel.org and my own configuration tweaks (such as
CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y , MCORE2=y , PREEMPT_VOLUNTARY=y ,
NO_HZ_FULL=y , HZ_100=y , disable RCU_FAST_NO_HZ , disable X86_IOPL_IOPERM ,
disable USERFAULTFD , disable XEN, disable PAGE_POISONING , disable
RAID6_PQ_BENCHMARK)
The hypervisor box has 2x Xeon 2697v2 (12 cores per CPU, two sockets, 48
physical threads) so there is plenty of CPU power left for I/O etc, despite 12
cores (24 threads) reserved for vcpus only: nohz_full=6-17,30-41
rcu_nocbs=6-17,30-41 isolcpus=6-17,30-41 . There are two VMs using these
reserved cores:
<vcpu placement='static'>12</vcpu>
<iothreads>4</iothreads>
<cputune>
<vcpupin vcpu='0' cpuset='6'/>
<vcpupin vcpu='1' cpuset='30'/>
<vcpupin vcpu='2' cpuset='7'/>
<vcpupin vcpu='3' cpuset='31'/>
<vcpupin vcpu='4' cpuset='8'/>
<vcpupin vcpu='5' cpuset='32'/>
<vcpupin vcpu='6' cpuset='9'/>
<vcpupin vcpu='7' cpuset='33'/>
<vcpupin vcpu='8' cpuset='10'/>
<vcpupin vcpu='9' cpuset='34'/>
<vcpupin vcpu='10' cpuset='11'/>
<vcpupin vcpu='11' cpuset='35'/>
<emulatorpin cpuset='4-5,28-29'/>
<iothreadpin iothread='1' cpuset='0,24'/>
<iothreadpin iothread='2' cpuset='1,25'/>
<iothreadpin iothread='3' cpuset='2,26'/>
<iothreadpin iothread='4' cpuset='3,27'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0'/>
</numatune>
... and:
<vcpu placement='static'>12</vcpu>
<iothreads>4</iothreads>
<cputune>
<vcpupin vcpu='0' cpuset='12'/>
<vcpupin vcpu='1' cpuset='36'/>
<vcpupin vcpu='2' cpuset='13'/>
<vcpupin vcpu='3' cpuset='37'/>
<vcpupin vcpu='4' cpuset='14'/>
<vcpupin vcpu='5' cpuset='38'/>
<vcpupin vcpu='6' cpuset='15'/>
<vcpupin vcpu='7' cpuset='39'/>
<vcpupin vcpu='8' cpuset='16'/>
<vcpupin vcpu='9' cpuset='40'/>
<vcpupin vcpu='10' cpuset='17'/>
<vcpupin vcpu='11' cpuset='41'/>
<emulatorpin cpuset='18-19,42-43'/>
<iothreadpin iothread='1' cpuset='20,44'/>
<iothreadpin iothread='2' cpuset='21,45'/>
<iothreadpin iothread='3' cpuset='22,46'/>
<iothreadpin iothread='4' cpuset='23,47'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='1'/>
</numatune
There is no swap at all, the box has almost 400GB RAM.
B.
--
Bronek Kozicki
b...@incorrekt.com
_______________________________________________
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users
