Re: [Qemu-devel] Cross-posted : Odd QXL/KVM performance issue with a Windows 7 Guest
Brad Campbell writes:
> On 6/9/19 21:38, Brad Campbell wrote:
>> 7022@1567775824.002106:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c970
>> 7022@1567775824.002115:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c980
>> 7022@1567775824.003122:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c970
>
>> Does this look familiar to anyone?
>
> Ugh. System timer.
>
> So with the timer interrupt removed and an added trace on IRQ > 0:
>
> qxl/guest-0: 79096403248: qxldd: DrvCopyBits
> 14955@1567780063.149527:kvm_vcpu_ioctl cpu_index 2, type 0xae80, arg (nil)
> 14956@1567780063.150291:qxl_ring_res_put 0 #res=1
> 14955@1567780063.163672:kvm_run_exit cpu_index 2, reason 2
> 14955@1567780063.163688:qxl_io_write 0 native addr=4 (QXL_IO_NOTIFY_OOM)
> val=0 size=1 async=0
> 14955@1567780063.163704:qxl_spice_oom 0
> 14955@1567780063.163720:kvm_vcpu_ioctl cpu_index 2, type 0xae80, arg (nil)
> 14956@1567780063.163755:qxl_ring_command_check 0 native
> 14956@1567780063.163779:qxl_ring_res_push 0 native s#=0 res#=1
> last=0x7f3c0d44b6e0 notify=yes
> 14956@1567780063.163816:qxl_ring_res_push_rest 0 ring 1/8 [326,325]
> 14956@1567780063.163841:qxl_send_events 0 1
> 14956@1567780063.163868:qxl_ring_cursor_check 0 native
> 14956@1567780063.163888:qxl_ring_command_check 0 native
> 14924@1567780063.163879:kvm_set_irq irq 11, level 1, status 1
> 14954@1567780063.163895:kvm_run_exit cpu_index 1, reason 2
> 14954@1567780063.163965:qxl_io_write 0 native addr=3 (QXL_IO_UPDATE_IRQ)
> val=0 size=1 async=0
> 14954@1567780063.164006:kvm_set_irq irq 11, level 0, status 1
> 14954@1567780063.164029:kvm_vcpu_ioctl cpu_index 1, type 0xae80, arg (nil)
> 14954@1567780063.164065:kvm_run_exit cpu_index 1, reason 2
> 14954@1567780063.164080:qxl_io_write 0 native addr=3 (QXL_IO_UPDATE_IRQ)
> val=0 size=1 async=0
> 14954@1567780063.164104:kvm_vcpu_ioctl cpu_index 1, type 0xae80, arg (nil)
> 14955@1567780063.266778:kvm_run_exit cpu_index 2, reason 2
> 14955@1567780063.266790:qxl_io_write 0 native addr=0 (QXL_IO_NOTIFY_CMD)
> val=0 size=1 async=0
> 14955@1567780063.266809:kvm_vcpu_ioctl cpu_index 2, type 0xae80, arg (nil)
> 14956@1567780063.266822:qxl_ring_cursor_check 0 native
> 14956@1567780063.266842:qxl_ring_command_check 0 native
> 79213750625 qxl-0/cmd: cmd @ 0x1000104b598 draw: surface_id 0 type copy
> effect opaque src 10001fecbf8 (id 9fe0870780 type 0 flags 0 width 1920
> height 1080, fmt 8 flags 0 x 1920 y 1080 stride 7680 palette 0 data
> 10001fecc28) area 1920x1080+0+0 rop 8
> 14956@1567780063.266983:qxl_ring_command_get 0 native
> 14956@1567780063.267044:qxl_ring_command_check 0 native
> 14956@1567780063.267070:qxl_ring_cursor_check 0 native
> 14956@1567780063.267087:qxl_ring_command_check 0 native
> 14956@1567780063.267109:qxl_ring_command_req_notification 0
> 14955@1567780063.267967:kvm_run_exit cpu_index 2, reason 2
> 14955@1567780063.267987:qxl_io_write 0 native addr=7 (QXL_IO_LOG) val=0
> size=1 async=0
> 14955@1567780063.268015:qxl_io_log 0 qxldd: DrvCopyBits
>
> So if I'm not mistaken (for the nth time), we have KVM_RUN on cpu index 2
> here:
>
> 14955@1567780063.163720:kvm_vcpu_ioctl cpu_index 2, type 0xae80, arg (nil)
>
> And it returns here :
>
> 14955@1567780063.266778:kvm_run_exit cpu_index 2, reason 2
>
> Does that imply guest code is running for ~100ms on that vcpu?
Yes. In the KVM game vmexits is what kills performance. If QEMU is
involved in doing the emulation you have to exit the guest, go through
the kernel, exit the ioctl and then QEMU does it's thing before you
restart.
You can't avoid all exits - indeed VIRTIO is designed to limit the exits
to a single exit per transmission. However if you are emulating a legacy
device in QEMU every access to a memory mapped register will involve an
exit.
If you monitor QEMU with "perf record" and then look at the result
("perf report") see what QEMU is doing all that time. It's likely there
is a legacy device somewhere which the guest kernel is hammering.
--
Alex.
Re: [Qemu-devel] Cross-posted : Odd QXL/KVM performance issue with a Windows 7 Guest
* Brad Campbell (lists2...@fnarfbargle.com) wrote: > > On 7/9/19 03:03, Dr. David Alan Gilbert wrote: > > * Brad Campbell (lists2...@fnarfbargle.com) wrote: > > > On 2/9/19 6:23 pm, Brad Campbell wrote: > > > > > > > Here is the holdup : > > > > > > > > 11725@1567416625.003504:qxl_ring_command_check 0 native > > > > 11725@1567416625.102653:qxl_io_write 0 native addr=0 (QXL_IO_NOTIFY_CMD) > > > > val=0 size=1 async=0 > > > > > > > > ~100ms delay prior to each logged QXL_IO_NOTIFY_CMD on the AMD box which > > > > explains the performance difference. Now I just need to figure out if > > > > that lies in the guest, the guest QXL driver, QEMU or SPICE and why it > > > > exhibits on the AMD box and not the i7. > > > > > > > > To get to this point, I recompiled the kernel on the i7 box with both > > > > AMD and Intel KVM modules. Once that was running I cloned the drive and > > > > put it in the AMD box, so the OS, software stack and all dependencies > > > > are identical. > > > Reacp : > > > > > > I have a machine with a Windows 7 VM which is running on an i7-3770. This > > > works perfectly. > > > > > > Clone the disk and put it in a new(ish) AMD Ryzen 1500x machine and the > > > display output using qxl/spice is now limited to ~5-7fps. > > > > > > I originally cloned the entire machine to keep the software versions > > > identical. > > > > > > To simplify debugging and reproduction I'm now using : > > > - An identical SPICE version to that on the i7. > > > - A fresh 64 bit Windows 7 VM. > > > - The D2D benchmark from Crystalmark 2004R7. > > > > > > The machine is booted with : > > > > > > qemu -enable-kvm \ > > > -m 8192\ > > > -rtc base=localtime\ > > > -vga qxl\ > > > -device qxl\ > > > -global qxl-vga.guestdebug=3\ > > > -global qxl-vga.cmdlog=1\ > > > -global qxl-vga.vram_size=65536\ > > > -global qxl.vram_size=65536\ > > > -global qxl-vga.ram_size=65536\ > > > -global qxl.ram_size=65536\ > > > -net nic,model=virtio\ > > > -net tap,ifname=tap0,script=/etc/qemu-ifup,vhost=on\ > > > -usbdevice tablet\ > > > -spice port=5930,disable-ticketing\ > > > -device virtio-serial\ > > > -chardev spicevmc,id=vdagent,name=vdagent\ > > > -device virtserialport,chardev=vdagent,name=com.redhat.spice.0\ > > > -smp 3,maxcpus=3,cores=3,threads=1,sockets=1\ > > > -cpu qemu64,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \ > > -cpu qemu64 is almost always a bad idea; does -cpu host help ? > > > > Dave > > > No. I was using -cpu host. I changed it to qemu64 for testing so I could add > & remove -enable-kvm for testing without the machine changing drivers about. Oh, hmm. Sorry I don't know too much where to look then; you have any of: a) Windows b) guest graphics drivers c) spice server in qemu and probalby some more. So I think it's going to be a case of profiling on the two different systems and see if you can spot anything in particular that stands out. Dave > Regards, > > Brad > > -- > An expert is a person who has found out by his own painful > experience all the mistakes that one can make in a very > narrow field. - Niels Bohr > -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
Re: [Qemu-devel] Cross-posted : Odd QXL/KVM performance issue with a Windows 7 Guest
On 7/9/19 03:03, Dr. David Alan Gilbert wrote: * Brad Campbell (lists2...@fnarfbargle.com) wrote: On 2/9/19 6:23 pm, Brad Campbell wrote: Here is the holdup : 11725@1567416625.003504:qxl_ring_command_check 0 native 11725@1567416625.102653:qxl_io_write 0 native addr=0 (QXL_IO_NOTIFY_CMD) val=0 size=1 async=0 ~100ms delay prior to each logged QXL_IO_NOTIFY_CMD on the AMD box which explains the performance difference. Now I just need to figure out if that lies in the guest, the guest QXL driver, QEMU or SPICE and why it exhibits on the AMD box and not the i7. To get to this point, I recompiled the kernel on the i7 box with both AMD and Intel KVM modules. Once that was running I cloned the drive and put it in the AMD box, so the OS, software stack and all dependencies are identical. Reacp : I have a machine with a Windows 7 VM which is running on an i7-3770. This works perfectly. Clone the disk and put it in a new(ish) AMD Ryzen 1500x machine and the display output using qxl/spice is now limited to ~5-7fps. I originally cloned the entire machine to keep the software versions identical. To simplify debugging and reproduction I'm now using : - An identical SPICE version to that on the i7. - A fresh 64 bit Windows 7 VM. - The D2D benchmark from Crystalmark 2004R7. The machine is booted with : qemu -enable-kvm \ -m 8192\ -rtc base=localtime\ -vga qxl\ -device qxl\ -global qxl-vga.guestdebug=3\ -global qxl-vga.cmdlog=1\ -global qxl-vga.vram_size=65536\ -global qxl.vram_size=65536\ -global qxl-vga.ram_size=65536\ -global qxl.ram_size=65536\ -net nic,model=virtio\ -net tap,ifname=tap0,script=/etc/qemu-ifup,vhost=on\ -usbdevice tablet\ -spice port=5930,disable-ticketing\ -device virtio-serial\ -chardev spicevmc,id=vdagent,name=vdagent\ -device virtserialport,chardev=vdagent,name=com.redhat.spice.0\ -smp 3,maxcpus=3,cores=3,threads=1,sockets=1\ -cpu qemu64,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \ -cpu qemu64 is almost always a bad idea; does -cpu host help ? Dave No. I was using -cpu host. I changed it to qemu64 for testing so I could add & remove -enable-kvm for testing without the machine changing drivers about. Regards, Brad -- An expert is a person who has found out by his own painful experience all the mistakes that one can make in a very narrow field. - Niels Bohr
Re: [Qemu-devel] Cross-posted : Odd QXL/KVM performance issue with a Windows 7 Guest
* Brad Campbell (lists2...@fnarfbargle.com) wrote: > On 2/9/19 6:23 pm, Brad Campbell wrote: > > > > > Here is the holdup : > > > > 11725@1567416625.003504:qxl_ring_command_check 0 native > > 11725@1567416625.102653:qxl_io_write 0 native addr=0 (QXL_IO_NOTIFY_CMD) > > val=0 size=1 async=0 > > > > ~100ms delay prior to each logged QXL_IO_NOTIFY_CMD on the AMD box which > > explains the performance difference. Now I just need to figure out if > > that lies in the guest, the guest QXL driver, QEMU or SPICE and why it > > exhibits on the AMD box and not the i7. > > > > To get to this point, I recompiled the kernel on the i7 box with both > > AMD and Intel KVM modules. Once that was running I cloned the drive and > > put it in the AMD box, so the OS, software stack and all dependencies > > are identical. > > Reacp : > > I have a machine with a Windows 7 VM which is running on an i7-3770. This > works perfectly. > > Clone the disk and put it in a new(ish) AMD Ryzen 1500x machine and the > display output using qxl/spice is now limited to ~5-7fps. > > I originally cloned the entire machine to keep the software versions > identical. > > To simplify debugging and reproduction I'm now using : > - An identical SPICE version to that on the i7. > - A fresh 64 bit Windows 7 VM. > - The D2D benchmark from Crystalmark 2004R7. > > The machine is booted with : > > qemu -enable-kvm \ > -m 8192\ > -rtc base=localtime\ > -vga qxl\ > -device qxl\ > -global qxl-vga.guestdebug=3\ > -global qxl-vga.cmdlog=1\ > -global qxl-vga.vram_size=65536\ > -global qxl.vram_size=65536\ > -global qxl-vga.ram_size=65536\ > -global qxl.ram_size=65536\ > -net nic,model=virtio\ > -net tap,ifname=tap0,script=/etc/qemu-ifup,vhost=on\ > -usbdevice tablet\ > -spice port=5930,disable-ticketing\ > -device virtio-serial\ > -chardev spicevmc,id=vdagent,name=vdagent\ > -device virtserialport,chardev=vdagent,name=com.redhat.spice.0\ > -smp 3,maxcpus=3,cores=3,threads=1,sockets=1\ > -cpu qemu64,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \ -cpu qemu64 is almost always a bad idea; does -cpu host help ? Dave > -drive > file=/server/VM/Cadbox.raw,if=virtio,aio=threads,format=raw,cache=unsafe > -boot c \ > -drive > file=/server/VM/Cadbox_swap.raw,if=virtio,aio=threads,format=raw,cache=unsafe > \ > > > The D2D benchmark runs through a series of Sprites (stars) and it just > shuffles them around the screen. > > With KVM enabled I get : > > Sprite10 - 8.66fps > Sprite 100 - 8.47fps > Sprite 500 - 8.45fps > Sprite 1000 - 8.18fps > Sprite 5000 - 7.64fps > Sprite 1 - 7.26fps > > With the identical system, with KVM disabled I get : > > Sprite10 - 28.97fps > Sprite 100 - 27.24fps > Sprite 500 - 23.85fps > Sprite 1000 - 22.00fps > Sprite 5000 - 11.11fps > Sprite 1 - 4.50fps > > On the i7 with the same software version and kvm enabled : > > Sprite10 - 88.58fps > Sprite 100 - 88.35fps > Sprite 500 - 85.64fps > Sprite 1000 - 83.33fps > Sprite 5000 - 58.08fps > Sprite 1 - 45.29fps > > > cpuinfo from the host : > processor : 7 > vendor_id : AuthenticAMD > cpu family: 23 > model : 1 > model name: AMD Ryzen 5 1500X Quad-Core Processor > stepping : 1 > microcode : 0x8001138 > cpu MHz : 2877.596 > cache size: 512 KB > physical id : 0 > siblings : 8 > core id : 5 > cpu cores : 4 > apicid: 11 > initial apicid: 11 > fpu : yes > fpu_exception : yes > cpuid level : 13 > wp: yes > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov > pat > pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp > lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni > pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx > f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse > 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext > perfctr_llc mwaitx cpb hw_pstate sme ssbd sev ibpb vmmcall fsgsbase bmi1 > avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 > xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale > vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic > v_vmsave_vmload vgif overflow_recov succor smca > bugs : sysret_ss_attrs null_seg spectre_v1 spectre_v2 > spec_store_bypass > bogomips : 6985.80 > TLB size : 2560 4K pages > clflush size : 64 > cache_alignment : 64 > address sizes : 43 bits physical, 48 bits virtual > power management: ts ttp tm hwpstate eff_freq_ro [13] [14] > > qemu configured with : > PKG_CONFIG_PATH=/usr/local/libvirt/lib/pkgconfig:/usr/local/libvirt/share/pkgconfig/ > ./configure --target-list=x86_64-softmmu --disable-gtk && make -j6 > test:~# uname -a > > Linux test 5.2.9 #42 SMP Tue Aug 20 16:41:13 AWST 2019 x86_64 GNU/Linux > > test:~#
Re: [Qemu-devel] Cross-posted : Odd QXL/KVM performance issue with a Windows 7 Guest
On 6/9/19 21:38, Brad Campbell wrote: 7022@1567775824.002106:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c970 7022@1567775824.002115:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c980 7022@1567775824.003122:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c970 Does this look familiar to anyone? Ugh. System timer. So with the timer interrupt removed and an added trace on IRQ > 0: qxl/guest-0: 79096403248: qxldd: DrvCopyBits 14955@1567780063.149527:kvm_vcpu_ioctl cpu_index 2, type 0xae80, arg (nil) 14956@1567780063.150291:qxl_ring_res_put 0 #res=1 14955@1567780063.163672:kvm_run_exit cpu_index 2, reason 2 14955@1567780063.163688:qxl_io_write 0 native addr=4 (QXL_IO_NOTIFY_OOM) val=0 size=1 async=0 14955@1567780063.163704:qxl_spice_oom 0 14955@1567780063.163720:kvm_vcpu_ioctl cpu_index 2, type 0xae80, arg (nil) 14956@1567780063.163755:qxl_ring_command_check 0 native 14956@1567780063.163779:qxl_ring_res_push 0 native s#=0 res#=1 last=0x7f3c0d44b6e0 notify=yes 14956@1567780063.163816:qxl_ring_res_push_rest 0 ring 1/8 [326,325] 14956@1567780063.163841:qxl_send_events 0 1 14956@1567780063.163868:qxl_ring_cursor_check 0 native 14956@1567780063.163888:qxl_ring_command_check 0 native 14924@1567780063.163879:kvm_set_irq irq 11, level 1, status 1 14954@1567780063.163895:kvm_run_exit cpu_index 1, reason 2 14954@1567780063.163965:qxl_io_write 0 native addr=3 (QXL_IO_UPDATE_IRQ) val=0 size=1 async=0 14954@1567780063.164006:kvm_set_irq irq 11, level 0, status 1 14954@1567780063.164029:kvm_vcpu_ioctl cpu_index 1, type 0xae80, arg (nil) 14954@1567780063.164065:kvm_run_exit cpu_index 1, reason 2 14954@1567780063.164080:qxl_io_write 0 native addr=3 (QXL_IO_UPDATE_IRQ) val=0 size=1 async=0 14954@1567780063.164104:kvm_vcpu_ioctl cpu_index 1, type 0xae80, arg (nil) 14955@1567780063.266778:kvm_run_exit cpu_index 2, reason 2 14955@1567780063.266790:qxl_io_write 0 native addr=0 (QXL_IO_NOTIFY_CMD) val=0 size=1 async=0 14955@1567780063.266809:kvm_vcpu_ioctl cpu_index 2, type 0xae80, arg (nil) 14956@1567780063.266822:qxl_ring_cursor_check 0 native 14956@1567780063.266842:qxl_ring_command_check 0 native 79213750625 qxl-0/cmd: cmd @ 0x1000104b598 draw: surface_id 0 type copy effect opaque src 10001fecbf8 (id 9fe0870780 type 0 flags 0 width 1920 height 1080, fmt 8 flags 0 x 1920 y 1080 stride 7680 palette 0 data 10001fecc28) area 1920x1080+0+0 rop 8 14956@1567780063.266983:qxl_ring_command_get 0 native 14956@1567780063.267044:qxl_ring_command_check 0 native 14956@1567780063.267070:qxl_ring_cursor_check 0 native 14956@1567780063.267087:qxl_ring_command_check 0 native 14956@1567780063.267109:qxl_ring_command_req_notification 0 14955@1567780063.267967:kvm_run_exit cpu_index 2, reason 2 14955@1567780063.267987:qxl_io_write 0 native addr=7 (QXL_IO_LOG) val=0 size=1 async=0 14955@1567780063.268015:qxl_io_log 0 qxldd: DrvCopyBits So if I'm not mistaken (for the nth time), we have KVM_RUN on cpu index 2 here: 14955@1567780063.163720:kvm_vcpu_ioctl cpu_index 2, type 0xae80, arg (nil) And it returns here : 14955@1567780063.266778:kvm_run_exit cpu_index 2, reason 2 Does that imply guest code is running for ~100ms on that vcpu? Brad -- An expert is a person who has found out by his own painful experience all the mistakes that one can make in a very narrow field. - Niels Bohr
Re: [Qemu-devel] Cross-posted : Odd QXL/KVM performance issue with a Windows 7 Guest
On 6/9/19 16:49, Brad Campbell wrote: On 2/9/19 6:23 pm, Brad Campbell wrote: Here is the holdup : 11725@1567416625.003504:qxl_ring_command_check 0 native 11725@1567416625.102653:qxl_io_write 0 native addr=0 (QXL_IO_NOTIFY_CMD) val=0 size=1 async=0 ~100ms delay prior to each logged QXL_IO_NOTIFY_CMD on the AMD box which explains the performance difference. Now I just need to figure out if that lies in the guest, the guest QXL driver, QEMU or SPICE and why it exhibits on the AMD box and not the i7. To get to this point, I recompiled the kernel on the i7 box with both AMD and Intel KVM modules. Once that was running I cloned the drive and put it in the AMD box, so the OS, software stack and all dependencies are identical. Had a bit more of a poke and traced kvm and qxl on both machines at approximately comparative times of the same benchmark. From what I can ascertain in this context ioctl type 0xc008ae67 is KVM_IRQ_LINE_STATUS. The i7 trace shows maybe 10ms spent hitting that. The Ryzen trace below shows the majority of the frame spent hitting that ioctl (some ~110ms). i7 : qxl/guest-0: 161680464256: qxldd: DrvCopyBits 7022@1567775823.999850:kvm_vcpu_ioctl cpu_index 1, type 0xae80, arg (nil) 7022@1567775824.000106:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c970 7022@1567775824.000115:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c980 7022@1567775824.001106:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c970 7022@1567775824.001117:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c980 7022@1567775824.002106:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c970 7022@1567775824.002115:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c980 7022@1567775824.003122:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c970 7022@1567775824.003145:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c980 7022@1567775824.004122:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c970 7022@1567775824.004144:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c980 7022@1567775824.005122:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c970 7022@1567775824.005144:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c980 7022@1567775824.006122:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c970 7022@1567775824.006144:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c980 7022@1567775824.007122:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c970 7022@1567775824.007144:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c980 7022@1567775824.008128:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c970 7022@1567775824.008150:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c980 7022@1567775824.009122:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c970 7022@1567775824.009144:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c980 7022@1567775824.010118:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c970 7022@1567775824.010139:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c980 7022@1567775824.08:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c970 7022@1567775824.011139:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c980 7022@1567775824.012117:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c970 7022@1567775824.012138:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c980 7022@1567775824.012200:kvm_run_exit cpu_index 1, reason 2 7022@1567775824.012214:qxl_io_write 0 native addr=4 (QXL_IO_NOTIFY_OOM) val=0 size=1 async=0 7022@1567775824.012226:qxl_spice_oom 0 7022@1567775824.012238:kvm_vcpu_ioctl cpu_index 1, type 0xae80, arg (nil) 7022@1567775824.012280:qxl_ring_command_check 0 native 7022@1567775824.012288:qxl_ring_res_push 0 native s#=0 res#=2 last=0x7faee4c2d458 notify=yes 7022@1567775824.012296:qxl_ring_res_push_rest 0 ring 1/8 [2399,2398] 7022@1567775824.012307:qxl_send_events 0 1 7022@1567775824.012319:qxl_ring_cursor_check 0 native 7022@1567775824.012325:qxl_ring_command_check 0 native 7022@1567775824.012332:qxl_ring_cursor_check 0 native 7022@1567775824.012336:qxl_ring_command_check 0 native 7022@1567775824.012342:qxl_ring_cursor_check 0 native 7022@1567775824.012349:qxl_ring_command_check 0 native 7022@1567775824.012367:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c890 7022@1567775824.012388:kvm_run_exit cpu_index 1, reason 2 7022@1567775824.012399:qxl_io_write 0 native addr=3 (QXL_IO_UPDATE_IRQ) val=0 size=1 async=0 7022@1567775824.012416:kvm_vm_ioctl type 0xc008ae67, arg 0x7fb4f5a296b0 7022@1567775824.012426:kvm_vcpu_ioctl cpu_index 1, type 0xae80, arg (nil) 7022@1567775824.012444:kvm_run_exit cpu_index 1, reason 2 7022@1567775824.012452:qxl_io_write 0 native addr=3 (QXL_IO_UPDATE_IRQ) val=0 size=1 async=0 7022@1567775824.012466:kvm_vcpu_ioctl cpu_index 1, type 0xae80, arg (nil) 7022@1567775824.013106:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c970 7022@1567775824.013124:kvm_vm_ioctl type 0xc008ae67, arg 0x7ffe13b0c980 7022@1567775824.013642:kvm_run_exit cpu_index 1, reason 2 7022@1567775824.013652:qxl_io_write 0 native addr=0 (QXL_IO_NOTIFY_CMD) val=0 size=1 async=0 7022@1567775824.013667:kvm_vcpu_ioctl cpu_index 1, type 0xae80, arg (nil) 7022@1567775824.013726:qxl_ring_cursor_check 0 native
