Le 30/04/2016 07:20, Alexandre DERUMIER a écrit : > ok, > > can you check with perf command ? > > check both qemu and vhost process > > > "perf top -p pidofqemu" > > "perf top -p pidofvhost"
OK, I did it, and hit the bug again, on a different hardware. Context is: Ubuntu 16.04 VM, single core, host CPU, using virtio-net to blast at 1 Gbps through the physical NIC. I still have 100% of a core used by the kvm process, and the perf top results are here: https://gist.github.com/gilou/15b620a7a067fd1d58a7616942e025b4#file-perf_virtionet_4-4-txt This was done again on a 4.1 updated to a 4.2. I will probably try on a fresh 4.2 install to check it out, but I doubt it will change much. If I fall back to the 4.2.6 kernel, bug isn't there. I can grant you access to the host should you need that, but I'm pretty sure anybody can reproduce that issue. Which brings quite some worries. Regards, Gilles Pietri > > ----- Mail original ----- > De: "Gilou" <[email protected]> > À: "proxmoxve" <[email protected]> > Envoyé: Vendredi 29 Avril 2016 13:53:17 > Objet: Re: [PVE-User] Regression on virtio-net ? (Was: Re: Proxmox VE 4.2 > released!) > > Le 29/04/2016 10:31, Alexandre DERUMIER a écrit : >> Ok, so maybe it's a kvm module regression in kernel. >> >> Just to be sure, can you check that vhost_net module is loaded ? >> >> # lsmod|grep vhost_net >> >> and also that you have a process >> >> [vhost-xxxx], where xxxx is the pid of the current qemu-kvm process > > Hi, > > # lsmod | grep vhost_net > vhost_net 20480 1 > vhost 32768 1 vhost_net > macvtap 20480 1 vhost_net > > root@px3:~# ps aux | grep vhost > root 3308 33.8 3.4 1181136 560584 ? Sl 13:41 1:59 > /usr/bin/kvm -id 3004 [...] > net-pci,mac=32:36:65:35:30:62,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 > > root 3330 1.2 0.0 0 0 ? S 13:41 0:04 > [vhost-3308] > > > So: yes, vhost_net is loaded, and I do have a vhost-3308 associated with > my VM. Yet, the CPU is at 100% for a 1 Gbps traffic :( > > @+ > Gilles > > >> >> >> >> >> and lastly, try to use perf, to see which system calls are using cpu >> >>> #apt-get install linux-tools-4.4 >>> >>> and do a >>> >>> #perf top >> >> >> ----- Mail original ----- >> De: "Gilou" <[email protected]> >> À: "proxmoxve" <[email protected]> >> Envoyé: Vendredi 29 Avril 2016 08:48:20 >> Objet: Re: [PVE-User] Regression on virtio-net ? (Was: Re: Proxmox VE 4.2 >> released!) >> >> Le 29/04/2016 00:47, Alexandre DERUMIER a écrit : >>> Can you try with previous kernel 4.2 ? (and keep proxmox 4.2 ) >>> >>> I would like to known if it's a regression in qemu or kernel. >> >> As I said, I did that, the bug is not there (or not that obvious). On a >> 4.2.6 kernel, using qemu 2.5, I get ~30% of a core for 1 Gbps, which >> seems reasonnable. >> >> I also tried on VirtualBox/KVM, on Ubuntu 16.04 host, and the bug does >> not appear there. I can try on qemu probably too, but it seems to me the >> kernel is at fault here. >> >>> >>> >>> also, you can try to install >>> >>> #apt-get install linux-tools-4.4 >>> >>> and do a >>> >>> #perf top >>> >>> >>> you should see which system call use a lot of cpu >> >> >> Here's the result, while spitting ~1 Gbps on the LAN: >> https://framapic.org/eT9JbqGsqSEK/txwSUJRpiIV3.png >> On the left, perf top, on the right top with relevant fields, showing >> all threads. >> >> so read_tsc using ~23%, ktime_get ~10%, and single thread (even if the >> VM has multiple cores) using 100% on the host. So maybe it's not >> virtio-net directly. I have a lot of CPU used on a E1000 as well, but >> that didn't strike me as obvious, as I expected this. >> >> So you know: >> # uname -a >> Linux px1 4.4.6-1-pve #1 SMP Thu Apr 21 11:25:40 CEST 2016 x86_64 GNU/Linux >> >> # pveversion -v >> proxmox-ve: 4.2-48 (running kernel: 4.4.6-1-pve) >> pve-manager: 4.2-2 (running version: 4.2-2/725d76f0) >> pve-kernel-4.4.6-1-pve: 4.4.6-48 >> pve-kernel-4.2.6-1-pve: 4.2.6-36 >> lvm2: 2.02.116-pve2 >> corosync-pve: 2.3.5-2 >> libqb0: 1.0-1 >> pve-cluster: 4.0-39 >> qemu-server: 4.0-72 >> pve-firmware: 1.1-8 >> libpve-common-perl: 4.0-59 >> libpve-access-control: 4.0-16 >> libpve-storage-perl: 4.0-50 >> pve-libspice-server1: 0.12.5-2 >> vncterm: 1.2-1 >> pve-qemu-kvm: 2.5-14 >> pve-container: 1.0-62 >> pve-firewall: 2.0-25 >> pve-ha-manager: 1.0-28 >> ksm-control-daemon: 1.2-1 >> glusterfs-client: 3.5.2-2+deb8u1 >> lxc-pve: 1.1.5-7 >> lxcfs: 2.0.0-pve2 >> cgmanager: 0.39-pve1 >> criu: 1.6.0-1 >> zfsutils: 0.6.5-pve9~jessie >> >> If I can troubleshoot that further, let me know... >> >> Cheers, >> >> Gilles Pietri >> >> >>> >>> >>> >>> >>> >>> ----- Mail original ----- >>> De: "Gilou" <[email protected]> >>> À: "proxmoxve" <[email protected]> >>> Envoyé: Jeudi 28 Avril 2016 19:31:19 >>> Objet: Re: [PVE-User] Regression on virtio-net ? (Was: Re: Proxmox VE 4.2 >>> released!) >>> >>> Le 28/04/2016 19:18, Gilou a écrit : >>>> Le 27/04/2016 15:35, Martin Maurer a écrit : >>>>> Hi all! >>>>> >>>>> We are proud to announce the final release of our Proxmox VE 4.2 with >>>>> the new GUI. >>>>> >>>>> The Sencha Ext JS 6 framework brings a modern 'flat design' look and >>>>> feel to the Proxmox VE GUI with a reworked icon set providing >>>>> consistency and an improved user experience. >>>>> >>>>> Watch our short introduction video - What's new in Proxmox VE 4.2 >>>>> http://www.proxmox.com/en/training/video-tutorials/item/what-s-new-in-proxmox-ve-4-2 >>>>> >>>>> >>>>> >>>>> This point release is based on latest and greatest Debian Jessie and a >>>>> 4.4.6 Linux kernel, based on Ubuntu 16.04 LTS kernel. >>>> >>>> Hi, >>>> >>>> I seem to have a regression, and a weird one I see neither on Proxmox >>>> 4.1 4.2.6's kernel, nor on Ubuntu 4.4.0, or >>>> >>>> Context: Ubuntu Guest VM using Ubuntu 16.04 on a virtio_net device >>>> (tested also on Debian 8, and CentOS7), connected on a bridge. >>>> >>>> on 4.1 : 1 Gbps bandwidth, ~1 GHz of CPU >>>> on 4.2 : 1 Gbps bandwidth, ~3 GHz of CPU >>>> >>>> Tested using transfer to /dev/null & iperf, so no disk involved. >>>> >>>> I do not have the issue on ubuntu 4.4.0 using virtualbox/virtio-net. >>>> >>>> Haven't tested further, but this is an isssue. >>> >>> Looked into it, it's really 100% of a core, in the following setups: >>> - single CPU on the VM >>> - local or remote traffic (local goes up to 24 Gbps, remote is 1 Gbps) >>> - no matter what distro (haven't tested Windows) >>> >>> Has anyone noticed something like this or is this something specific? >>> >>> Regards, >>> >>> >>>> >>>> Regards, >>>> Gilles Pietri >>>> >>>> >>>>> >>>>> We included as always countless small improvements and bugfixes, like >>>>> the ability to use Let´s Encrypt certificates. >>>>> >>>>> Release notes >>>>> http://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_4.2 >>>>> >>>>> ISO Download >>>>> >>>>> http://www.proxmox.com/en/downloads >>>>> >>>>> Alternate ISO download: >>>>> http://download.proxmox.com/iso/ >>>>> >>>>> Upgrading >>>>> http://pve.proxmox.com/wiki/Downloads >>>>> >>>>> Bugtracker >>>>> https://bugzilla.proxmox.com >>>>> >>>>> A big THANK-YOU to our active community for all feedback, testing, bug >>>>> reporting and patch submissions. >>>>> >>>> >>>> _______________________________________________ >>>> pve-user mailing list >>>> [email protected] >>>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>> >>> >>> _______________________________________________ >>> pve-user mailing list >>> [email protected] >>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> _______________________________________________ >>> pve-user mailing list >>> [email protected] >>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> >> >> _______________________________________________ >> pve-user mailing list >> [email protected] >> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> >> _______________________________________________ >> pve-user mailing list >> [email protected] >> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > > _______________________________________________ > pve-user mailing list > [email protected] > http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ > pve-user mailing list > [email protected] > http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ pve-user mailing list [email protected] http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
