[Bug 1100843] Re: Live Migration Causes Performance Issues
QEMU 1.7 was released, Quantal has 10ish days left of support, and Raring is EOL ** Changed in: qemu Status: Fix Committed = Fix Released ** Changed in: qemu-kvm (Ubuntu Quantal) Status: Triaged = Invalid ** Changed in: qemu-kvm (Ubuntu Raring) Status: Triaged = Invalid -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Fix will be part of QEMU 1.7.0 (commit fc1c4a5, migration: drop MADVISE_DONT_NEED for incoming zero pages, 2013-10-24). ** Changed in: qemu Status: New = Fix Committed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
** Changed in: qemu-kvm (Ubuntu Quantal) Assignee: Chris J Arges (arges) = (unassigned) ** Changed in: qemu-kvm (Ubuntu Raring) Assignee: Chris J Arges (arges) = (unassigned) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
This bug was fixed in the package qemu-kvm - 1.0+noroms-0ubuntu14.12 --- qemu-kvm (1.0+noroms-0ubuntu14.12) precise-proposed; urgency=low * migration-do-not-overwrite-zero-pages.patch, call-madv-hugepage-for-guest-ram-allocations.patch: Fix performance degradation after migrations, and savevm/loadvm. (LP: #1100843) -- Chris J Arges chris.j.ar...@ubuntu.com Wed, 02 Oct 2013 16:26:27 -0500 ** Changed in: qemu-kvm (Ubuntu Precise) Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
I have verified this on my local machine using virt-manager's save memory, savevm/loadvm via the qemu monitor , and migrate via qemu monitor. ** Tags removed: verification-needed ** Tags added: verification-done -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Hello Mark, or anyone else affected, Accepted qemu-kvm into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/qemu-kvm/1.0+noroms- 0ubuntu14.12 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: qemu-kvm (Ubuntu Precise) Status: In Progress = Fix Committed ** Tags added: verification-needed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
I found that two patches need to be backported to solve this issue: ad0b5321f1f797274603ebbe20108b0750baee94 211ea74022f51164a7729030b28eec90b6c99a08 I've added the necessary bits into precise and tried a few tests: 1) Measure performance before and after savevm/loadvm. 2) Measure performance before and after a migrate to the same host. In both cases the performance measured by something like lmbench was the same as the previous run. A test build is available here: http://people.canonical.com/~arges/lp1100843/precise_v2/ ** Patch added: fix-lp1100843-precise.debdiff https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+attachment/3864309/+files/fix-lp1100843-precise.debdiff -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
** Description changed: + SRU Justification + [Impact] + * Users of QEMU that save their memory states using savevm/loadvm or migrate experience worse performance after the migration/loadvm. To workaround these issues VMs must be completely rebooted. Optimally we should be able to restore a VM's memory state an expect no performance issue. + + [Test Case] + + * savevm/loadvm: +- Create a VM and install a test suite such as lmbench. +- Get numbers right after boot and record them. +- Open up the qemu monitor and type the following: + stop + savevm 0 + loadvm 0 + c +- Measure performance and record numbers. +- Compare if numbers are within margin of error. + * migrate: +- Create VM, install lmbench, get numbers. +- Open up qemu monitor and type the following: + stop + migrate exec:dd of=~/save.vm + quit +- Start a new VM using qemu but add the following argument: + -incoming exec:dd if=~/save.vm +- Run performance test and compare. + + If performance measured is similar then we pass the test case. + + [Regression Potential] + + * The fix is a backport of two upstream patches: + ad0b5321f1f797274603ebbe20108b0750baee94 + 211ea74022f51164a7729030b28eec90b6c99a08 + + On patch allows QEMU to use THP if its enabled. + The other patch changes logic to not memset pages to zero when loading memory for the vm (on an incoming migration). + + -- + I have 2 physical hosts running Ubuntu Precise. With 1.0+noroms- 0ubuntu14.7 and qemu-kvm 1.2.0+noroms-0ubuntu7 (source from quantal, built for Precise with pbuilder.) I attempted to build qemu-1.3.0 debs from source to test, but libvirt seems to have an issue with it that I haven't been able to track down yet. - I'm seeing a performance degradation after live migration on Precise, + I'm seeing a performance degradation after live migration on Precise, but not Lucid. These hosts are managed by libvirt (tested both 0.9.8-2ubuntu17 and 1.0.0-0ubuntu4) in conjunction with OpenNebula. I don't seem to have this problem with lucid guests (running a number of standard kernels, 3.2.5 mainline and backported linux- image-3.2.0-35-generic as well.) I first noticed this problem with phoronix doing compilation tests, and then tried lmbench where even simple calls experience performance degradation. I've attempted to post to the kvm mailing list, but so far the only suggestion was it may be related to transparent hugepages not being used after migration, but this didn't pan out. Someone else has a similar problem here - http://thread.gmane.org/gmane.comp.emulators.kvm.devel/100592 qemu command line example: /usr/bin/kvm -name one-2 -S -M pc-1.2 -cpu Westmere -enable-kvm -m 73728 -smp 16,sockets=2,cores=8,threads=1 -uuid f89e31a4-4945-c12c-6544-149ba0746c2f -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-2.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/one//datastores/0/2/disk.0,if=none,id=drive-virtio- disk0,format=raw,cache=none -device virtio-blk- pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio- disk0,bootindex=1 -drive file=/var/lib/one//datastores/0/2/disk.1,if=none,id=drive- ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive =drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net- pci,netdev=hostnet0,id=net0,mac=02:00:0a:64:02:fe,bus=pci.0,addr=0x3 -vnc 0.0.0.0:2,password -vga cirrus -incoming tcp:0.0.0.0:49155 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 Disk backend is LVM running on SAN via FC connection (using symlink from /var/lib/one/datastores/0/2/disk.0 above) - ubuntu-12.04 - first boot == Simple syscall: 0.0527 microseconds Simple read: 0.1143 microseconds Simple write: 0.0953 microseconds Simple open/close: 1.0432 microseconds Using phoronix pts/compuational ImageMagick - 31.54s Linux Kernel 3.1 - 43.91s Mplayer - 30.49s PHP - 22.25s - ubuntu-12.04 - post live migration == Simple syscall: 0.0621 microseconds Simple read: 0.2485 microseconds Simple write: 0.2252 microseconds Simple open/close: 1.4626 microseconds Using phoronix pts/compilation ImageMagick - 43.29s Linux Kernel 3.1 - 76.67s Mplayer - 45.41s PHP - 29.1s - - I don't have phoronix results for 10.04 handy, but they were within 1% of each other... + I don't have phoronix results for 10.04 handy, but they were within 1% + of each other... ubuntu-10.04 - first boot == Simple syscall: 0.0524 microseconds Simple read: 0.1135
[Bug 1100843] Re: Live Migration Causes Performance Issues
** Description changed: SRU Justification - [Impact] - * Users of QEMU that save their memory states using savevm/loadvm or migrate experience worse performance after the migration/loadvm. To workaround these issues VMs must be completely rebooted. Optimally we should be able to restore a VM's memory state an expect no performance issue. + [Impact] + * Users of QEMU that save their memory states using savevm/loadvm or migrate experience worse performance after the migration/loadvm. To workaround these issues VMs must be completely rebooted. Optimally we should be able to restore a VM's memory state an expect no performance issue. [Test Case] - * savevm/loadvm: -- Create a VM and install a test suite such as lmbench. -- Get numbers right after boot and record them. -- Open up the qemu monitor and type the following: - stop - savevm 0 - loadvm 0 - c -- Measure performance and record numbers. -- Compare if numbers are within margin of error. - * migrate: -- Create VM, install lmbench, get numbers. -- Open up qemu monitor and type the following: - stop - migrate exec:dd of=~/save.vm - quit -- Start a new VM using qemu but add the following argument: - -incoming exec:dd if=~/save.vm -- Run performance test and compare. - - If performance measured is similar then we pass the test case. + * savevm/loadvm: + - Create a VM and install a test suite such as lmbench. + - Get numbers right after boot and record them. + - Open up the qemu monitor and type the following: + stop + savevm 0 + loadvm 0 + c + - Measure performance and record numbers. + - Compare if numbers are within margin of error. + * migrate: + - Create VM, install lmbench, get numbers. + - Open up qemu monitor and type the following: + stop + migrate exec:dd of=~/save.vm + quit + - Start a new VM using qemu but add the following argument: + -incoming exec:dd if=~/save.vm + - Run performance test and compare. + + If performance measured is similar then we pass the test case. [Regression Potential] - * The fix is a backport of two upstream patches: + * The fix is a backport of two upstream patches: ad0b5321f1f797274603ebbe20108b0750baee94 211ea74022f51164a7729030b28eec90b6c99a08 On patch allows QEMU to use THP if its enabled. The other patch changes logic to not memset pages to zero when loading memory for the vm (on an incoming migration). + * I've also run the qa-regression-testing test-qemu.py script and it passes all tests. -- I have 2 physical hosts running Ubuntu Precise. With 1.0+noroms- 0ubuntu14.7 and qemu-kvm 1.2.0+noroms-0ubuntu7 (source from quantal, built for Precise with pbuilder.) I attempted to build qemu-1.3.0 debs from source to test, but libvirt seems to have an issue with it that I haven't been able to track down yet. I'm seeing a performance degradation after live migration on Precise, but not Lucid. These hosts are managed by libvirt (tested both 0.9.8-2ubuntu17 and 1.0.0-0ubuntu4) in conjunction with OpenNebula. I don't seem to have this problem with lucid guests (running a number of standard kernels, 3.2.5 mainline and backported linux- image-3.2.0-35-generic as well.) I first noticed this problem with phoronix doing compilation tests, and then tried lmbench where even simple calls experience performance degradation. I've attempted to post to the kvm mailing list, but so far the only suggestion was it may be related to transparent hugepages not being used after migration, but this didn't pan out. Someone else has a similar problem here - http://thread.gmane.org/gmane.comp.emulators.kvm.devel/100592 qemu command line example: /usr/bin/kvm -name one-2 -S -M pc-1.2 -cpu Westmere -enable-kvm -m 73728 -smp 16,sockets=2,cores=8,threads=1 -uuid f89e31a4-4945-c12c-6544-149ba0746c2f -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-2.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/one//datastores/0/2/disk.0,if=none,id=drive-virtio- disk0,format=raw,cache=none -device virtio-blk- pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio- disk0,bootindex=1 -drive file=/var/lib/one//datastores/0/2/disk.1,if=none,id=drive- ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive =drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net- pci,netdev=hostnet0,id=net0,mac=02:00:0a:64:02:fe,bus=pci.0,addr=0x3 -vnc 0.0.0.0:2,password -vga cirrus -incoming tcp:0.0.0.0:49155 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 Disk backend is LVM running on SAN via FC connection
[Bug 1100843] Re: Live Migration Causes Performance Issues
** Description changed: SRU Justification [Impact] * Users of QEMU that save their memory states using savevm/loadvm or migrate experience worse performance after the migration/loadvm. To workaround these issues VMs must be completely rebooted. Optimally we should be able to restore a VM's memory state an expect no performance issue. [Test Case] * savevm/loadvm: - Create a VM and install a test suite such as lmbench. - Get numbers right after boot and record them. - Open up the qemu monitor and type the following: stop savevm 0 loadvm 0 c - Measure performance and record numbers. - Compare if numbers are within margin of error. * migrate: - Create VM, install lmbench, get numbers. - Open up qemu monitor and type the following: stop migrate exec:dd of=~/save.vm quit - Start a new VM using qemu but add the following argument: -incoming exec:dd if=~/save.vm - Run performance test and compare. If performance measured is similar then we pass the test case. [Regression Potential] * The fix is a backport of two upstream patches: ad0b5321f1f797274603ebbe20108b0750baee94 211ea74022f51164a7729030b28eec90b6c99a08 - On patch allows QEMU to use THP if its enabled. + One patch allows QEMU to use THP if its enabled. The other patch changes logic to not memset pages to zero when loading memory for the vm (on an incoming migration). - * I've also run the qa-regression-testing test-qemu.py script and it passes all tests. + * I've also run the qa-regression-testing test-qemu.py script and it + passes all tests. + + [Additional Information] + + Kernels from 3.2 onwards are affected, and all have the config: + CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y. Therefore enabling THP is + applicable. + -- I have 2 physical hosts running Ubuntu Precise. With 1.0+noroms- 0ubuntu14.7 and qemu-kvm 1.2.0+noroms-0ubuntu7 (source from quantal, built for Precise with pbuilder.) I attempted to build qemu-1.3.0 debs from source to test, but libvirt seems to have an issue with it that I haven't been able to track down yet. I'm seeing a performance degradation after live migration on Precise, but not Lucid. These hosts are managed by libvirt (tested both 0.9.8-2ubuntu17 and 1.0.0-0ubuntu4) in conjunction with OpenNebula. I don't seem to have this problem with lucid guests (running a number of standard kernels, 3.2.5 mainline and backported linux- image-3.2.0-35-generic as well.) I first noticed this problem with phoronix doing compilation tests, and then tried lmbench where even simple calls experience performance degradation. I've attempted to post to the kvm mailing list, but so far the only suggestion was it may be related to transparent hugepages not being used after migration, but this didn't pan out. Someone else has a similar problem here - http://thread.gmane.org/gmane.comp.emulators.kvm.devel/100592 qemu command line example: /usr/bin/kvm -name one-2 -S -M pc-1.2 -cpu Westmere -enable-kvm -m 73728 -smp 16,sockets=2,cores=8,threads=1 -uuid f89e31a4-4945-c12c-6544-149ba0746c2f -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-2.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/one//datastores/0/2/disk.0,if=none,id=drive-virtio- disk0,format=raw,cache=none -device virtio-blk- pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio- disk0,bootindex=1 -drive file=/var/lib/one//datastores/0/2/disk.1,if=none,id=drive- ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive =drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net- pci,netdev=hostnet0,id=net0,mac=02:00:0a:64:02:fe,bus=pci.0,addr=0x3 -vnc 0.0.0.0:2,password -vga cirrus -incoming tcp:0.0.0.0:49155 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 Disk backend is LVM running on SAN via FC connection (using symlink from /var/lib/one/datastores/0/2/disk.0 above) ubuntu-12.04 - first boot == Simple syscall: 0.0527 microseconds Simple read: 0.1143 microseconds Simple write: 0.0953 microseconds Simple open/close: 1.0432 microseconds Using phoronix pts/compuational ImageMagick - 31.54s Linux Kernel 3.1 - 43.91s Mplayer - 30.49s PHP - 22.25s ubuntu-12.04 - post live migration == Simple syscall: 0.0621 microseconds Simple read: 0.2485 microseconds Simple write: 0.2252 microseconds Simple open/close: 1.4626 microseconds Using phoronix pts/compilation ImageMagick - 43.29s Linux Kernel 3.1 - 76.67s Mplayer - 45.41s PHP - 29.1s I
[Bug 1100843] Re: Live Migration Causes Performance Issues
** Changed in: qemu-kvm (Ubuntu) Status: Triaged = In Progress -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
From my testing this has been fixed in the saucy version (1.5.0) of qemu. It is fixed by this patch: f1c72795af573b24a7da5eb52375c9aba8a37972 However later in the history this commit was reverted, and again broke this. The other commit that fixes this is: 211ea74022f51164a7729030b28eec90b6c99a08 So 211ea740 needs to be backported to P/Q/R to fix this issue. I have a v1 packages of a precise backport here, I've confirmed performance differences between savevm/loadvm cycles: http://people.canonical.com/~arges/lp1100843/precise/ ** No longer affects: linux (Ubuntu) ** Also affects: qemu-kvm (Ubuntu Precise) Importance: Undecided Status: New ** Also affects: qemu-kvm (Ubuntu Quantal) Importance: Undecided Status: New ** Also affects: qemu-kvm (Ubuntu Raring) Importance: Undecided Status: New ** Also affects: qemu-kvm (Ubuntu Saucy) Importance: High Assignee: Chris J Arges (arges) Status: In Progress ** Changed in: qemu-kvm (Ubuntu Precise) Assignee: (unassigned) = Chris J Arges (arges) ** Changed in: qemu-kvm (Ubuntu Quantal) Assignee: (unassigned) = Chris J Arges (arges) ** Changed in: qemu-kvm (Ubuntu Raring) Assignee: (unassigned) = Chris J Arges (arges) ** Changed in: qemu-kvm (Ubuntu Precise) Importance: Undecided = High ** Changed in: qemu-kvm (Ubuntu Quantal) Importance: Undecided = High ** Changed in: qemu-kvm (Ubuntu Raring) Importance: Undecided = High ** Changed in: qemu-kvm (Ubuntu Saucy) Assignee: Chris J Arges (arges) = (unassigned) ** Changed in: qemu-kvm (Ubuntu Saucy) Status: In Progress = Fix Released ** Changed in: qemu-kvm (Ubuntu Raring) Status: New = Triaged ** Changed in: qemu-kvm (Ubuntu Quantal) Status: New = Triaged ** Changed in: qemu-kvm (Ubuntu Precise) Status: New = In Progress -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
** Changed in: qemu-kvm (Ubuntu) Assignee: (unassigned) = Chris J Arges (arges) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
This is being looked at in an upstream thread at http://lists.gnu.org/archive/html/qemu-devel/2013-07/msg01850.html Cheers, -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
We are reliably seeing this post live-migration on an openstack platform. Setup: hypervisor == Ubuntu 12.04.3 LTS libvirt === 1.0.2-0ubuntu11.13.04.2~cloud0 qemu-kvm === 1.0+noroms-0ubuntu14.10 storage: NFS exports Guest VM OS: Ubuntu 12.04.1 LTS and CentOS 6.4 We have ept enabled. Sample instance: domain type=kvm uuidf3c16d27-2586-44c8-b9d9-84b74b42b5d3/uuid nameinstance-0508/name memory4194304/memory vcpu2/vcpu os typehvm/type boot dev=hd/ /os features acpi/ /features clock offset=utc timer name=pit tickpolicy=delay/ timer name=rtc tickpolicy=catchup/ /clock cpu mode=host-model match=exact/ devices disk type=file device=disk driver name=qemu type=qcow2 cache=none/ source file=/var/lib/nova/instances/instance-0508/disk/ target bus=virtio dev=vda/ /disk interface type=bridge mac address=fa:16:3e:5d:0e:6a/ model type=virtio/ source bridge=qbrf43e9d83-56/ filterref filter=nova-instance-instance-0508-fa163e5d0e6a parameter name=IP value=10.253.138.156/ parameter name=DHCPSERVER value=10.253.138.51/ /filterref /interface serial type=file source path=/var/lib/nova/instances/instance-0508/console.log/ /serial serial type=pty/ input type=tablet bus=usb/ graphics type=vnc autoport=yes keymap=en-us listen=0.0.0.0/ /devices /domain We have a test environment and are willing to assist in debugging. Please let us know what we can do to help. Cheers, -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
My HyperDex cluster nodes performance dropped significantly after migrating them (virsh migrate --live ...).they are hosted on precise KVM (12.04.2 Precise Pangolin). first Google search result landed me on this page. it seems i'm not the only one who's encountering this problem. I hope this gets resolved soon as livemigration is a major feature for any hypervisor solution in my opinion ... cheers -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Can you please check if you have EPT enabled? This could be https://bugzilla.kernel.org/show_bug.cgi?id=58771 ** Bug watch added: Linux Kernel Bug Tracker #58771 http://bugzilla.kernel.org/show_bug.cgi?id=58771 -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Oops, I missed Chris's comment #28. Thanks. From comment #23, the 1.4 machine type seems to be fast, while 1.3 is slow. This doesn't make much sense, given the differences between the two machine types: enable_compat_apic_id_mode(); .driver = usb-tablet,\ .property = usb_version,\ .value= stringify(1),\ .driver = virtio-net-pci,\ .property = ctrl_mac_addr,\ .value= off, \ .driver = virtio-net-pci, \ .property = mq, \ .value= off, \ .driver = e1000,\ .property = autonegotiation,\ .value= off,\ This is why I suspected the issue was not 100% reproducible. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
@Paolo yes, when i was doing that testing i was able to consistently reproduce those results in #23, but it was a red herring, as of now i cannot reproduce the results in #23 consistently (i suspect it may have had something to do with the order i was executing tests but didn’t chase it any further). Yes, EPT enabled, I submitted that kernel bug in #30. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
** Changed in: linux (Ubuntu) Status: Incomplete = Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Update: From our testing this bug affects KVM Hypervisors on Intel processors that have the EPT feature enabled with Kernels 3.0 and greater. A list of Intel EPT supported CPUs here (http://ark.intel.com/Products/VirtualizationTechnology). When using a KVM Hypervisor Host with Linux kernel 3.0 or newer kernel with Intel EPT this bug shows itself. If the kvm_intel module is loaded with option ept=N guest performance is significantly decreased versus enabled, but it does maintain consistent performance pre and post restoration/migration. Exceptions: -A KVM Host with 2.6.32 or 2.6.39 Kernel with EPT enabled this bug is not triggered. -A KVM Host without the EPT feature enabled Intel CPU this bug is not triggered. -A KVM Host with Kernel 3.0+ and EPT kvm_intel module option disabled in this bug is not triggered A KVM hypervisor with EPT enabled on Linux Kernel 3.0 appears to be the key here. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
I used this handy tool to run system call preliminary benchmarks: http://code.google.com/p/byte-unixbench/ In a nutshell, what I found is a confirmation that live migration does indeed degrade performance on precise KVM. I hope the below results help narrow down this critical problem to eventually have it resolved in 12.04 LTS version. detail results: Compiled the benchmarking tool and then: root@sample-vm:~/UnixBench# ./Run syscall Output: ** before live-migration ** Benchmark Run: Wed May 01 2013 20:29:54 - 20:32:04 1 CPU in system; running 1 parallel copy of tests System Call Overhead4177612.4 lps (10.0 s, 7 samples) System Benchmarks Partial Index BASELINE RESULTINDEX System Call Overhead 15000.04177612.4 2785.1 System Benchmarks Index Score (Partial Only) 2785.1 ** after live-migration ** Benchmark Run: Wed May 01 2013 20:35:16 - 20:37:26 1 CPU in system; running 1 parallel copy of tests System Call Overhead3065118.3 lps (10.0 s, 7 samples) System Benchmarks Partial Index BASELINE RESULTINDEX System Call Overhead 15000.03065118.3 2043.4 System Benchmarks Index Score (Partial Only) 2043.4 XML domain dump: memory1048576/memory currentMemory1048576/currentMemory vcpu1/vcpu cputune shares1024/shares /cputune os type arch='x86_64' machine='pc-1.0'hvm/type boot dev='hd'/ /os features acpi/ /features clock offset='utc'/ on_poweroffdestroy/on_poweroff on_rebootrestart/on_reboot on_crashdestroy/on_crash devices emulator/usr/bin/kvm/emulator disk type='file' device='disk' driver name='qemu' type='qcow2'/ source file='HIDEME'/ target dev='vda' bus='virtio'/ alias name='virtio-disk0'/ address type='pci' domain='0x' bus='0x00' slot='0x04' function='0x0'/ /disk disk type='file' device='cdrom' driver name='qemu' type='raw'/ source file='HIDEME'/ target dev='hda' bus='ide'/ readonly/ alias name='ide0-0-0'/ address type='drive' controller='0' bus='0' unit='0'/ /disk controller type='ide' index='0' alias name='ide0'/ address type='pci' domain='0x' bus='0x00' slot='0x01' function='0x1'/ /controller -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
I have a few VMs (precise) that process high-volume transaction jobs each night. After I've done a live-migrate operation to replace faulty power supply on a bare-metal server, we encountered sluggish performance on the migrated VMs, significant higher CPU is recorded in particular, where the same nightly job would consume way more CPU and took more time to finish on identical hardware. Upon investigation, we noticed that the only change introduced was the live migrate operation. Upon rebooting the guest OS of the VMs, the performance is back to normal. I suspect we're hitting the same problem as the one filed here.. I will attempt to run lmbench next to see if I would notice similar behavior on system calls costs as the one recorded in comments #19..21,23. -- Latest KVM is used from Ubuntu 12.04 LTS :: qemu-kvm (1.0+noroms-0ubuntu14.8) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
** Also affects: qemu Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
The results of comment 23 suggest that the issue is not 100% reproducible. Can you please run the benchmark 3-4 times (presave/postrestore) and showall 4 results? One benchmark only, e.g. simple read will do. Also please try putting a big file on disk (something like dd if=/dev/zero of=bigfile count=64K bs=64K) and then doing cat bigfile /dev/null after restoring. Please check if that makes performance more consistent. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Can you clarify what's not 100% reproducible? The only time that it is not reproducible on my system is between different qemu machine types as I listed. If tests are performed on same machine-type they are reproducible 100% of the time on the same host and vm guest as shown in comment #23. I have re-run what your requesting for machine type pc-1.0. ---machine type pc-1.0--- -Presave- Simple read: 0.1273 microseconds Simple read: 0.1259 microseconds Simple read: 0.1270 microseconds Simple read: 0.1268 microseconds -postrestore- performing: dd if=/dev/zero of=bigfile count=32K bs=64K 32768+0 records in 32768+0 records out 2147483648 bytes (2.1 GB) copied, 15.2912 s, 140 MB/s performing: cat bigfile /dev/null Simple read: 0.2700 microseconds Simple read: 0.2736 microseconds Simple read: 0.2713 microseconds Simple read: 0.2747 microseconds -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
I did some tests using Raring Server Beta 2. There are some interesting results for this test the results are mixing. Using different machine types produces different results. At this time i've only ran these simple lat_syscall tests from lmbench and haven't run some of the more exhaustive benchmarks or compilations. Summary of the lat_syscall results using different machine types of qemu-kvm: pc-1.0 - affected pc-1.1 - normal pc-1.2 - affected pc-1.3 - affected pc-i440fx-1.4 - normal Test Setup Hypervisor OS: Raring Beta2 qemu: 1.4.0+dfsg-1expubuntu4 Kernel: 3.8.0-16-generic Guest: OS: Ubuntu 12.04.02 Minimal install using ubuntu-vmbuilder LMBench results for lat syscall tests ---machine type pc-1.0--- -PRESAVE- Simple read: 0.1301 microseconds Simple write: 0.1052 microseconds Simple open/close: 1.3819 microseconds -POSTRESTORE- Simple read: 0.2699 microseconds Simple write: 0.2401 microseconds Simple open/close: 1.4112 microseconds ---machine type pc-1.1--- -PRESAVE- Simple read: 0.1265 microseconds Simple write: 0.1033 microseconds Simple open/close: 1.2974 microseconds -POSTRESTORE- Simple read: 0.1266 microseconds Simple write: 0.1042 microseconds Simple open/close: 1.2093 microseconds ---machine type pc-1.2--- -PRESAVE- Simple syscall: 0.0484 microseconds Simple read: 0.1291 microseconds Simple write: 0.1042 microseconds Simple open/close: 1.2501 microseconds -POSTRESTORE- Simple syscall: 0.0485 microseconds Simple read: 0.2737 microseconds Simple write: 0.2414 microseconds Simple open/close: 1.3953 microseconds ---machine type pc-1.3--- -PRESAVE- Simple read: 0.1276 microseconds Simple write: 0.1101 microseconds Simple open/close: 1.1501 microseconds -POSTRESTORE- Simple read: 0.2717 microseconds Simple write: 0.2415 microseconds Simple open/close: 1.3997 microseconds ---machine type pc-i440fx-1.4--- -PRESAVE- Simple read: 0.1291 microseconds Simple write: 0.1039 microseconds Simple open/close: 1.2344 microseconds -POSTRESTORE- Simple read: 0.1335 microseconds Simple write: 0.1080 microseconds Simple open/close: 1.0453 microseconds -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
@serge-hallyn I'm using Netapp filer in c-mode for NFS storage, mount options are: (rw,nosuid,nodev,noatime,hard,nfsvers=3,tcp,intr,rsize=32768,wsize=32768,addr=x.x.x.x). However, I can reproduce this on a a host with or without NFS, using local disk, qcow2, or raw images and the OP was using FC SAN. I've tried to reproduce your success by buiding a new VM the latest preciese 12.04.02 with updates, but i have been unsuccessful. It happens immediately after the restore 100% of the time. The quickest test i've found is this, it helps aid in your reproduction as it's an immediate indication. -Get/Compile lmbench -build it -run # touch tmpfile # for i in null read write open ; do sleep 1; ./lat_syscall -N 1 $i tmpfile; done - the latencies for read/write generally increase up to 2x, and the open/close by about 30% post restore. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Can you reproduce this with 1 or 2 cores and 1G ram? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 1100843] Re: Live Migration Causes Performance Issues
Quoting Chris Cormier (ccorm...@gmail.com): @serge-hallyn I'm using Netapp filer in c-mode for NFS storage, mount options are: (rw,nosuid,nodev,noatime,hard,nfsvers=3,tcp,intr,rsize=32768,wsize=32768,addr=x.x.x.x). However, I can reproduce this on a a host with or without NFS, using local disk, qcow2, or raw images and the OP was using FC SAN. I've tried to reproduce your success by buiding a new VM the latest preciese 12.04.02 with updates, but i have been unsuccessful. It happens immediately after the restore 100% of the time. The quickest test i've found is this, it helps aid in your reproduction as it's an immediate indication. -Get/Compile lmbench -build it -run # touch tmpfile # for i in null read write open ; do sleep 1; ./lat_syscall -N 1 $i tmpfile; done - the latencies for read/write generally increase up to 2x, and the open/close by about 30% post restore. i did this and got the attached results (.1 before save, .2 after restoer). This is with precise guest on precise host over nfs (from a raring nfs server). results.1 (before save) Simple syscall: 0.1468 microseconds Simple read: 0.3555 microseconds Simple write: 0.2785 microseconds Simple open/close: 3.3682 microseconds serge@p1:~/lmbench3$ ./sergetest Simple syscall: 0.1466 microseconds Simple read: 0.3412 microseconds Simple write: 0.2813 microseconds Simple open/close: 3.3175 microseconds serge@p1:~/lmbench3$ ./sergetest Simple syscall: 0.1582 microseconds Simple read: 0.3403 microseconds Simple write: 0.2871 microseconds Simple open/close: 2.7587 microseconds serge@p1:~/lmbench3$ ./sergetest Simple syscall: 0.1453 microseconds Simple read: 0.3371 microseconds Simple write: 0.2790 microseconds Simple open/close: 3.3391 microseconds results.2 (after restore) Simple syscall: 0.1457 microseconds Simple read: 0.3370 microseconds Simple write: 0.2832 microseconds Simple open/close: 3.1675 microseconds Simple syscall: 0.1470 microseconds Simple read: 0.3436 microseconds Simple write: 0.2812 microseconds Simple open/close: 2.8002 microseconds Simple syscall: 0.1452 microseconds Simple read: 0.3428 microseconds Simple write: 0.2817 microseconds Simple open/close: 2.7974 microseconds Simple syscall: 0.1456 microseconds Simple read: 0.3722 microseconds Simple write: 0.2798 microseconds Simple open/close: 2.7494 microseconds Simple syscall: 0.1470 microseconds Simple read: 0.3362 microseconds Simple write: 0.2830 microseconds Simple open/close: 2.7640 microsecond -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Could you confirm that your .1 tests were on a freshly booted Guest OS? Our hardware likely different... but your latencies are close to my Post Restore times. I just reproded with 1GB RAM and Single CPU. -Pre Save- Simple syscall: 0.0519 microseconds Simple read: 0.1356 microseconds Simple write: 0.1086 microseconds Simple open/close: 1.0265 microseconds local@cc-precise120402-raw:~/lmbench-3.0-a9/bin/x86_64-linux-gnu$ -Post Restore- Simple syscall: 0.0500 microseconds Simple read: 0.2125 microseconds Simple write: 0.1913 microseconds Simple open/close: 1.4482 microseconds local@cc-precise120402-raw:~/lmbench-3.0-a9/bin/x86_64-linux-gnu$ -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 1100843] Re: Live Migration Causes Performance Issues
Quoting C Cormier (1100...@bugs.launchpad.net): Could you confirm that your .1 tests were on a freshly booted Guest OS? Yup, it was a fresh boot. Since the vms are on shared storage, do you have a box you could wire up running raring, to test more recent qemu? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
I couldn't reproduce this with a vm whose xml includes: memory524288/memory currentMemory524288/currentMemory vcpu1/vcpu os type arch='x86_64' machine='pc-1.2'hvm/type boot dev='hd'/ /os features acpi/ apic/ pae/ /features clock offset='utc'/ on_poweroffdestroy/on_poweroff on_rebootrestart/on_reboot on_crashrestart/on_crash devices emulator/usr/bin/kvm/emulator disk type='file' device='disk' driver name='qemu' type='raw'/ source file='/home/serge/machines/clean-precise-mini-amd64/disk0.img'/ target dev='vda' bus='virtio'/ address type='pci' domain='0x' bus='0x00' slot='0x04' function='0x0'/ /disk I started the vm, ran 6 kernel compilations, did 'virsh save/virsh restore', then did 6 more kernel compilations. All were in 272 or 273 seconds. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
I guess the final thing to test is virsh save/restore with nfs backend. Can you tell me the configuration of the nfs server? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
So far I've run the equivalentn of test 1 in comment #14, and also didn't find any performance degradation. I left the host crunching a few other VMs for several hours, but performance on a kernel compilation stayed the same. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
** Changed in: qemu-kvm (Ubuntu) Importance: Medium = High ** Changed in: qemu-kvm (Ubuntu) Status: Confirmed = Triaged -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Few updates from a few tests we ran: Setup: hypervisor == Ubuntu 12.04.2 LTS libvirt === 0.9.8-2ubuntu17.7 qemu-kvm === 1.0+noroms-0ubuntu14.7 storage: NFS exports Guest VM OS: Ubuntu 12.04.1 LTS Test 1: Created a new VM and kept it Idle for 60+ hours, then ran Unix benchmark test against it. (NO Live-migration/migration) Result 1: The performance was NOT affected and the result was the same as a fresh VM. Test 2: Created a new VM, and perform a virsh save domain file followed by a virsh restore file on the SAME hypervisor. Then run unix benchmark against it. Result 2: The VM is AFFECTED (cost to perform system calls degraded by ~21% ) Test 3: Rebooted the affected VM from test (2) and re-ran unix-benchmark tests against it. Result 3: Performance is back to normal and is identical to a newly created and non-migrated/not-saved VM. Thus, it seems the problem is also exposed by saving the running precise domain (memory pages) to state file followed by a restore even on the same hypervisor and not necessarily live-migration to a different host. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
We are also getting reports from some OpenNebula users with this very same issue. Is there any extra information about about what is causing the slowdown or a fix? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Can anyone confirm if you similar slowdowns if you leave the VM running for a few days? I thought it was related to live migration, but I saw my performance degrade if the VM/physical host was up and idle for a couple days. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
I've been able to confirm the same as the OP regarding the different Ubuntu distrubutions as guests. These tests should help ruleout/pinpoint the kernel and modules. Using the same 12.04 hypervisors for all tests. Testing different guests I was able to determine. -Lucid with default kernel is NOT affected. -Precise with default kernel is affected. The test... I tared up each guest kernels and dropped them into the oposing distribution. Results -Lucid with Precise Kernel is NOT affected. -Precise with Lucid Kernel is affected. Conclusion 12.04 Precise distribution guest is affected regardless of the kernel used. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
@ccormier I've thought all along it might be a libc issue but testing libc 2.13 on precise would be rather difficult. To some extent I feel like this rules out the kernel as an issue though, since the same kernel on precise/lucid yield different results. Have you tried letting a precise VM idle without a livemigration to see if performance degrades? If not perhaps you could leave one idle over the weekend and performance test on Monday? I assume you're testing with qemu-kvm 1.0.0? I've been testing with qemu-kvm 1.2.0 as the performance is remarkably better for me. This would seem to indicate it's not qemu-kvm at fault either. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Same thing happened with us with running ubuntu precise kvm for our hypervisors. VMs that are live-migrated suffer noticeable performance degradation. We tried a few performance tests against live-migrated vs non-migrated VMs and the problem is easily reproducible.. (we used jmeter) Ubuntu (hypervisor): 12.04.1 LTS libvirt:0.9.8-2ubuntu17.4 qemu-kvm:1.0+noroms-0ubuntu14.3 storage: NFS exports. Guest OS: Ubuntu 12.04.1 RHEL 5.5 -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
I tested with qemu-kvm 1.3.0. It seems that the issue still exists, but that it exists without a live migration if you wait long enough. That is if you start a VM on one node and run phoronix batch-run pts/compilation, wait 4 hours (with the VM and physical host doing nothing else) an re-run the test you'll get results on the VM similar to if you run the test and then live migrate to a new host. I have no idea what's causing this behavior, but it seems to be reproducible. For now this can probably be closed. I'll resubmit a new bug (possibly upsteam) if I can figure out how to get more details to help properly diagnose how/when/why the VM's slow down over time. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
It doesn't sound like this bug should be removed, rather re-titled. Are there any messages in syslog about expecting performance degradation? Have you tried to reproduce this with and without hugepages? Can you reproduce the same thing with a simple local raw file or LVM backend? Can you give the precise commands you use to start the tests, so it can be reproduced by others? ** Changed in: qemu-kvm (Ubuntu) Status: Triaged = Incomplete -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
(marking incomplete pending response. In addition to retitling, I think the bug should also be targeted to project QEMU and qemu Ubuntu source package) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
There's nothing in syslog for the VM or host that would imply performance degradation. I have done this with hugepages and made sure huge page use was consistent. Previously I disabled hugepages and didn't see a difference but I haven't tested again. I'm using (C)LVM back off FCoE/SAN but I haven't tried local LVM/qcow2 type backing. I'm using libvirt, but the command line ends up looking like this, if you would like I can provide XML for libvirt as well. /usr/bin/kvm -name one-10 -S -M pc-1.3 -cpu Westmere -enable-kvm -m 73728 -smp 16,sockets=2,cores=8,threads=1 -uuid 5ee0afd3-df3f-fb1f-02bd- 7cde2bc4ee95 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-10.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-shutdown -device piix3-usb- uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/one//datastores/0/10/disk.0,if=none,id=drive-virtio- disk0,format=raw,cache=none -device virtio-blk- pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio- disk0,bootindex=1 -drive file=/var/lib/one//datastores/0/10/disk.1,if=none,id=drive- ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive =drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=23,id=hostnet0 -device virtio-net- pci,netdev=hostnet0,id=net0,mac=02:00:0a:64:02:fd,bus=pci.0,addr=0x3 -vnc 0.0.0.0:10,password -vga cirrus -device virtio-balloon- pci,id=balloon0,bus=pci.0,addr=0x5 I'll try experimenting more with THP disabled and different IO backends. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Launchpad had previously marked this confirmed for affecting several users. I'm curious who else has seen this behavior, and under what circumstances? ** Changed in: qemu-kvm (Ubuntu) Status: Incomplete = New ** Changed in: qemu-kvm (Ubuntu) Status: New = Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
I don't see qemu-kvm 1.3.0 yet. Will test when you get it pushed, hopefully Tuesday (01/22/2013) if you've pushed by then. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: qemu-kvm (Ubuntu) Status: New = Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Thanks, that is very interesting. I'll be pushing qemu 1.3.0 to ubuntu raring hopefully tomorrow - it would be interesting to know if this still happens there. ** Changed in: qemu-kvm (Ubuntu) Importance: Undecided = Medium ** Changed in: qemu-kvm (Ubuntu) Status: Confirmed = Triaged -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
