[Bug 1100843] Re: Live Migration Causes Performance Issues
@Paolo yes, when i was doing that testing i was able to consistently reproduce those results in #23, but it was a red herring, as of now i cannot reproduce the results in #23 consistently (i suspect it may have had something to do with the order i was executing tests but didn’t chase it any further). Yes, EPT enabled, I submitted that kernel bug in #30. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
@Paolo yes, when i was doing that testing i was able to consistently reproduce those results in #23, but it was a red herring, as of now i cannot reproduce the results in #23 consistently (i suspect it may have had something to do with the order i was executing tests but didn’t chase it any further). Yes, EPT enabled, I submitted that kernel bug in #30. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Update: From our testing this bug affects KVM Hypervisors on Intel processors that have the EPT feature enabled with Kernels 3.0 and greater. A list of Intel EPT supported CPUs here (http://ark.intel.com/Products/VirtualizationTechnology). When using a KVM Hypervisor Host with Linux kernel 3.0 or newer kernel with Intel EPT this bug shows itself. If the kvm_intel module is loaded with option ept=N guest performance is significantly decreased versus enabled, but it does maintain consistent performance pre and post restoration/migration. Exceptions: -A KVM Host with 2.6.32 or 2.6.39 Kernel with EPT enabled this bug is not triggered. -A KVM Host without the EPT feature enabled Intel CPU this bug is not triggered. -A KVM Host with Kernel 3.0+ and EPT kvm_intel module option disabled in this bug is not triggered A KVM hypervisor with EPT enabled on Linux Kernel 3.0 appears to be the key here. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Update: From our testing this bug affects KVM Hypervisors on Intel processors that have the EPT feature enabled with Kernels 3.0 and greater. A list of Intel EPT supported CPUs here (http://ark.intel.com/Products/VirtualizationTechnology). When using a KVM Hypervisor Host with Linux kernel 3.0 or newer kernel with Intel EPT this bug shows itself. If the kvm_intel module is loaded with option ept=N guest performance is significantly decreased versus enabled, but it does maintain consistent performance pre and post restoration/migration. Exceptions: -A KVM Host with 2.6.32 or 2.6.39 Kernel with EPT enabled this bug is not triggered. -A KVM Host without the EPT feature enabled Intel CPU this bug is not triggered. -A KVM Host with Kernel 3.0+ and EPT kvm_intel module option disabled in this bug is not triggered A KVM hypervisor with EPT enabled on Linux Kernel 3.0 appears to be the key here. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Can you clarify what's not 100% reproducible? The only time that it is not reproducible on my system is between different qemu machine types as I listed. If tests are performed on same machine-type they are reproducible 100% of the time on the same host and vm guest as shown in comment #23. I have re-run what your requesting for machine type pc-1.0. ---machine type pc-1.0--- -Presave- Simple read: 0.1273 microseconds Simple read: 0.1259 microseconds Simple read: 0.1270 microseconds Simple read: 0.1268 microseconds -postrestore- performing: dd if=/dev/zero of=bigfile count=32K bs=64K 32768+0 records in 32768+0 records out 2147483648 bytes (2.1 GB) copied, 15.2912 s, 140 MB/s performing: cat bigfile /dev/null Simple read: 0.2700 microseconds Simple read: 0.2736 microseconds Simple read: 0.2713 microseconds Simple read: 0.2747 microseconds -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Can you clarify what's not 100% reproducible? The only time that it is not reproducible on my system is between different qemu machine types as I listed. If tests are performed on same machine-type they are reproducible 100% of the time on the same host and vm guest as shown in comment #23. I have re-run what your requesting for machine type pc-1.0. ---machine type pc-1.0--- -Presave- Simple read: 0.1273 microseconds Simple read: 0.1259 microseconds Simple read: 0.1270 microseconds Simple read: 0.1268 microseconds -postrestore- performing: dd if=/dev/zero of=bigfile count=32K bs=64K 32768+0 records in 32768+0 records out 2147483648 bytes (2.1 GB) copied, 15.2912 s, 140 MB/s performing: cat bigfile /dev/null Simple read: 0.2700 microseconds Simple read: 0.2736 microseconds Simple read: 0.2713 microseconds Simple read: 0.2747 microseconds -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1100843/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
I did some tests using Raring Server Beta 2. There are some interesting results for this test the results are mixing. Using different machine types produces different results. At this time i've only ran these simple lat_syscall tests from lmbench and haven't run some of the more exhaustive benchmarks or compilations. Summary of the lat_syscall results using different machine types of qemu-kvm: pc-1.0 - affected pc-1.1 - normal pc-1.2 - affected pc-1.3 - affected pc-i440fx-1.4 - normal Test Setup Hypervisor OS: Raring Beta2 qemu: 1.4.0+dfsg-1expubuntu4 Kernel: 3.8.0-16-generic Guest: OS: Ubuntu 12.04.02 Minimal install using ubuntu-vmbuilder LMBench results for lat syscall tests ---machine type pc-1.0--- -PRESAVE- Simple read: 0.1301 microseconds Simple write: 0.1052 microseconds Simple open/close: 1.3819 microseconds -POSTRESTORE- Simple read: 0.2699 microseconds Simple write: 0.2401 microseconds Simple open/close: 1.4112 microseconds ---machine type pc-1.1--- -PRESAVE- Simple read: 0.1265 microseconds Simple write: 0.1033 microseconds Simple open/close: 1.2974 microseconds -POSTRESTORE- Simple read: 0.1266 microseconds Simple write: 0.1042 microseconds Simple open/close: 1.2093 microseconds ---machine type pc-1.2--- -PRESAVE- Simple syscall: 0.0484 microseconds Simple read: 0.1291 microseconds Simple write: 0.1042 microseconds Simple open/close: 1.2501 microseconds -POSTRESTORE- Simple syscall: 0.0485 microseconds Simple read: 0.2737 microseconds Simple write: 0.2414 microseconds Simple open/close: 1.3953 microseconds ---machine type pc-1.3--- -PRESAVE- Simple read: 0.1276 microseconds Simple write: 0.1101 microseconds Simple open/close: 1.1501 microseconds -POSTRESTORE- Simple read: 0.2717 microseconds Simple write: 0.2415 microseconds Simple open/close: 1.3997 microseconds ---machine type pc-i440fx-1.4--- -PRESAVE- Simple read: 0.1291 microseconds Simple write: 0.1039 microseconds Simple open/close: 1.2344 microseconds -POSTRESTORE- Simple read: 0.1335 microseconds Simple write: 0.1080 microseconds Simple open/close: 1.0453 microseconds -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
I did some tests using Raring Server Beta 2. There are some interesting results for this test the results are mixing. Using different machine types produces different results. At this time i've only ran these simple lat_syscall tests from lmbench and haven't run some of the more exhaustive benchmarks or compilations. Summary of the lat_syscall results using different machine types of qemu-kvm: pc-1.0 - affected pc-1.1 - normal pc-1.2 - affected pc-1.3 - affected pc-i440fx-1.4 - normal Test Setup Hypervisor OS: Raring Beta2 qemu: 1.4.0+dfsg-1expubuntu4 Kernel: 3.8.0-16-generic Guest: OS: Ubuntu 12.04.02 Minimal install using ubuntu-vmbuilder LMBench results for lat syscall tests ---machine type pc-1.0--- -PRESAVE- Simple read: 0.1301 microseconds Simple write: 0.1052 microseconds Simple open/close: 1.3819 microseconds -POSTRESTORE- Simple read: 0.2699 microseconds Simple write: 0.2401 microseconds Simple open/close: 1.4112 microseconds ---machine type pc-1.1--- -PRESAVE- Simple read: 0.1265 microseconds Simple write: 0.1033 microseconds Simple open/close: 1.2974 microseconds -POSTRESTORE- Simple read: 0.1266 microseconds Simple write: 0.1042 microseconds Simple open/close: 1.2093 microseconds ---machine type pc-1.2--- -PRESAVE- Simple syscall: 0.0484 microseconds Simple read: 0.1291 microseconds Simple write: 0.1042 microseconds Simple open/close: 1.2501 microseconds -POSTRESTORE- Simple syscall: 0.0485 microseconds Simple read: 0.2737 microseconds Simple write: 0.2414 microseconds Simple open/close: 1.3953 microseconds ---machine type pc-1.3--- -PRESAVE- Simple read: 0.1276 microseconds Simple write: 0.1101 microseconds Simple open/close: 1.1501 microseconds -POSTRESTORE- Simple read: 0.2717 microseconds Simple write: 0.2415 microseconds Simple open/close: 1.3997 microseconds ---machine type pc-i440fx-1.4--- -PRESAVE- Simple read: 0.1291 microseconds Simple write: 0.1039 microseconds Simple open/close: 1.2344 microseconds -POSTRESTORE- Simple read: 0.1335 microseconds Simple write: 0.1080 microseconds Simple open/close: 1.0453 microseconds -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Could you confirm that your .1 tests were on a freshly booted Guest OS? Our hardware likely different... but your latencies are close to my Post Restore times. I just reproded with 1GB RAM and Single CPU. -Pre Save- Simple syscall: 0.0519 microseconds Simple read: 0.1356 microseconds Simple write: 0.1086 microseconds Simple open/close: 1.0265 microseconds local@cc-precise120402-raw:~/lmbench-3.0-a9/bin/x86_64-linux-gnu$ -Post Restore- Simple syscall: 0.0500 microseconds Simple read: 0.2125 microseconds Simple write: 0.1913 microseconds Simple open/close: 1.4482 microseconds local@cc-precise120402-raw:~/lmbench-3.0-a9/bin/x86_64-linux-gnu$ -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1100843] Re: Live Migration Causes Performance Issues
Could you confirm that your .1 tests were on a freshly booted Guest OS? Our hardware likely different... but your latencies are close to my Post Restore times. I just reproded with 1GB RAM and Single CPU. -Pre Save- Simple syscall: 0.0519 microseconds Simple read: 0.1356 microseconds Simple write: 0.1086 microseconds Simple open/close: 1.0265 microseconds local@cc-precise120402-raw:~/lmbench-3.0-a9/bin/x86_64-linux-gnu$ -Post Restore- Simple syscall: 0.0500 microseconds Simple read: 0.2125 microseconds Simple write: 0.1913 microseconds Simple open/close: 1.4482 microseconds local@cc-precise120402-raw:~/lmbench-3.0-a9/bin/x86_64-linux-gnu$ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1100843 Title: Live Migration Causes Performance Issues To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1100843/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
