[Bug 1341195] Re: KVM guests getting slow by time
I believe I've found the fix for this issue for 3.13. If you can, please test the kernel posted on comment #1 on this bug: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917 Make sure KSM is enabled; and any workarounds for this bug are disabled. If this fixes the issue for you, you are welcome to mark this bug as a duplicate of 1346917. Thanks! ** No longer affects: qemu (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1341195 Title: KVM guests getting slow by time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1341195/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1341195] Re: KVM guests getting slow by time
** Tags added: ksm-numa-guest-perf -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1341195 Title: KVM guests getting slow by time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1341195/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1341195] Re: KVM guests getting slow by time
I think we can be sure, that KSM_ENABLED=0 definitely helps. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1341195 Title: KVM guests getting slow by time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1341195/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1341195] Re: KVM guests getting slow by time
FYI, I see no issue since I set KSM_ENABLED=0 (1 day). -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1341195 Title: KVM guests getting slow by time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1341195/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 1341195] Re: KVM guests getting slow by time
On 07/14/2014 05:13 PM, Serge Hallyn wrote: Since switching to saucy kernel is a good workaround, marking this as affecting the kernel. Could you try disabling KSM and see if that avoids the issue? I set KSM_ENABLED=0 in /etc/default/qemu-kvm and restarted the server. Also today after 3 weeks it started the same thing with Saucy kernel too. tamas -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1341195 Title: KVM guests getting slow by time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1341195/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1341195] Re: KVM guests getting slow by time
Hi Tamas, To be clear, you have not yet gotten the bug since you've set KSM_ENABLED to 0, or have you? Interesting about the saucy kernel. Ideally we could reproduce this reliably enough to really bisect. I wonder if we can reproduce without qemu. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1341195 Title: KVM guests getting slow by time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1341195/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 1341195] Re: KVM guests getting slow by time
On 07/16/2014 06:35 PM, Serge Hallyn wrote: To be clear, you have not yet gotten the bug since you've set KSM_ENABLED to 0, or have you? I just did it, so no, not yet. This happened by time: Saucy (everything is fine) - Trusty (issue happened) - Trusty with Saucy kernel 3 weeks ago (no issue) - today with 3 weeks uptime (issue happened) - KSM_ENABLED=0 + reboot with Trusty kernel - now I just want to be clear, that both of us understand the same on this. Interesting about the saucy kernel. Ideally we could reproduce this reliably enough to really bisect. I wonder if we can reproduce without qemu. I have a some other servers with LXC with no issue, If there is KVM, then issue happens. Does it really not show up anywhere else? sysctl.conf: kernel.printk = 3 4 1 3 net.ipv4.conf.default.rp_filter=1 net.ipv4.conf.all.rp_filter=1 net.ipv4.tcp_syncookies=1 net.ipv4.conf.all.secure_redirects = 1 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.all.accept_source_route = 0 net.ipv6.conf.all.accept_source_route = 0 kernel.shmmax=33554432 net.ipv4.tcp_rmem = 4096 16777216 33554432 net.ipv4.tcp_wmem = 4096 16777216 33554432 net.ipv4.tcp_mem = 4096 16777216 33554432 net.core.rmem_default = 16777216 net.core.wmem_default = 16777216 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.core.netdev_max_backlog = 3 net.core.netdev_max_backlog = 3 net.ipv6.neigh.default.gc_thresh1 = 512 net.ipv6.neigh.default.gc_thresh2 = 2048 net.ipv6.neigh.default.gc_thresh3 = 4096 vm.swappiness = 1 fs.inotify.max_user_watches = 81920 fs.inotify.max_user_instances = 1024 May that be important? tamas -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1341195 Title: KVM guests getting slow by time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1341195/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 1341195] Re: KVM guests getting slow by time
Quoting Tamas Papp (1341...@bugs.launchpad.net): On 07/16/2014 06:35 PM, Serge Hallyn wrote: To be clear, you have not yet gotten the bug since you've set KSM_ENABLED to 0, or have you? I just did it, so no, not yet. This happened by time: Saucy (everything is fine) - Trusty (issue happened) - Trusty with Saucy kernel 3 weeks ago (no issue) - today with 3 weeks uptime (issue happened) - KSM_ENABLED=0 + reboot with Trusty kernel - now I just want to be clear, that both of us understand the same on this. Interesting about the saucy kernel. Ideally we could reproduce this reliably enough to really bisect. I wonder if we can reproduce without qemu. I have a some other servers with LXC with no issue, If there is KVM, then issue happens. Does it really not show up anywhere else? I've personally not seen it, and noone on my team, who use a lot of kvm instances, has seen it. Our two current theories are that (a) it has to do with ksm page migration across numa-nodes, or (b) it has to do with a race with transparent hugepages versus ksm. For (b) there is already a commit in Linus' tree (f72e7dcd). For (a), you can probably test by setting /sys/kernel/mm/ksm/merge_across_nodes to 0. (We are trying these as well, but as the testcases are not 100% positive it'd be good to have you try as well) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1341195 Title: KVM guests getting slow by time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1341195/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 1341195] Re: KVM guests getting slow by time
On 07/16/2014 09:39 PM, Serge Hallyn wrote I've personally not seen it, and noone on my team, who use a lot of kvm instances, has seen it. Our two current theories are that (a) it has to do with ksm page migration across numa-nodes, or (b) it has to do with a race with transparent hugepages versus ksm. For (b) there is already a commit in Linus' tree (f72e7dcd). For (a), you can probably Are you referring to this or a different one? https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1323165 test by setting /sys/kernel/mm/ksm/merge_across_nodes to 0. (We are trying these as well, but as the testcases are not 100% positive it'd be good to have you try as well) It's 1 now. Do you you want me to set it now or wait to the result of KSM_ENABLED=0? I'm not familiar enough with these settings. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1341195 Title: KVM guests getting slow by time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1341195/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 1341195] Re: KVM guests getting slow by time
Quoting Tamas Papp (1341...@bugs.launchpad.net): On 07/16/2014 09:39 PM, Serge Hallyn wrote I've personally not seen it, and noone on my team, who use a lot of kvm instances, has seen it. Our two current theories are that (a) it has to do with ksm page migration across numa-nodes, or (b) it has to do with a race with transparent hugepages versus ksm. For (b) there is already a commit in Linus' tree (f72e7dcd). For (a), you can probably Are you referring to this or a different one? https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1323165 No, this commit is commit f72e7dcdd25229446b102e587ef2f826f76bff28 Author: Hugh Dickins hu...@google.com Date: Mon Jun 23 13:22:05 2014 -0700 mm: let mm_find_pmd fix buggy race with THP fault which seems later than that bug. test by setting /sys/kernel/mm/ksm/merge_across_nodes to 0. (We are trying these as well, but as the testcases are not 100% positive it'd be good to have you try as well) It's 1 now. Do you you want me to set it now or wait to the result of KSM_ENABLED=0? Oh - I forgot you had KSM_ENABLED=0 :) there's no sense changing merge_across_nodes in the meantime. (To be sure - /sys/kernel/mm/ksm/run is in fact set to 0 now right?) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1341195 Title: KVM guests getting slow by time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1341195/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 1341195] Re: KVM guests getting slow by time
On 07/16/2014 10:24 PM, Serge Hallyn wrote: (To be sure - /sys/kernel/mm/ksm/run is in fact set to 0 now right?) Right. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1341195 Title: KVM guests getting slow by time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1341195/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1341195] Re: KVM guests getting slow by time
Since switching to saucy kernel is a good workaround, marking this as affecting the kernel. Could you try disabling KSM and see if that avoids the issue? ** Also affects: linux (Ubuntu) Importance: Undecided Status: New ** Changed in: linux (Ubuntu) Status: New = Incomplete ** Changed in: qemu (Ubuntu) Status: New = Incomplete -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1341195 Title: KVM guests getting slow by time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1341195/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
