[Kernel-packages] [Bug 503138] Re: Lucid & Natty, KVM, After kernel message hrtimer: interrupt too slow.... the SMP kvm guest becomes slow.
[Expired for qemu (Ubuntu) because there has been no activity for 60 days.] ** Changed in: qemu (Ubuntu) Status: Incomplete => Expired -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/503138 Title: Lucid & Natty, KVM, After kernel message hrtimer: interrupt too slow the SMP kvm guest becomes slow. Status in linux package in Ubuntu: Expired Status in qemu package in Ubuntu: Expired Bug description: KVM Host is running a clean 9.10 server install with just qemu-kvm and virt-manager kernel=2.6.31-17-generic uname -a: Linux VMMASTER 2.6.31-17-generic #54-Ubuntu SMP Thu Dec 10 16:20:31 UTC 2009 i686 GNU/Linux version signature: Ubuntu 2.6.31-17.54-generic KVM Guest is running a clean 9.10 server install with some userland services (apache/postfix/whatnot) kernel=2.6.31-16-generic-pae (from the linux-image-virtual package) uname -a: Linux VM1 2.6.31-16-generic-pae #53-Ubuntu SMP Tue Dec 8 05:20:21 UTC 2009 i686 GNU/Linux version signature: Ubuntu 2.6.31-16.53-generic-pae KVM guest startup command (as invoked by virt-manager): /usr/bin/kvm -S -M pc-0.11 -m 1024 -smp 2 -name lexx -uuid [UUID] -monitor unix:/var/run/libvirt/qemu/VM1.monitor,server,nowait -boot c -drive file=,if=ide,media=cdrom,index=2 -drive file=/var/lib/libvirt/images/VM1.img,if=virtio,index=0,boot=on -net nic,macaddr=[MAC],vlan=0,model=virtio,name=virtio.0 -net tap,fd=16,vlan=0,name=tap.0 -serial none -parallel none -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus Problem description: After a while (and high network IO) I see this pop up in my guest dmesg: hrtimer: interrupt too slow, forcing clock min delta to 215997540 ns after that the guest becomes very slow to respond, sometimes taking seconds to echo my ssh input back, on a local lan. Only a reboot of the kvm guest fixes this, until the dreaded hrtimer message pops up After a lot of googling and trying a lot of things I found this discussion on the patchwork kernel mailinglist, which contains a possible solution: http://patchwork.kernel.org/patch/51561/ Please look into it, perhaps this solves a lot of kvm-users' problems I'd like to patch my kvm guests' kernel myself to test this hrtimer patch, do you have the correct procedure for me so i can create a custom kernel? To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/503138/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 503138] Re: Lucid & Natty, KVM, After kernel message hrtimer: interrupt too slow.... the SMP kvm guest becomes slow.
[Expired for linux (Ubuntu) because there has been no activity for 60 days.] ** Changed in: linux (Ubuntu) Status: Incomplete => Expired -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/503138 Title: Lucid & Natty, KVM, After kernel message hrtimer: interrupt too slow the SMP kvm guest becomes slow. Status in linux package in Ubuntu: Expired Status in qemu package in Ubuntu: Expired Bug description: KVM Host is running a clean 9.10 server install with just qemu-kvm and virt-manager kernel=2.6.31-17-generic uname -a: Linux VMMASTER 2.6.31-17-generic #54-Ubuntu SMP Thu Dec 10 16:20:31 UTC 2009 i686 GNU/Linux version signature: Ubuntu 2.6.31-17.54-generic KVM Guest is running a clean 9.10 server install with some userland services (apache/postfix/whatnot) kernel=2.6.31-16-generic-pae (from the linux-image-virtual package) uname -a: Linux VM1 2.6.31-16-generic-pae #53-Ubuntu SMP Tue Dec 8 05:20:21 UTC 2009 i686 GNU/Linux version signature: Ubuntu 2.6.31-16.53-generic-pae KVM guest startup command (as invoked by virt-manager): /usr/bin/kvm -S -M pc-0.11 -m 1024 -smp 2 -name lexx -uuid [UUID] -monitor unix:/var/run/libvirt/qemu/VM1.monitor,server,nowait -boot c -drive file=,if=ide,media=cdrom,index=2 -drive file=/var/lib/libvirt/images/VM1.img,if=virtio,index=0,boot=on -net nic,macaddr=[MAC],vlan=0,model=virtio,name=virtio.0 -net tap,fd=16,vlan=0,name=tap.0 -serial none -parallel none -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus Problem description: After a while (and high network IO) I see this pop up in my guest dmesg: hrtimer: interrupt too slow, forcing clock min delta to 215997540 ns after that the guest becomes very slow to respond, sometimes taking seconds to echo my ssh input back, on a local lan. Only a reboot of the kvm guest fixes this, until the dreaded hrtimer message pops up After a lot of googling and trying a lot of things I found this discussion on the patchwork kernel mailinglist, which contains a possible solution: http://patchwork.kernel.org/patch/51561/ Please look into it, perhaps this solves a lot of kvm-users' problems I'd like to patch my kvm guests' kernel myself to test this hrtimer patch, do you have the correct procedure for me so i can create a custom kernel? To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/503138/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 503138] Re: Lucid & Natty, KVM, After kernel message hrtimer: interrupt too slow.... the SMP kvm guest becomes slow.
** Tags added: bjf-tracking -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/503138 Title: Lucid & Natty, KVM, After kernel message hrtimer: interrupt too slow the SMP kvm guest becomes slow. Status in linux package in Ubuntu: Incomplete Status in qemu package in Ubuntu: Incomplete Bug description: KVM Host is running a clean 9.10 server install with just qemu-kvm and virt-manager kernel=2.6.31-17-generic uname -a: Linux VMMASTER 2.6.31-17-generic #54-Ubuntu SMP Thu Dec 10 16:20:31 UTC 2009 i686 GNU/Linux version signature: Ubuntu 2.6.31-17.54-generic KVM Guest is running a clean 9.10 server install with some userland services (apache/postfix/whatnot) kernel=2.6.31-16-generic-pae (from the linux-image-virtual package) uname -a: Linux VM1 2.6.31-16-generic-pae #53-Ubuntu SMP Tue Dec 8 05:20:21 UTC 2009 i686 GNU/Linux version signature: Ubuntu 2.6.31-16.53-generic-pae KVM guest startup command (as invoked by virt-manager): /usr/bin/kvm -S -M pc-0.11 -m 1024 -smp 2 -name lexx -uuid [UUID] -monitor unix:/var/run/libvirt/qemu/VM1.monitor,server,nowait -boot c -drive file=,if=ide,media=cdrom,index=2 -drive file=/var/lib/libvirt/images/VM1.img,if=virtio,index=0,boot=on -net nic,macaddr=[MAC],vlan=0,model=virtio,name=virtio.0 -net tap,fd=16,vlan=0,name=tap.0 -serial none -parallel none -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus Problem description: After a while (and high network IO) I see this pop up in my guest dmesg: hrtimer: interrupt too slow, forcing clock min delta to 215997540 ns after that the guest becomes very slow to respond, sometimes taking seconds to echo my ssh input back, on a local lan. Only a reboot of the kvm guest fixes this, until the dreaded hrtimer message pops up After a lot of googling and trying a lot of things I found this discussion on the patchwork kernel mailinglist, which contains a possible solution: http://patchwork.kernel.org/patch/51561/ Please look into it, perhaps this solves a lot of kvm-users' problems I'd like to patch my kvm guests' kernel myself to test this hrtimer patch, do you have the correct procedure for me so i can create a custom kernel? To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/503138/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 503138] Re: Lucid & Natty, KVM, After kernel message hrtimer: interrupt too slow.... the SMP kvm guest becomes slow.
Thanks Arie for providing your configuration fix to the issue for anyone else being affected - that will be really helpful. In that sense it is much more like bug 1346917 which was about KSM. Since the fix for that old bug they at least stopped being migrated around, but I can see how a lot of KSM based overcommit on remote nodes can make memory to be slow. But IMHO that is just one of the potential prices to pay with KSM, not that much of a bug but a configuration which happens to overload what the system can deliver. It needs a lot very slow memory to trigger the hrtimer issue, if you think there really is an issue to fix (other than the configuration) then I'd suggest try with the latest stack in Disco (19.04) which is very up to date in regard to kernel - if it still happens there you might report that upstream to the kernel ML - maybe there is an idea how to improve MM to cause less issues, but I'd expect they say that is what the tunable is for. But as said, if you report that to us as well - maybe once a patch exists to backport - I'd appreciate if that would be in a new bug. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/503138 Title: Lucid & Natty, KVM, After kernel message hrtimer: interrupt too slow the SMP kvm guest becomes slow. Status in linux package in Ubuntu: Incomplete Status in qemu package in Ubuntu: Incomplete Bug description: KVM Host is running a clean 9.10 server install with just qemu-kvm and virt-manager kernel=2.6.31-17-generic uname -a: Linux VMMASTER 2.6.31-17-generic #54-Ubuntu SMP Thu Dec 10 16:20:31 UTC 2009 i686 GNU/Linux version signature: Ubuntu 2.6.31-17.54-generic KVM Guest is running a clean 9.10 server install with some userland services (apache/postfix/whatnot) kernel=2.6.31-16-generic-pae (from the linux-image-virtual package) uname -a: Linux VM1 2.6.31-16-generic-pae #53-Ubuntu SMP Tue Dec 8 05:20:21 UTC 2009 i686 GNU/Linux version signature: Ubuntu 2.6.31-16.53-generic-pae KVM guest startup command (as invoked by virt-manager): /usr/bin/kvm -S -M pc-0.11 -m 1024 -smp 2 -name lexx -uuid [UUID] -monitor unix:/var/run/libvirt/qemu/VM1.monitor,server,nowait -boot c -drive file=,if=ide,media=cdrom,index=2 -drive file=/var/lib/libvirt/images/VM1.img,if=virtio,index=0,boot=on -net nic,macaddr=[MAC],vlan=0,model=virtio,name=virtio.0 -net tap,fd=16,vlan=0,name=tap.0 -serial none -parallel none -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus Problem description: After a while (and high network IO) I see this pop up in my guest dmesg: hrtimer: interrupt too slow, forcing clock min delta to 215997540 ns after that the guest becomes very slow to respond, sometimes taking seconds to echo my ssh input back, on a local lan. Only a reboot of the kvm guest fixes this, until the dreaded hrtimer message pops up After a lot of googling and trying a lot of things I found this discussion on the patchwork kernel mailinglist, which contains a possible solution: http://patchwork.kernel.org/patch/51561/ Please look into it, perhaps this solves a lot of kvm-users' problems I'd like to patch my kvm guests' kernel myself to test this hrtimer patch, do you have the correct procedure for me so i can create a custom kernel? To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/503138/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 503138] Re: Lucid & Natty, KVM, After kernel message hrtimer: interrupt too slow.... the SMP kvm guest becomes slow.
This happened because guest used lots of memory on NUMA machine. The KSM might merge similar memory pages of different VMs, sitting on different NUMA memory regions, causing the affected processes to crawl. Disabled KSM merge_across_nodes: echo 2 > /sys/kernel/mm/ksm/run && sleep 300 && cat /sys/kernel/mm/ksm/pages_shared If there are no pages shared : echo 0 > /sys/kernel/mm/ksm/merge_across_nodes && echo 1 > /sys/kernel/mm/ksm/run make sure to set merge_across_nodes in /etc/sysctl.d to stay across reboots. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/503138 Title: Lucid & Natty, KVM, After kernel message hrtimer: interrupt too slow the SMP kvm guest becomes slow. Status in linux package in Ubuntu: Incomplete Status in qemu package in Ubuntu: Incomplete Bug description: KVM Host is running a clean 9.10 server install with just qemu-kvm and virt-manager kernel=2.6.31-17-generic uname -a: Linux VMMASTER 2.6.31-17-generic #54-Ubuntu SMP Thu Dec 10 16:20:31 UTC 2009 i686 GNU/Linux version signature: Ubuntu 2.6.31-17.54-generic KVM Guest is running a clean 9.10 server install with some userland services (apache/postfix/whatnot) kernel=2.6.31-16-generic-pae (from the linux-image-virtual package) uname -a: Linux VM1 2.6.31-16-generic-pae #53-Ubuntu SMP Tue Dec 8 05:20:21 UTC 2009 i686 GNU/Linux version signature: Ubuntu 2.6.31-16.53-generic-pae KVM guest startup command (as invoked by virt-manager): /usr/bin/kvm -S -M pc-0.11 -m 1024 -smp 2 -name lexx -uuid [UUID] -monitor unix:/var/run/libvirt/qemu/VM1.monitor,server,nowait -boot c -drive file=,if=ide,media=cdrom,index=2 -drive file=/var/lib/libvirt/images/VM1.img,if=virtio,index=0,boot=on -net nic,macaddr=[MAC],vlan=0,model=virtio,name=virtio.0 -net tap,fd=16,vlan=0,name=tap.0 -serial none -parallel none -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus Problem description: After a while (and high network IO) I see this pop up in my guest dmesg: hrtimer: interrupt too slow, forcing clock min delta to 215997540 ns after that the guest becomes very slow to respond, sometimes taking seconds to echo my ssh input back, on a local lan. Only a reboot of the kvm guest fixes this, until the dreaded hrtimer message pops up After a lot of googling and trying a lot of things I found this discussion on the patchwork kernel mailinglist, which contains a possible solution: http://patchwork.kernel.org/patch/51561/ Please look into it, perhaps this solves a lot of kvm-users' problems I'd like to patch my kvm guests' kernel myself to test this hrtimer patch, do you have the correct procedure for me so i can create a custom kernel? To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/503138/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 503138] Re: Lucid & Natty, KVM, After kernel message hrtimer: interrupt too slow.... the SMP kvm guest becomes slow.
Hi Arie, I saw you also posting on some other old bugs with the same symptom (e.g. 1346917). And that exactly is the problem with this issue - the message only represents the symptom but not the root cause. Fixing the symptom makes no sense, because if your IRQ delivery is stalled then it is stalled - there is only very little the system can do other than trottling it down. If you follow the discussion around [1] you'll see that this was the thought back then and it didn't change since then AFAIK. There were cases around KSM (bug 1346917), but also others around broken RAM, then others had just overloaded their CPUs, others had scheduler bug triggering the same while in other cases there was a thundering herd issue. The only thing all of these cases share is that eventually something (tm) happened which made IRQ/hrtimers stutter to then throttle them down. Many of the underlying issues causing that have been fixed over the years - this got more rare nowadays. But for each case still left one would not need a "this happened again here" message. But instead a way to reproduce, to then debug the root cause of this exact case and check for a fix then. Therefore three recommendations for people affected: 1. Try as good as you can to get it to reproduce reliably and then outline these steps, this will help people to hopefully get a grasp of the root cause. Do -not- just report the "hrtimer: interrupt took" being the symptom 2. Always try the same setup you have, but with the very latest virtualization stack provided - quite often things are fixed there and if that is confirmed it becomes "only" a binary search what would need to be backported. 3. open a new bug for these, because until a root cause of a given case is found and identified to be the same we don't know if it is the same issue. [1]: http://www.mail-archive.com/kvm@vger.kernel.org/msg23491.html ** Also affects: linux (Ubuntu) Importance: Undecided Status: New ** Package changed: kvm (Ubuntu) => qemu (Ubuntu) ** Changed in: qemu (Ubuntu) Status: Confirmed => Incomplete ** Changed in: linux (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/503138 Title: Lucid & Natty, KVM, After kernel message hrtimer: interrupt too slow the SMP kvm guest becomes slow. Status in linux package in Ubuntu: Incomplete Status in qemu package in Ubuntu: Incomplete Bug description: KVM Host is running a clean 9.10 server install with just qemu-kvm and virt-manager kernel=2.6.31-17-generic uname -a: Linux VMMASTER 2.6.31-17-generic #54-Ubuntu SMP Thu Dec 10 16:20:31 UTC 2009 i686 GNU/Linux version signature: Ubuntu 2.6.31-17.54-generic KVM Guest is running a clean 9.10 server install with some userland services (apache/postfix/whatnot) kernel=2.6.31-16-generic-pae (from the linux-image-virtual package) uname -a: Linux VM1 2.6.31-16-generic-pae #53-Ubuntu SMP Tue Dec 8 05:20:21 UTC 2009 i686 GNU/Linux version signature: Ubuntu 2.6.31-16.53-generic-pae KVM guest startup command (as invoked by virt-manager): /usr/bin/kvm -S -M pc-0.11 -m 1024 -smp 2 -name lexx -uuid [UUID] -monitor unix:/var/run/libvirt/qemu/VM1.monitor,server,nowait -boot c -drive file=,if=ide,media=cdrom,index=2 -drive file=/var/lib/libvirt/images/VM1.img,if=virtio,index=0,boot=on -net nic,macaddr=[MAC],vlan=0,model=virtio,name=virtio.0 -net tap,fd=16,vlan=0,name=tap.0 -serial none -parallel none -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus Problem description: After a while (and high network IO) I see this pop up in my guest dmesg: hrtimer: interrupt too slow, forcing clock min delta to 215997540 ns after that the guest becomes very slow to respond, sometimes taking seconds to echo my ssh input back, on a local lan. Only a reboot of the kvm guest fixes this, until the dreaded hrtimer message pops up After a lot of googling and trying a lot of things I found this discussion on the patchwork kernel mailinglist, which contains a possible solution: http://patchwork.kernel.org/patch/51561/ Please look into it, perhaps this solves a lot of kvm-users' problems I'd like to patch my kvm guests' kernel myself to test this hrtimer patch, do you have the correct procedure for me so i can create a custom kernel? To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/503138/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
