[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-31 Thread nutznboltz
This report (in German) https://forge.univention.org/bugzilla/show_bug.cgi?id=23258 suggests that when the guest VM is reset by a reboot the clock jumps backward in time. Two patches were suggested, both being necessary together: KVM: x86: Fix kvmclock bug backported to Natty Aug 11, 2011 but

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-31 Thread nutznboltz
This patch is missing from linux-lts-backport-natty-2.6.38 http://patchwork.ozlabs.org/patch/109436/ diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 980317a..0556e05 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -2100,8 +2100,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-31 Thread nutznboltz
Please help apply patch http://patchwork.ozlabs.org/patch/109436/ to linux-lts-backport-natty-2.6.38 thanks -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-31 Thread nutznboltz
** Changed in: libvirt (Ubuntu) Status: Incomplete = Confirmed ** Description changed: + To reproduce: + + apt-get source linux-image-2.6.38-12-generic + + examine linux-lts-backport-natty-2.6.38/arch/x86/kvm/x86.c + + See missing patch: + + diff --git a/arch/x86/kvm/x86.c

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-31 Thread nutznboltz
** Tags removed: natty running-unity ** Tags added: lucid testcase -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation:

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-31 Thread Serge Hallyn
** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-31 Thread nutznboltz
This is a bug in the kernel not libvirt. ** Changed in: libvirt (Ubuntu) Status: Confirmed = Invalid -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-31 Thread nutznboltz
*** This bug is a duplicate of bug 795717 *** https://bugs.launchpad.net/bugs/795717 ** This bug has been marked a duplicate of bug 795717 32bit rhel and centos 5.(5|6) hangs on boot on natty -- You received this bug notification because you are a member of Ubuntu Server Team, which is

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-30 Thread nutznboltz
Probably related thread: kvm linux guest hanging for minutes at a time http://www.spinics.net/lists/kvm/msg59479.html Notable in that thread: a post about using -F with trace-cmd too: http://www.linux-kvm.org/page/Tracing http://www.spinics.net/lists/kvm/msg59716.html a post which states: When

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-30 Thread nutznboltz
I have one that rebooted this morning and is spinning attaching gdb shows very little due to lack of symbol tables: (gdb) where #0 0x7f671aab3ff3 in select () from /lib/libc.so.6 strace shows a loop: Process 971 attached - interrupt to quit select(16, [7 10 13 14 15], [], [], {0, 995193})

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-28 Thread nutznboltz
I've been able to narrow this down a little more. The number of VM guests is not a factor, this has happened on a VM host with only one guest. The VM guest kernel does not seem to matter either, even VM guests which are Linux distros other than Ubuntu are affected. It seems that the

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-27 Thread nutznboltz
-- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-27 Thread nutznboltz
Attached the output of for p in `pidof libvirtd`; do echo -n $p: cat /proc/$p/cmdline cat /proc/$p/cgroup cat /proc/$p/status done for p in `pidof kvm`; do echo -n $p: cat /proc/$p/cmdline cat /proc/$p/cgroup cat /proc/$p/status done **

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-27 Thread Dave Walker
This really needs a reproducible test case. Are you able to offer any further guidance on how you managed to encounter this? Thanks. ** Changed in: libvirt (Ubuntu) Importance: Undecided = Medium ** Changed in: libvirt (Ubuntu) Status: Confirmed = Incomplete -- You received this

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-27 Thread Serge Hallyn
Thanks for filing this new bug. (I think it is in fact a dup of bug 734777, but it's not certain) I'm afraid I'll simply have to try to reproduce this on a maverick system and debug with gdb, as per the linked redhat bug, upstream is not sure what fixed the problem between 0.8.8 and 0.9.0. If

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-27 Thread nutznboltz
While the VM guests were hung they were using 100% of the CPU. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-27 Thread nutznboltz
If this issue was seen on Maverick it is still an issue with Natty. I could try running tests on an Oneiric system too. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title:

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-27 Thread nutznboltz
The system with 53 VM guests: ProLiant DL585 G5, Four CPU sockets each with Quad-Core AMD Opteron(tm) Processor 8382 The system with 14 VM guests: Sun Fire X4600 M2, Eight CPU sockets each with Dual-Core AMD Opteron(tm) Processor 8220 -- You received this bug notification because you are a

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-27 Thread nutznboltz
Today two VM hosts running kernel 2.6.38-11, one with with 14 guests and another with 53 had scheduled reboots of the many of their guests. On the VM host with 14 guests, 11 rebooted. On the one with 53 guests, 48 rebooted. VM guests that automatically rebooted this morning at 6:30 AM via at(8)

Re: [Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-27 Thread Serge Hallyn
Quoting nutznboltz (882...@bugs.launchpad.net): While the VM guests were hung they were using 100% of the CPU. Is 'virsh destroy' and 'virsh shutdown' what always triggers this? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-27 Thread nutznboltz
Using kill -9 directly on the /usr/bin/kvm process of the hung VM guest that was using 100% CPU did restore the ability to use virsh start on the guest. That is consistent with the comment #26 by Daniel Berrange in the Red Hat ticket about The QEMU process has hung. -- You received this bug

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-27 Thread nutznboltz
If you look at the last comment in the Red Hat bug report: https://bugzilla.redhat.com/show_bug.cgi?id=676205#c26 {{ 1. The QEMU process has hung. QEMU won't respond to monitor commands. The API call making the first monitor command will wait forever, any subsequent API calls issuing

[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock

2011-10-27 Thread nutznboltz
I don't think virsh shutdown and virsh destroy are triggers. I think when the guest rebooted something went wrong and subsequent attempts to use virsh shutdown and virsh destroy fail. This morning the en-mass VM guest reboot was triggered by cron jobs inside the guests. I tried unsuccessfully