[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
This report (in German) https://forge.univention.org/bugzilla/show_bug.cgi?id=23258 suggests that when the guest VM is reset by a reboot the clock jumps backward in time. Two patches were suggested, both being necessary together: KVM: x86: Fix kvmclock bug backported to Natty Aug 11, 2011 but not to linux-lts-backport-natty-2.6.38 yet!!! http://patchwork.ozlabs.org/patch/109436/ http://lkml.org/lkml/2011/5/5/436 http://bugs.launchpad.net/bugs/714335 KVM: x86: Fix a possible backwards warp of kvmclock http://kerneltrap.org/mailarchive/linux-kernel/2010/8/20/4609037 http://choon.net/forum/read.php?21,76764 ** Bug watch added: forge.univention.org/bugzilla/ #23258 https://forge.univention.org/bugzilla/show_bug.cgi?id=23258 -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
This patch is missing from linux-lts-backport-natty-2.6.38 http://patchwork.ozlabs.org/patch/109436/ diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 980317a..0556e05 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -2100,8 +2100,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) if (check_tsc_unstable()) { kvm_x86_ops-adjust_tsc_offset(vcpu, -tsc_delta); vcpu-arch.tsc_catchup = 1; - kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); } + kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); if (vcpu-cpu != cpu) kvm_migrate_timers(vcpu); vcpu-cpu = cpu; -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
Please help apply patch http://patchwork.ozlabs.org/patch/109436/ to linux-lts-backport-natty-2.6.38 thanks -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
** Changed in: libvirt (Ubuntu) Status: Incomplete = Confirmed ** Description changed: + To reproduce: + + apt-get source linux-image-2.6.38-12-generic + + examine linux-lts-backport-natty-2.6.38/arch/x86/kvm/x86.c + + See missing patch: + + diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c + index 980317a..0556e05 100644 + --- a/arch/x86/kvm/x86.c + +++ b/arch/x86/kvm/x86.c + @@ -2100,8 +2100,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) +if (check_tsc_unstable()) { + kvm_x86_ops-adjust_tsc_offset(vcpu, -tsc_delta); + vcpu-arch.tsc_catchup = 1; + - kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); +} + + kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); +if (vcpu-cpu != cpu) + kvm_migrate_timers(vcpu); +vcpu-cpu = cpu; + + = + While attempting to use virsh commands this error occurs: Oct 27 08:52:22 boron libvirtd: 08:52:23.000: 25851: error : qemuDomainObjBeginJobWithDriver:453 : Timed out during operation: cannot acquire state change lock While the error state is present certain commands like virsh shutdown and virsh destroy elicit that error message and fail to work completely I found that if I kill the kvm guest processes $ kill -9 $(ps ax | grep '/usr/bin/kvm.*guestname' | awk '{print $1}') Then virsh start guestname would work. I'm not certain that the number of VM guests is the issue, however, this happens more frequently to me on systems with larger numbers of guests. We have one with 14 guests and another with 53. - There seems to be a corresponding Red Hat bug at: + There seems to be a corresponding Red Hat bug at: https://bugzilla.redhat.com/show_bug.cgi?id=676205 ProblemType: Bug DistroRelease: Ubuntu 11.04 Package: libvirt-bin 0.8.8-1ubuntu6.5 ProcVersionSignature: Ubuntu 2.6.38-12.51-generic 2.6.38.8 Uname: Linux 2.6.38-12-generic x86_64 Architecture: amd64 Date: Thu Oct 27 09:41:23 2011 InstallationMedia: Ubuntu 10.10 Maverick Meerkat - Release amd64 (20101007) ProcEnviron: - LANGUAGE=en_US:en - PATH=(custom, user) - LANG=en_US.UTF-8 - SHELL=/bin/bash + LANGUAGE=en_US:en + PATH=(custom, user) + LANG=en_US.UTF-8 + SHELL=/bin/bash SourcePackage: libvirt UpgradeStatus: Upgraded to natty on 2011-05-26 (153 days ago) ** Description changed: To reproduce: + On Lucid run: apt-get source linux-image-2.6.38-12-generic examine linux-lts-backport-natty-2.6.38/arch/x86/kvm/x86.c See missing patch: diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 980317a..0556e05 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -2100,8 +2100,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) -if (check_tsc_unstable()) { - kvm_x86_ops-adjust_tsc_offset(vcpu, -tsc_delta); - vcpu-arch.tsc_catchup = 1; + if (check_tsc_unstable()) { + kvm_x86_ops-adjust_tsc_offset(vcpu, -tsc_delta); + vcpu-arch.tsc_catchup = 1; - kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); -} + } + kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); -if (vcpu-cpu != cpu) - kvm_migrate_timers(vcpu); -vcpu-cpu = cpu; + if (vcpu-cpu != cpu) + kvm_migrate_timers(vcpu); + vcpu-cpu = cpu; = While attempting to use virsh commands this error occurs: Oct 27 08:52:22 boron libvirtd: 08:52:23.000: 25851: error : qemuDomainObjBeginJobWithDriver:453 : Timed out during operation: cannot acquire state change lock While the error state is present certain commands like virsh shutdown and virsh destroy elicit that error message and fail to work completely I found that if I kill the kvm guest processes $ kill -9 $(ps ax | grep '/usr/bin/kvm.*guestname' | awk '{print $1}') Then virsh start guestname would work. I'm not certain that the number of VM guests is the issue, however, this happens more frequently to me on systems with larger numbers of guests. We have one with 14 guests and another with 53. There seems to be a corresponding Red Hat bug at: https://bugzilla.redhat.com/show_bug.cgi?id=676205 ProblemType: Bug DistroRelease: Ubuntu 11.04 Package: libvirt-bin 0.8.8-1ubuntu6.5 ProcVersionSignature: Ubuntu 2.6.38-12.51-generic 2.6.38.8 Uname: Linux 2.6.38-12-generic x86_64 Architecture: amd64 Date: Thu Oct 27 09:41:23 2011 InstallationMedia: Ubuntu 10.10 Maverick Meerkat - Release amd64 (20101007) ProcEnviron: LANGUAGE=en_US:en PATH=(custom, user) LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: libvirt UpgradeStatus: Upgraded to natty on 2011-05-26 (153 days ago) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to:
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
** Tags removed: natty running-unity ** Tags added: lucid testcase -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
This is a bug in the kernel not libvirt. ** Changed in: libvirt (Ubuntu) Status: Confirmed = Invalid -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
*** This bug is a duplicate of bug 795717 *** https://bugs.launchpad.net/bugs/795717 ** This bug has been marked a duplicate of bug 795717 32bit rhel and centos 5.(5|6) hangs on boot on natty -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
Probably related thread: kvm linux guest hanging for minutes at a time http://www.spinics.net/lists/kvm/msg59479.html Notable in that thread: a post about using -F with trace-cmd too: http://www.linux-kvm.org/page/Tracing http://www.spinics.net/lists/kvm/msg59716.html a post which states: When booting with clocksouce=hpet, the system does not hang. http://www.spinics.net/lists/kvm/msg59837.html a post which states: Using the wrong clock easily causes hangs. The system schedules a wakeup in 3 ms, wrong clock causes it to wakeup in 3 years, you get a hang for (3 years - 3 ms). http://www.spinics.net/lists/kvm/msg59884.html -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
I have one that rebooted this morning and is spinning attaching gdb shows very little due to lack of symbol tables: (gdb) where #0 0x7f671aab3ff3 in select () from /lib/libc.so.6 strace shows a loop: Process 971 attached - interrupt to quit select(16, [7 10 13 14 15], [], [], {0, 995193}) = 1 (in [14], left {0, 970413}) read(14, \16\0\0\0\0\0\0\0\376\377\377\377\0\0\0\0\0\0\0\0\0\0\0\0\5\0\0\0\0\0\0\0..., 128) = 128 rt_sigaction(SIGALRM, NULL, {0x4a9900, ~[KILL STOP RTMIN RT_1], SA_RESTORER, 0x7f671d28c8f0}, 8) = 0 write(8, \1\0\0\0\0\0\0\0, 8) = 8 read(14, 0x7fff53785d20, 128) = -1 EAGAIN (Resource temporarily unavailable) timer_gettime(0x5, {it_interval={0, 0}, it_value={0, 0}}) = 0 timer_settime(0x5, 0, {it_interval={0, 0}, it_value={0, 25}}, NULL) = 0 timer_gettime(0x5, {it_interval={0, 0}, it_value={0, 185840}}) = 0 select(16, [7 10 13 14 15], [], [], {1, 0}) = 1 (in [7], left {0, 97}) read(7, \1\0\0\0\0\0\0\0, 512)= 8 select(16, [7 10 13 14 15], [], [], {1, 0}) = 1 (in [14], left {0, 97}) read(14, \16\0\0\0\0\0\0\0\376\377\377\377\0\0\0\0\0\0\0\0\0\0\0\0\5\0\0\0\0\0\0\0..., 128) = 128 rt_sigaction(SIGALRM, NULL, {0x4a9900, ~[KILL STOP RTMIN RT_1], SA_RESTORER, 0x7f671d28c8f0}, 8) = 0 write(8, \1\0\0\0\0\0\0\0, 8) = 8 read(14, 0x7fff53785d20, 128) = -1 EAGAIN (Resource temporarily unavailable) timer_gettime(0x5, {it_interval={0, 0}, it_value={0, 0}}) = 0 timer_settime(0x5, 0, {it_interval={0, 0}, it_value={0, 28902469}}, NULL) = 0 (repeats) $ sudo lsof -p 971 COMMAND PID USER FD TYPE DEVICESIZE/OFF NODE NAME kvm 971 libvirt-qemu cwdDIR 259,04096 128 / kvm 971 libvirt-qemu rtdDIR 259,04096 128 / kvm 971 libvirt-qemu txtREG 259,0 3217592 16817685 /usr/bin/qemu-system-x86_64 kvm 971 libvirt-qemu memREG 259,0 18520 33597583 /usr/lib/sasl2/libplain.so.2.0.23 kvm 971 libvirt-qemu memREG 259,0 34904 33597580 /usr/lib/sasl2/libntlm.so.2.0.23 kvm 971 libvirt-qemu memREG 259,0 43296 33595849 /lib/libcrypt-2.11.1.so kvm 971 libvirt-qemu memREG 259,0 18520 33597534 /usr/lib/sasl2/liblogin.so.2.0.23 kvm 971 libvirt-qemu memREG 259,0 1622304 33596048 /lib/libcrypto.so.0.9.8 kvm 971 libvirt-qemu memREG 259,0 51752 33597531 /usr/lib/sasl2/libdigestmd5.so.2.0.23 kvm 971 libvirt-qemu memREG 259,0 22624 33597528 /usr/lib/sasl2/libcrammd5.so.2.0.23 kvm 971 libvirt-qemu memREG 259,0 18528 33597524 /usr/lib/sasl2/libanonymous.so.2.0.23 kvm 971 libvirt-qemu memREG 259,0 1494424 187 /usr/lib/libdb-4.8.so kvm 971 libvirt-qemu memREG 259,0 22464 33597518 /usr/lib/sasl2/libsasldb.so.2.0.23 kvm 971 libvirt-qemu memREG 259,0 26840 8762051 /usr/lib/libogg.so.0.6.0 kvm 971 libvirt-qemu memREG 259,0 18258456392 /usr/lib/libvorbis.so.0.4.3 kvm 971 libvirt-qemu memREG 259,0 1943600 8762070 /usr/lib/libvorbisenc.so.2.0.6 kvm 971 libvirt-qemu memREG 259,0 306744 8762063 /usr/lib/libFLAC.so.8.2.0 kvm 971 libvirt-qemu memREG 259,0 97256 33595855 /lib/libnsl-2.11.1.so kvm 971 libvirt-qemu memREG 259,0 22560 530121 /usr/lib/libXdmcp.so.6.0.0 kvm 971 libvirt-qemu memREG 259,0 14488 530127 /usr/lib/libXau.so.6.0.0 kvm 971 libvirt-qemu memREG 259,0 256768 33607491 /lib/libdbus-1.so.3.4.0 kvm 971 libvirt-qemu memREG 259,0 397304 9012660 /usr/lib/libsndfile.so.1.0.21 kvm 971 libvirt-qemu memREG 259,0 40720 34176892 /lib/libwrap.so.0.7.6 kvm 971 libvirt-qemu memREG 259,0 63864 467747 /usr/lib/libXi.so.6.1.0 kvm 971 libvirt-qemu memREG 259,0 73600 533951 /usr/lib/libXext.so.6.4.0 kvm 971 libvirt-qemu memREG 259,0 14344 33597467 /lib/libgpg-error.so.0.4.0 kvm 971 libvirt-qemu memREG 259,0 10224 33597288 /lib/libkeyutils-1.2.so kvm 971 libvirt-qemu memREG 259,0 31168 1063 /usr/lib/libkrb5support.so.0.1 kvm 971 libvirt-qemu memREG 259,0 14584 33595836 /lib/libcom_err.so.2.1 kvm 971 libvirt-qemu memREG 259,0 154048 1076 /usr/lib/libk5crypto.so.3.1 kvm 971 libvirt-qemu memREG 259,0 803192 1073 /usr/lib/libkrb5.so.3.3 kvm 971 libvirt-qemu memREG 259,0 113072 530136 /usr/lib/libxcb.so.1.1.0 kvm 971
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
I've been able to narrow this down a little more. The number of VM guests is not a factor, this has happened on a VM host with only one guest. The VM guest kernel does not seem to matter either, even VM guests which are Linux distros other than Ubuntu are affected. It seems that the /usr/bin/kvm process goes into an infinite loop during VM guest shutdown. I'm still investigating this further. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
-- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
Attached the output of for p in `pidof libvirtd`; do echo -n $p: cat /proc/$p/cmdline cat /proc/$p/cgroup cat /proc/$p/status done for p in `pidof kvm`; do echo -n $p: cat /proc/$p/cmdline cat /proc/$p/cgroup cat /proc/$p/status done ** Attachment added: LP-882579-pidof.txt https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+attachment/2575222/+files/LP-882579-pidof.txt -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
This really needs a reproducible test case. Are you able to offer any further guidance on how you managed to encounter this? Thanks. ** Changed in: libvirt (Ubuntu) Importance: Undecided = Medium ** Changed in: libvirt (Ubuntu) Status: Confirmed = Incomplete -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
Thanks for filing this new bug. (I think it is in fact a dup of bug 734777, but it's not certain) I'm afraid I'll simply have to try to reproduce this on a maverick system and debug with gdb, as per the linked redhat bug, upstream is not sure what fixed the problem between 0.8.8 and 0.9.0. If it is feasible for you to upgrade to oneiric, the newer version of libvirt there should bypass this problem, however that certainly is not a satisfactory solution, especially as this appears to be hitting multiple users. ** Changed in: libvirt (Ubuntu) Status: New = Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
While the VM guests were hung they were using 100% of the CPU. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
If this issue was seen on Maverick it is still an issue with Natty. I could try running tests on an Oneiric system too. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
The system with 53 VM guests: ProLiant DL585 G5, Four CPU sockets each with Quad-Core AMD Opteron(tm) Processor 8382 The system with 14 VM guests: Sun Fire X4600 M2, Eight CPU sockets each with Dual-Core AMD Opteron(tm) Processor 8220 -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
Today two VM hosts running kernel 2.6.38-11, one with with 14 guests and another with 53 had scheduled reboots of the many of their guests. On the VM host with 14 guests, 11 rebooted. On the one with 53 guests, 48 rebooted. VM guests that automatically rebooted this morning at 6:30 AM via at(8) hung instead of rebooted. All of the VM guests were rebooting from kernel 2.6.38-11 into 2.6.38-12 Not all of the VM guests on the hosts were scheduled to be rebooted. virsh shutdown and virsh destroy appeared to always hang. After using kill -9 on the PID of the VM guest virsh start for that guest worked every time. Initially on the VM hosts virsh list would output some VM names then hang. After enough VM guests were reset via kill -9 (but before all of them were) virsh list became able to run to completion. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
Quoting nutznboltz (882...@bugs.launchpad.net): While the VM guests were hung they were using 100% of the CPU. Is 'virsh destroy' and 'virsh shutdown' what always triggers this? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
Using kill -9 directly on the /usr/bin/kvm process of the hung VM guest that was using 100% CPU did restore the ability to use virsh start on the guest. That is consistent with the comment #26 by Daniel Berrange in the Red Hat ticket about The QEMU process has hung. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
If you look at the last comment in the Red Hat bug report: https://bugzilla.redhat.com/show_bug.cgi?id=676205#c26 {{ 1. The QEMU process has hung. QEMU won't respond to monitor commands. The API call making the first monitor command will wait forever, any subsequent API calls issuing monitor commands will timeout after ~30 seconds with this libvirt error message. This is expected behaviour when QEMU has hung. }} I think this might be the issue I am experiencing. ** Bug watch added: Red Hat Bugzilla #676205 https://bugzilla.redhat.com/show_bug.cgi?id=676205 -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 882579] Re: Some Commands return: Timed out during operation: cannot acquire state change lock
I don't think virsh shutdown and virsh destroy are triggers. I think when the guest rebooted something went wrong and subsequent attempts to use virsh shutdown and virsh destroy fail. This morning the en-mass VM guest reboot was triggered by cron jobs inside the guests. I tried unsuccessfully to reproduce this on a VM host that only has a single VM guest. I'm still investigating if having several VM guests reboot at the same time is the trigger or not. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/882579 Title: Some Commands return: Timed out during operation: cannot acquire state change lock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/882579/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs