Public bug reported:
Problem Description:
=========================
Tried to take dump using virsh dumpxml command and it fails with Timed out
'held by remoteDispatchDomainCoreDump' error.
Steps to re-create:
============================
1. boslcp3g4 is installed with 4.15.0-15-generic kernel.
2. LTP & memory map tests were running inside guest.
3. After some time guest in hung state.
4. Tried to take dump using virsh dumpxml.
root@boslcp3:~# virsh dump boslcp3g4 boslcp3g4_mmap_ltp --memory-only
error: Failed to core dump domain boslcp3g4 to boslcp3g4_mmap_ltp
error: Disconnected from qemu:///system due to keepalive timeout
error: Timed out during operation: cannot acquire state change lock (held by
remoteDispatchDomainCoreDump)
root@boslcp3:~# virsh list --all
Id Name State
1 boslcp3g3 running
2 boslcp3g4 paused
4 boslcp3g1 running
5. It fails with Timed out during opearation & with held by
remoteDispatchDomainCoreDump error
6. /var/log/syslog dumps
Apr 18 03:29:14 boslcp3 libvirtd[5538]: 2018-04-18 08:29:13.956+0000: 5576:
warning : qemuDomainObjBeginJobInternal:4863 : Cannot start job (query, none)
for domain boslcp3g4; current job is (async nested, dump) owned by (5574
remoteDispatchDomainCoreDump, 5574 remoteDispatchDomainCoreDump) for (701s,
701s)
Apr 18 03:29:14 boslcp3 libvirtd[5538]: 2018-04-18 08:29:13.958+0000: 5576:
error : qemuDomainObjBeginJobInternal:4875 : Timed out during operation: cannot
acquire state change lock (held by remoteDispatchDomainCoreDump)
Apr 18 03:29:44 boslcp3 libvirtd[5538]: 2018-04-18 08:29:44.492+0000: 5573:
warning : qemuDomainObjBeginJobInternal:4863 : Cannot start job (query, none)
for domain boslcp3g4; current job is (async nested, dump) owned by (5574
remoteDispatchDomainCoreDump, 5574 remoteDispatchDomainCoreDump) for (731s,
732s)
7. Attached syslog & sosreport
== Comment: #3 - Application Cdeadmin <cdead...@us.ibm.com> - 2018-04-18
08:11:01 ==
When i tried for second time same command it was successful but syslog dumps
below warnings continuously
warning : :4863 : Cannot start job (query, none) for domain boslcp3g4;
current job is (async nested, dump) owned by (5574
remoteDispatchDomainCoreDump, 5574
root@boslcp3:~# virsh dump boslcp3g4 boslcp3g4_mmapltp --memory-only
Domain boslcp3g4 dumped to boslcp3g4_mmapltp
vmcore located at:
vmcore at kte111:/LOGS/boslcp3g4/boslcp3g4_mmapltp
Access kte111 using debug@9.3.111.155 (don2rry)
== Comment: #8 - Application Cdeadmin <cdead...@us.ibm.com> - 2018-04-19
05:26:32 ==
Tried to start the guest boslcp3g1 guest which has qlogic disk as boot & IO disk
root@boslcp3:~# virsh list --all
Id Name State
1 boslcp3g4 running
3 boslcp3g3 running
- boslcp3g1 shut off
root@boslcp3:~# echo 10240 > /proc/sys/vm/nr_hugepages
root@boslcp3:~# virsh start --console boslcp3g1
--> Than saw guest went to paused state.
root@boslcp3:/home# virsh list --all
Id Name State
1 boslcp3g4 running
3 boslcp3g3 running
5 boslcp3g1 paused
Then tried to destroy the guest and its fails with Timed out during
operation: cannot acquire state change lock. Even resume command also
failing as below
Corresponding syslog from /var/log:
Apr 19 05:17:09 boslcp3 libvirtd[5576]: 2018-04-19 10:17:09.056+0000: 5635:
error : virProcessKillPainfully:401 : Failed to terminate process 142520 with
SIGKILL: Device or resource busy
== Comment: #26 - Shivaprasad G. Bhat <shiva...@in.ibm.com> - 2018-05-17
08:57:25 ==
Got to test the patches independently. The below commits from upstream fix the
false alarms and allows the dump to go through clean.
a5bc7130f3
e712579200
150930e309
9a1755b7fe
501e3c3c96
88c2360753
3455a7359c
fd1a9e5c56
2a4d847e77
9d73df98c2
93412bb827
a8ef7b69dc
5870f95a7a
3f99bb06d1
** Affects: libvirt (Ubuntu)
Importance: Undecided
Assignee: Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage)
Status: New
** Tags: architecture-ppc64le bugnameltc-166912 severity-medium
targetmilestone-inin1804
** Tags added: architecture-ppc64le bugnameltc-166912 severity-medium
targetmilestone-inin1804
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1771827
Title:
Timed out message while taking dump using virsh dumpxml command &
fails with 'held by remoteDispatchDomainCoreDump' error
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1771827/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
