** Description changed: - On a brand new installation of Ubuntu Server 16.04.2 LTS the libvirt- - guests.sh script fails to shutdown my VMs if run with PARALLEL_SHUTDOWN - > 0, instead running forever thereby preventing me from shutting down my - system. I debugged the script with 'set -x' and found out that the - problem lies in the `check_guests_shutdown()` function (line 337). If a - guest is shutdown, it runs eval_gettext() to print that the state of the - guest cannot be determined. This is used as a return value on line 428, - causing the script to interpret the error message as a list of VM IDs - and trying to shutdown these. This of course fails again and the cycle - continues. For now I have commented out the eval_gettext line and the - script works fine again. + [Environment] - My libvirt version is 1.3.1-1ubuntu10.8. + No LSB modules are available. + Distributor ID: Ubuntu + Description: Ubuntu 16.04.3 LTS + Release: 16.04 + Codename: xenial + + + [Impact] + + There is a bug/race condition on libvirt-guests.service, that prevents the shutdown of guests to happen in parallel and by consequence locking + the machine shutdown during the entire SHUTDOWN_TIMEOUT. + + The critical chain for this service is: + + libvirt-guests.service +20ms + └─libvirt-bin.service @2.784s +140ms + └─remote-fs.target @2.777s + └─remote-fs-pre.target @2.775s + └─open-iscsi.service @2.554s +116ms + └─iscsid.service @2.525s +18ms + └─network-online.target @2.502s + └─network.target @1.955s + └─networking.service @1.625s +299ms + └─network-pre.target @1.601s + └─cloud-init-local.service @405ms +1.072s + └─systemd-remount-fs.service @232ms +64ms + └─systemd-journald.socket @178ms + └─-.slice @117ms + + + As an example, I have the following kvm host with 42 virtual + machines. + + ubuntu@xenial-base:~$ virsh list --all + Id Name State + ---------------------------------------------------- + 12 locked-trusty-2 running + 13 locked-trusty-3 running + [...] + 41 locked-trusty-42 running + + After rebooting the machine: + + [ 250.999516] libvirt-guests.sh[4215]: Running guests on default URI: locked-trusty-2, locked-trusty-4, locked-trusty-12, locked-trusty-3, locked-trusty-5, locked-trusty-11, locked-trusty-10, locked-trusty-8, locked-trusty-9, locked-trusty-7, locked-trusty-6, locked-trusty-13, locked-trusty-14, locked-trusty-15, locked-trusty-16, locked-trusty-17, locked-trusty-18, locked-trusty-19, locked-trusty-20, locked-trusty-21, locked-trusty-22, locked-trusty-23, locked-trusty-24, locked-trusty-25, locked-trusty-26, locked-trusty-27, locked-trusty-28, locked-trusty-29, locked-trusty-30, locked-trusty-31, locked-trusty-32, locked-trusty-33, locked-trusty-34, locked-trusty-35, locked-trusty-36, locked-trusty-37, locked-trusty-38, locked-trusty-39, locked-trusty-40, locked-trusty-41, locked-trusty-42 + [ 251.011367] libvirt-guests.sh[4215]: Shutting down guests on default URI... + [ 251.027072] libvirt-guests.sh[4215]: Starting shutdown on guest: locked-trusty-2 + [...] + [ 391.949941] libvirt-guests.sh[4215]: Waiting for 28 guests to shut down, 10 seconds left + [ 398.074405] libvirt-guests.sh[4215]: Waiting for 28 guests to shut down, 5 seconds left + [ 403.020479] libvirt-guests.sh[4215]: Timeout expired while shutting down domains + [ OK ] Stopped Suspend Active Libvirt Guests. + [ OK ] Stopped target System Time Synchronized. + + [Test Case] + + * Make sure the following variables are set in /etc/default/libvirt- + guests (which are all default options): + + ON_SHUTDOWN=shutdown + PARALLEL_SHUTDOWN=10 + SHUTDOWN_TIMEOUT=120 + + * Create over 20 virtual machines (in my case, using uvt-kvm). + + $ for f in $(seq 0 40); do uvt-kvm create --memory 2000 --cpu 1 locked- + trusty-$f release=xenial arch=amd64 ; done + + * Reboot the machine and monitor the systemd service stop sequence + or console output. + + (With systemd: systemctl start debug-shell and jumpt to ctrl+alt+f9) + + * Error message "Timeout expired while shutting down domains" should + be displayed. + + [Regression Potential] + + * None identified. + + [Other Info] + + * There is a proposed patch in upstream already that has been already + linked to this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1450141
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1688508 Title: libvirt-guests.sh fails to shutdown guests in parallel To manage notifications about this bug go to: https://bugs.launchpad.net/libvirt/+bug/1688508/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
