[Bug 1803741] Re: libvirt-guests.sh cannot stop Paused domains
[Expired for libvirt (Ubuntu) because there has been no activity for 60 days.] ** Changed in: libvirt (Ubuntu) Status: Incomplete => Expired -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1803741 Title: libvirt-guests.sh cannot stop Paused domains To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1803741/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1803741] Re: libvirt-guests.sh cannot stop Paused domains
Incomplete as we are waiting for the reporter to test the proposed solution before upstreaming. ** Changed in: libvirt (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1803741 Title: libvirt-guests.sh cannot stop Paused domains To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1803741/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1803741] Re: libvirt-guests.sh cannot stop Paused domains
Classic handling on 3 guests (normal, paused, uncooperative) is currently like: # /usr/lib/libvirt/libvirt-guests.sh stop Running guests on default URI: testguest-normal, testguest-paused, testguest-nocoop Shutting down guests on default URI... Starting shutdown on guest: testguest-normal Starting shutdown on guest: testguest-paused error: Failed to shutdown domain 50ee425f-899e-4489-b5b0-22a97c2a2f42 error: Requested operation is not valid: domain is not running Starting shutdown on guest: testguest-nocoop Waiting for 3 guests to shut down, 120 seconds left Shutdown of guest testguest-normal complete. Waiting for 2 guests to shut down, 115 seconds left Waiting for 2 guests to shut down, 110 seconds left ... Waiting for 2 guests to shut down, 10 seconds left Waiting for 2 guests to shut down, 5 seconds left Timeout expired while shutting down domains And it would leave all but testguest-normal normal up as-is Since this - if set as a default - would change behavior it is not backportable easily. In general I'd not want to change the "normal" stop code nor the behavior of the current config. Unless I'm convinced otherwise by people here or on the upstreaming. IMHO we could make the config of NEW Ubuntu releases default to the new way, but only drop the code to older releases leaving it as opt-in for those users. In my experimental code I tried to define a third option to shutdown/suspend which would fallback to suspend for those who won't shut down. I tried but failed to cover the uncooperative guests in an easy way - and the code should be upstreamable so I keep the non cooperative case out of this fix. Not sure yet if/what we should do with a guest that crashed or is uncooperative. On my example with the three guests the new code handles the paused guest fine: # /usr/lib/libvirt/libvirt-guests.sh stop Running guests on default URI: testguest-nocoop, testguest-paused, testguest-normal Shutting down guests on default URI... Starting shutdown on guest: testguest-nocoop Starting managedsave on guest: testguest-paused Starting shutdown on guest: testguest-normal Waiting for 3 guests to shut down, 120 seconds left Shutdown of guest testguest-paused complete. Shutdown of guest testguest-normal complete. Waiting for 1 guests to shut down, 115 seconds left Waiting for 1 guests to shut down, 110 seconds left ... Waiting for 1 guests to shut down, 10 seconds left Waiting for 1 guests to shut down, 5 seconds left Timeout expired while shutting down domains I have put this in a PPA [1]. Would you mind testing that PPA with your case? To do so install the libvirt from the PPA and then set ON_SHUTDOWN=auto in /etc/default/libvirt-guests Then run your case and let me know if that would resolve it. Maybe I also have an idea until then how to cover uncooperative guests in a less invasive way than my tries so far. [1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3545 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1803741 Title: libvirt-guests.sh cannot stop Paused domains To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1803741/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1803741] Re: libvirt-guests.sh cannot stop Paused domains
The issue is manyfold: - On paused domains Starting shutdown on guest: testguest3 error: Failed to shutdown domain 50ee425f-899e-4489-b5b0-22a97c2a2f42 error: Requested operation is not valid: domain is not running - on running but non cooperative domains The softer shutdown will be ignored by the guest TL;DR it can't shut down either type. Now for the non-cooperative case it is preferred to do nothing as the reasons why it doesn't shut down are unclear, so any action taken could be wrong. I can't help you a lot for that. While "shutdown" won't work on any type of the above it is clear that the suspend action works. It ignores any guest cooperation and just saves memory to a file to restore it later on. This will make the qemu process eventually disappear which is what you need in your case for LVM. But "suspend" is not concurrent and therefore (too slow when scaling) therefore it is not the default. I'm glad that we can consider "setting ON_SHUTDOWN=suspend" a config workaround for you. Now lets consider how we could make this smarter. We could in theory - fall back to call "managedsave" on ANY guest either non-cooperative or paused which would ensure it is freed up. I'd need to check what the implications of this are for other cases, but it seems like an interesting approach. Writing some test code later on and then hopefully having something to upstream. @gene - for when I upstream I'd like to add you as Reported-by, but I'd need your email for that. You can find my email on launchpad (just click on my user) - if you are ok that I add you let me know your address please. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1803741 Title: libvirt-guests.sh cannot stop Paused domains To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1803741/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1803741] Re: libvirt-guests.sh cannot stop Paused domains
My thought would be that it depends on the default of whether to shutdown or save guests on host shutdown. FYI changing the default to save works around the problem so only the setting to shut down the guests needs to be fixed. On Sat, Nov 24, 2018, 2:45 AM Christian Ehrhardt < 1803...@bugs.launchpad.net wrote: > This script was not perfect upstream for a long time (and isn't yet) - > earlier this year we fixed plenty of issues in there. > And as a note - whoever touched the script shivers, but then nobody wants > to do a full rewrite either. > Without checking it yet I absolutely expect that this is right and failing. > > Although, I'm not entirely sure what the right state for paused guests > after reboot would be. > Maybe shutting them down is the right thing, I'll have to investigate. > But in any case it shouldn't take ulta-long to do so. > > Thanks for the report Gene, I'll take a look as soon as time permits! > > -- > You received this bug notification because you are subscribed to the bug > report. > https://bugs.launchpad.net/bugs/1803741 > > Title: > libvirt-guests.sh cannot stop Paused domains > > Status in libvirt package in Ubuntu: > New > > Bug description: > When I shut down my KVM host with a guest in the Paused state libvirt- > guests.sh eventually times out unable to shutdown the guests. This has > the side effect of preventing LVM from stopping the LVs used by the > paused guest and I end up in an infinite retry loop. libvirt-guests.sh > should know how to handle a Paused guest. > > Description:Ubuntu 18.04.1 LTS > Release:18.04 > > qemu-kvm/bionic-updates,now 1:2.11+dfsg-1ubuntu7.7 amd64 [installed] > > Expected the guest to either be resumed and shutdown OR saved and it > state restored on reboot. > > > What happened: > libvirt-guests.ksh timed out and shutdown process proceeded with guest > still in the paused state. This had the side-effect of hanging a later > operation (LVM shutdown). > > To manage notifications about this bug go to: > > https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1803741/+subscriptions > -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1803741 Title: libvirt-guests.sh cannot stop Paused domains To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1803741/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1803741] Re: libvirt-guests.sh cannot stop Paused domains
This script was not perfect upstream for a long time (and isn't yet) - earlier this year we fixed plenty of issues in there. And as a note - whoever touched the script shivers, but then nobody wants to do a full rewrite either. Without checking it yet I absolutely expect that this is right and failing. Although, I'm not entirely sure what the right state for paused guests after reboot would be. Maybe shutting them down is the right thing, I'll have to investigate. But in any case it shouldn't take ulta-long to do so. Thanks for the report Gene, I'll take a look as soon as time permits! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1803741 Title: libvirt-guests.sh cannot stop Paused domains To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1803741/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1803741] Re: libvirt-guests.sh cannot stop Paused domains
Right. Even without LVM, the behavior I described is enough to show a bug I think. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1803741 Title: libvirt-guests.sh cannot stop Paused domains To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1803741/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1803741] Re: libvirt-guests.sh cannot stop Paused domains
If your guest is using LVM for it's root disk it will never shut down On Tue, Nov 20, 2018, 2:41 PM Andreas Hasenack I gave this a try with nested virtualization, where the host was a 18.04 > kvm, and there was another 18.04 guest inside it. I paused the guest, > shutdown the host, and I did see it take a very long time to shutdown. > When the host came back, the guest state was off, no longer paused. > > Subscribing @cpaelzer for his opinion on this. > > -- > You received this bug notification because you are subscribed to the bug > report. > https://bugs.launchpad.net/bugs/1803741 > > Title: > libvirt-guests.sh cannot stop Paused domains > > Status in libvirt package in Ubuntu: > New > > Bug description: > When I shut down my KVM host with a guest in the Paused state libvirt- > guests.sh eventually times out unable to shutdown the guests. This has > the side effect of preventing LVM from stopping the LVs used by the > paused guest and I end up in an infinite retry loop. libvirt-guests.sh > should know how to handle a Paused guest. > > Description:Ubuntu 18.04.1 LTS > Release:18.04 > > qemu-kvm/bionic-updates,now 1:2.11+dfsg-1ubuntu7.7 amd64 [installed] > > Expected the guest to either be resumed and shutdown OR saved and it > state restored on reboot. > > > What happened: > libvirt-guests.ksh timed out and shutdown process proceeded with guest > still in the paused state. This had the side-effect of hanging a later > operation (LVM shutdown). > > To manage notifications about this bug go to: > > https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1803741/+subscriptions > -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1803741 Title: libvirt-guests.sh cannot stop Paused domains To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1803741/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1803741] Re: libvirt-guests.sh cannot stop Paused domains
I gave this a try with nested virtualization, where the host was a 18.04 kvm, and there was another 18.04 guest inside it. I paused the guest, shutdown the host, and I did see it take a very long time to shutdown. When the host came back, the guest state was off, no longer paused. Subscribing @cpaelzer for his opinion on this. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1803741 Title: libvirt-guests.sh cannot stop Paused domains To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1803741/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
