* Daniel P. Berrangé (berra...@redhat.com) wrote: > On Mon, May 11, 2020 at 01:07:18PM +0100, Dr. David Alan Gilbert wrote: > > * Daniel P. Berrangé (berra...@redhat.com) wrote: > > > On Mon, May 11, 2020 at 01:14:34PM +0200, Lukas Straub wrote: > > > > Hello Everyone, > > > > In many cases, if qemu has a network connection (qmp, migration, > > > > chardev, etc.) > > > > to some other server and that server dies or hangs, qemu hangs too. > > > > > > If qemu as a whole hangs due to a stalled network connection, that is a > > > bug in QEMU that we should be fixing IMHO. QEMU should be doing > > > non-blocking > > > I/O in general, such that if the network connection or remote server > > > stalls, > > > we simply stop sending I/O - we shouldn't ever hang the QEMU process or > > > main > > > loop. > > > > > > There are places in QEMU code which are not well behaved in this respect, > > > but many are, and others are getting fixed where found to be important. > > > > > > Arguably any place in QEMU code which can result in a hang of QEMU in the > > > event of a stalled network should be considered a security flaw, because > > > the network is untrusted in general. > > > > That's not really true of the 'management network' - people trust that > > and I don't see a lot of the qemu code getting fixed safely for all of > > them. > > It depends on the user / app / deployment scenario. In OpenStack alot of > work was done to beef up security between services on the mgmt network, > with TLS encryption as standard to reduce attack vectors. > > > > > These patches introduce the new 'yank' out-of-band qmp command to > > > > recover from > > > > these kinds of hangs. The different subsystems register callbacks which > > > > get > > > > executed with the yank command. For example the callback can shutdown() > > > > a > > > > socket. This is intended for the colo use-case, but it can be used for > > > > other > > > > things too of course. > > > > > > IIUC, invoking the "yank" command unconditionally kills every single > > > network connection in QEMU that has registered with the "yank" subsystem. > > > IMHO this is way too big of a hammer, even if we accept there are bugs in > > > QEMU not handling stalled networking well. > > > > But isn't this hammer conditional - I see that it's a migration > > capabiltiy for the migration socket, and a flag in nbd - so it only > > yanks things you've told it to. > > IIUC, you have to set these flags upfront when you launch QEMU, or > hotplug the device using the feature. When something gets stuck, > and you issue the "yank" command, then everything that has the flag > enabled gets torn down. So in practice it looks like the flag will > get enabled for everything at QEMU startup, and yanking down tear > down everything.
For COLO I really expect it for the migration stream, the disk mirroring stream and probably the network comparison/forwarding streams. > > > eg if a chardev hangs QEMU, and we tear down everything, killing the NBD > > > connection used for the guest disk, we needlessly break I/O. > > > > > > eg doing this in the chardev backend is not desirable, because the bugs > > > with hanging QEMU are typically caused by the way the frontend device > > > uses the chardev blocking I/O calls, instead of non-blocking I/O calls. > > > > > > > Having a way to get out of any of these problems from a single point is > > quite nice. To be useful in COLO you need to know for sure you can get > > out of any network screwup. > > > > We already use shutdown(2) in migrate_cancel and migrate-pause for > > basically the same reason; I don't think we've got anything similar for > > NBD, and we probably should have (I think I asked for it fairly > > recently). > > Yes, the migrate_cancel is an example of a more fine grained way to > recover. I was thinking that we need an equivalent fine control knob > for NBD too. I feel it might be nice not to have to create so many separate knobs. > That way if QEMU does get stuck, you can start by tearing down the > least distruptive channel. eg try tearing down the migration connection > first (which shouldn't negatively impact the guest), and only if that > doesn't work then, move on to tear down the NBD connection (which risks > data loss) I wonder if a different way would be to make all network connections register with yank, but then make yank take a list of connections to shutdown(2). Dave > Regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK