On Fri, Sep 12, 2025 at 11:20:01AM +0100, Daniel P. Berrangé wrote: > On Wed, Sep 10, 2025 at 12:36:44PM -0400, Peter Xu wrote: > > On Wed, Sep 10, 2025 at 08:10:57AM +0100, Daniel P. Berrangé wrote: > > > On Tue, Sep 09, 2025 at 05:58:49PM -0400, Peter Xu wrote: > > > > On Tue, Sep 09, 2025 at 04:09:23PM +0100, Daniel P. Berrangé wrote: > > > > > On Tue, Sep 09, 2025 at 05:01:24PM +0200, Juraj Marcin wrote: > > > > > > From: Juraj Marcin <jmar...@redhat.com> > > > > > > > > > > > > Usual system defaults for TCP keep-alive options are too long for > > > > > > migration workload. On Linux, a TCP connection waits idle for 2 > > > > > > hours > > > > > > before it starts checking if the connection is not broken. > > > > > > > > > > > > Now when InetSocketAddress supports keep-alive options [1], this > > > > > > patch > > > > > > applies migration specific defaults if they are not supplied by the > > > > > > user > > > > > > or the management software. With these defaults, a migration TCP > > > > > > stream > > > > > > waits idle for 1 minute and then sends 5 TCP keep-alive packets in > > > > > > 30 > > > > > > second interval before considering the connection as broken. > > > > > > > > > > > > System defaults can be still used by explicitly setting these > > > > > > parameters > > > > > > to 0. > > > > > > > > > > IMHO this is not a good idea. This is a very short default, which > > > > > may be fine for the scenario where your network conn is permanently > > > > > dead, but it is going to cause undesirable failures when the network > > > > > conn is only temporarily dead. > > > > > > > > > > Optimizing defaults for temporary outages is much more preferrable > > > > > as that maximises reliability of migration. In the case of permanent > > > > > outages, it is already possible to tear down the connection without > > > > > waiting for a keep-alive timeout, and liveliness checks can also be > > > > > perform by the mgmt app at a higher level too. The TCP keepalives > > > > > are just an eventual failsafe, and having those work on a long > > > > > timeframe is OK. > > > > > > > > For precopy it looks fine indeed, because migrate_cancel should always > > > > work > > > > on src if src socket hanged, and even if dest QEMU socket hanged, it can > > > > simply be killed if src QEMU can be gracefully cancelled and rolled > > > > back to > > > > RUNNING, disregarding the socket status on dest QEMU. > > > > > > > > For postcopy, we could still use migrate_pause to enforce src > > > > shutdown(). > > > > Initially I thought we have no way of doing that for dest QEMU, but I > > > > just > > > > noticed two years ago I added that to dest QEMU for migrate_paused when > > > > working on commit f8c543e808f20b.. So looks like that part is covered > > > > too, > > > > so that if dest QEMU socket hanged we can also kick it out. > > > > > > > > I'm not 100% sure though, on whether shutdown() would always be able to > > > > successfully kick out the hanged socket while the keepalive is ticking. > > > > Is > > > > it guaranteed? > > > > > > I don't know about shutdown(), but close() certainly works. If shutdown() > > > is not sufficient, then IMHO the migration code would need the ability to > > > use close() to deal with this situation. > > > > > > > > > > I also am not sure if that happens, whether libvirt would automatically > > > > do > > > > that, or provide some way so the user can trigger that. The goal IIUC > > > > here > > > > is we shouldn't put user into a situation where the migration hanged but > > > > without any way to either cancel or recover. With the default values > > > > Juraj > > > > provided here, it makes sure the hang won't happen more than a few > > > > minutes, > > > > which sounds like a sane timeout value. > > > > > > Sufficient migration QMP commands should exist to ensure migration can > > > always be cancelled. Short keepalive timeouts should not be considered > > > a solution to any gaps in that respect. > > > > > > Also there is yank, but IMHO apps shouldn't have to rely on yank - I see > > > yank as a safety net for apps to workaround limitations in QEMU. > > > > The QMP facility looks to be all present, which is migrate-cancel and > > migrate-pause mentioned above. > > > > For migrate_cancel (of precopy), is that Ctrl-C of "virsh migrate"? > > > > Does libvirt exposes migrate_pause via any virsh command? IIUC that's the > > only official way of pausing a postcopy VM on either side. I also agree we > > shouldn't make yank the official tool to use. > > virsh will call virDomainAbortJob when Ctrl-C is done to a 'migrate' > command. > > virDomainAbortJob will call migrate-cancel for pre-copy, or > 'migrate-pause' for post-copy.
Would it call "migrate-pause" on both sides? I believe the problem we hit was, when during postcopy and the NIC was misfunctioning, src fell into postcopy-paused successfully but dest didn't, stuck in postcopy-active. We'll want to make sure both sides to be kicked into paused stage to recover. Otherwise dest can hang in the stage for hours until the watchdog timeout triggers. > > > > OTOH, the default timeouts work without changing libvirt, making sure the > > customers will not be stuck in a likely-failing network for hours without > > providing a way to properly detach and recover when it's wanted. > > "timeouts work" has the implicit assumpton that the only reason a > timeout will fire is due to a unrecoverable situation. IMHO that > assumption is not valid. I agree adjusting timeout is not the best. If we can have solid way to kick two sides out, I think indeed we don't need to change the timeout. If not, we may still need to provide a way to allow user to try continue when the user found that the network is behaving abnormal. Here adjusting timeout is slightly better than any adhoc socket timeout that we'll adjust: it's the migration timeout, and we only have two cases: (1) precopy, which is ok to fail and retried, (2) postcopy, which is also ok to fail and recovered. The timeout made some corner cases more aggressive to converge to a failure/retry condition, but since we know exactly the property of the sockets so it's better than when we know nothing about what the sockets are used for. Thanks, -- Peter Xu
