Current memory operations like pinning may take a lot of time at the destination. Currently they are done after the source of the migration is stopped, and before the workload is resumed at the destination. This is a period where neigher traffic can flow, nor the VM workload can continue (downtime).
We can do better as we know the memory layout of the guest RAM at the destination from the moment the migration starts. Moving that operation allows QEMU to communicate the kernel the maps while the workload is still running in the source, so Linux can start mapping them. Also, the destination of the guest memory may finish before the destination QEMU maps all the memory. In this case, the rest of the memory will be mapped at the same time as before applying this series, when the device is starting. So we're only improving with this series. If the destination has the switchover_ack capability enabled, the destination hold the migration until all the memory is mapped. This needs to be applied on top of [1]. That series performs some code reorganization that allows to map the guest memory without knowing the queue layout the guest configure on the device. This series reduced the downtime in the stop-and-copy phase of the live migration from 20s~30s to 5s, with a 128G mem guest and two mlx5_vdpa devices, per [2]. Future directions on top of this series may include: * Iterative migration of virtio-net devices, as it may reduce downtime per [3]. vhost-vdpa net can apply the configuration through CVQ in the destination while the source is still migrating. * Move more things ahead of migration time, like DRIVER_OK. * Check that the devices of the destination are valid, and cancel the migration in case it is not. v1 from RFC v2: * Hold on migration if memory has not been mapped in full with switchover_ack. * Revert map if the device is not started. RFC v2: * Delegate map to another thread so it does no block QMP. * Fix not allocating iova_tree if x-svq=on at the destination. * Rebased on latest master. * More cleanups of current code, that might be split from this series too. [1] https://lists.nongnu.org/archive/html/qemu-devel/2023-12/msg01986.html [2] https://lists.nongnu.org/archive/html/qemu-devel/2023-12/msg00909.html [3] https://lore.kernel.org/qemu-devel/6c8ebb97-d546-3f1c-4cdd-54e23a566...@nvidia.com/T/ Eugenio Pérez (12): vdpa: do not set virtio status bits if unneeded vdpa: make batch_begin_once early return vdpa: merge _begin_batch into _batch_begin_once vdpa: extract out _dma_end_batch from _listener_commit vdpa: factor out stop path of vhost_vdpa_dev_start vdpa: check for iova tree initialized at net_client_start vdpa: set backend capabilities at vhost_vdpa_init vdpa: add vhost_vdpa_load_setup vdpa: approve switchover after memory map in the migration destination vdpa: add vhost_vdpa_net_load_setup NetClient callback vdpa: add vhost_vdpa_net_switchover_ack_needed virtio_net: register incremental migration handlers include/hw/virtio/vhost-vdpa.h | 32 ++++ include/net/net.h | 8 + hw/net/virtio-net.c | 48 ++++++ hw/virtio/vhost-vdpa.c | 274 +++++++++++++++++++++++++++------ net/vhost-vdpa.c | 43 +++++- 5 files changed, 357 insertions(+), 48 deletions(-) -- 2.39.3
