On Tue, Jul 22, 2025 at 12:41:25PM +0000, Jonah Palmer wrote:
> Lays out the initial groundwork for iteratively migrating the state of a
> virtio-net device, starting with its vmstate (via vmstate_save_state &
> vmstate_load_state).
>
> The original non-iterative vmstate framework still runs during the
> stop-and-copy phase when the guest is paused, which is still necessary
> to migrate over the final state of the virtqueues once the sourced has
> been paused.
>
> Although the vmstate framework is used twice (once during the iterative
> portion and once during the stop-and-copy phase), it appears that
> there's some modest improvement in guest-visible downtime when using a
> virtio-net device.
>
> When tracing the vmstate_downtime_save and vmstate_downtime_load
> tracepoints, for a virtio-net device using iterative live migration, the
> non-iterative downtime portion improved modestly, going from ~3.2ms to
> ~1.4ms:
>
> Before:
> -------
> vmstate_downtime_load type=non-iterable idstr=0000:00:03.0/virtio-net
> instance_id=0 downtime=3594
>
> After:
> ------
> vmstate_downtime_load type=non-iterable idstr=0000:00:03.0/virtio-net
> instance_id=0 downtime=1607
>
> This improvement is likely due to the initial vmstate_load_state call
> (while the guest is still running) "warming up" all related pages and
> structures on the destination. In other words, by the time the final
> stop-and-copy phase starts, the heavy allocations and page-fault
> latencies are reduced, making the device re-loads slightly faster and
> the guest-visible downtime window slightly smaller.
did I get it right it's just the vmstate load for this single device?
If the theory is right, is it not possible that while the
tracepoints are now closer together, you have pushed something
else out of the cache, making the effect on guest visible downtime
unpredictable? how about the total vmstate load time?
> Future patches could improve upon this by skipping the second
> vmstate_save/load_state calls (during the stop-and-copy phase) and
> instead only send deltas right before/after the source is stopped.
>
> Signed-off-by: Jonah Palmer <jonah.pal...@oracle.com>
> ---
> hw/net/virtio-net.c | 37 ++++++++++++++++++++++++++++++++++
> include/hw/virtio/virtio-net.h | 8 ++++++++
> 2 files changed, 45 insertions(+)
>
> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> index 19aa5b5936..86a6fe5b91 100644
> --- a/hw/net/virtio-net.c
> +++ b/hw/net/virtio-net.c
> @@ -3808,16 +3808,31 @@ static bool virtio_net_is_active(void *opaque)
>
> static int virtio_net_save_setup(QEMUFile *f, void *opaque, Error **errp)
> {
> + VirtIONet *n = opaque;
> +
> + qemu_put_be64(f, VNET_MIG_F_INIT_STATE);
> + vmstate_save_state(f, &vmstate_virtio_net, n, NULL);
> + qemu_put_be64(f, VNET_MIG_F_END_DATA);
> +
> return 0;
> }
>
> static int virtio_net_save_live_iterate(QEMUFile *f, void *opaque)
> {
> + bool new_data = false;
> +
> + if (!new_data) {
> + qemu_put_be64(f, VNET_MIG_F_NO_DATA);
> + return 1;
> + }
> +
> + qemu_put_be64(f, VNET_MIG_F_END_DATA);
> return 1;
> }
>
> static int virtio_net_save_live_complete_precopy(QEMUFile *f, void *opaque)
> {
> + qemu_put_be64(f, VNET_MIG_F_NO_DATA);
> return 0;
> }
>
> @@ -3833,6 +3848,28 @@ static int virtio_net_load_setup(QEMUFile *f, void
> *opaque, Error **errp)
>
> static int virtio_net_load_state(QEMUFile *f, void *opaque, int version_id)
> {
> + VirtIONet *n = opaque;
> + uint64_t flag;
> +
> + flag = qemu_get_be64(f);
> + if (flag == VNET_MIG_F_NO_DATA) {
> + return 0;
> + }
> +
> + while (flag != VNET_MIG_F_END_DATA) {
> + switch (flag) {
> + case VNET_MIG_F_INIT_STATE:
> + {
> + vmstate_load_state(f, &vmstate_virtio_net, n,
> VIRTIO_NET_VM_VERSION);
> + break;
> + }
> + default:
> + qemu_log_mask(LOG_GUEST_ERROR, "%s: Uknown flag 0x%"PRIx64,
> __func__, flag);
> + return -EINVAL;
> + }
> +
> + flag = qemu_get_be64(f);
> + }
> return 0;
> }
>
> diff --git a/include/hw/virtio/virtio-net.h b/include/hw/virtio/virtio-net.h
> index b9ea9e824e..d6c7619053 100644
> --- a/include/hw/virtio/virtio-net.h
> +++ b/include/hw/virtio/virtio-net.h
> @@ -163,6 +163,14 @@ typedef struct VirtIONetQueue {
> struct VirtIONet *n;
> } VirtIONetQueue;
>
> +/*
> + * Flags to be used as unique delimiters for virtio-net devices in the
> + * migration stream.
> + */
> +#define VNET_MIG_F_INIT_STATE (0xffffffffef200000ULL)
> +#define VNET_MIG_F_END_DATA (0xffffffffef200001ULL)
> +#define VNET_MIG_F_NO_DATA (0xffffffffef200002ULL)
> +
> struct VirtIONet {
> VirtIODevice parent_obj;
> uint8_t mac[ETH_ALEN];
> --
> 2.47.1
