Re: online blockdev-backup, a clarification (was: Summary on new backup interfaces in QEMU)
On 20.02.23 18:18, John Maline wrote:
As a qemu newcomer I had a related question and confusion from reading existing
docs. Searching qemu-block, this seemed related to my question so I’ll ask…
On Mar 15, 2022, at 12:57 PM, Vladimir Sementsov-Ogievskiy
wrote:
Hi all!
Here I want to summarize new interfaces and use cases for backup in QEMU.
TODO for me: convert this into good rst documentation in docs/.
The existing docs I found at
https://qemu.readthedocs.io/en/latest/interop/live-block-operations.html#live-disk-backup-blockdev-backup-and-the-deprecated-drive-backup
are confusing me. This, if I’m understanding, seem clearer.
OK, let's begin.
First, note that drive-backup qmp command is deprecated.
Next, some terminology:
push backup: the whole process is inside QEMU process, also may be called "internal
backup"
pull backup: QEMU only exports a kind of snapshot (for example by NBD), and third party
software reads this export and stores it somehow, also called "external backup"
copy-before-write operations: We usually do backup of active disk, guest is
running and may write to the disk during the process of backup. When guest
wants to rewrite data region which is not backed up yet, we must stop this
guest write, and copy original data to somewhere before continuing guest write.
That's a copy-before-write operation.
image-fleecing: the technique that allows to export a "snapshotted" state of the active disk with help of
copy-before-write operations. We create a temporary image - target for copy-before-write operations, and provide an
interface to the user to read the "snapshotted" state. And for read, we do read from temporary image the data
which is already changed in original active disk, and we read unchanged data directly from active disk. The temporary
image itself is also called "reverse delta" or "reversed delta".
== Simple push backup ==
Just use blockdev-backup, nothing new here. I just note some technical details,
that are relatively new:
1. First, backup job inserts copy-before-write filter above source disk, to do
copy-before-write operation.
2. Created copy-before-write filter shares internal block-copy state with
backup job, so they work in collaboration, to not copy same things twice.
The simple case I’m aiming for matches a push backup. I’m OK w/ a snapshot.
Environment - macos 12.6 on arm processor, guest is aarch64 centos linux using
hvf accelerator. Qemu 7.2.
I assume what you describe w/ copy-before-write is behavior in qemu 7.2. I’m
fine if the Linux client needs to do a bit of log replay if I revert to a
backup.
In the docs I link above it talks as if a VM shutdown is recommended after the
job completes. Seems to ruin the whole point of an online backup. I tried
instead finishing w/ a blockdev-del and I see the backup file closed by qemu.
I’m guessing that’s an appropriate way to flush/complete the backup. In an
experiment, it seemed the generated backup worked as expected.
Yes, shutdown is unrelated. Also, block-jobs do flush target on finish, so it's
really synced after block-job completion event. Still, blockdev-del(target) is
right thing to do.
I’m hoping for confirmation or correction on my approach.
Specifically I’m doing the following QMP commands.
{"execute": "qmp_capabilities"}
{"execute":"blockdev-add",
"arguments":{"node-name":"backup-node", "driver":"qcow2", "file":{"driver":"file",
"filename":"backups/backup1.img"}}
}
{"execute":"blockdev-backup",
"arguments":{"device":"drive0", "job-id":"job0", "target":"backup-node",
"sync":"full"}
}
... watch many job state change events ...
The last one should be BLOCK_JOB_COMPLETED, you wait for it, and check "error"
field - if it exist the job is failed.
You also can poll with query-block-jobs command.
{"execute":"blockdev-del",
"arguments": {"node-name":"backup-node"}
}
Yes your approach is correct. Note that ideally, you also should do fs-freeze /
fs-thaw in guest around blockdev-backup command call, to be sure that the
moment in time when we start the backup (the final target image will correspond
to this moment in time) is consistent and we'll be able to boot from the backup
image later.
--
Best regards,
Vladimir
online blockdev-backup, a clarification (was: Summary on new backup interfaces in QEMU)
As a qemu newcomer I had a related question and confusion from reading existing
docs. Searching qemu-block, this seemed related to my question so I’ll ask…
> On Mar 15, 2022, at 12:57 PM, Vladimir Sementsov-Ogievskiy
> wrote:
>
> Hi all!
>
> Here I want to summarize new interfaces and use cases for backup in QEMU.
>
> TODO for me: convert this into good rst documentation in docs/.
The existing docs I found at
https://qemu.readthedocs.io/en/latest/interop/live-block-operations.html#live-disk-backup-blockdev-backup-and-the-deprecated-drive-backup
are confusing me. This, if I’m understanding, seem clearer.
>
> OK, let's begin.
>
> First, note that drive-backup qmp command is deprecated.
>
> Next, some terminology:
>
> push backup: the whole process is inside QEMU process, also may be called
> "internal backup"
>
> pull backup: QEMU only exports a kind of snapshot (for example by NBD), and
> third party software reads this export and stores it somehow, also called
> "external backup"
>
> copy-before-write operations: We usually do backup of active disk, guest is
> running and may write to the disk during the process of backup. When guest
> wants to rewrite data region which is not backed up yet, we must stop this
> guest write, and copy original data to somewhere before continuing guest
> write. That's a copy-before-write operation.
>
> image-fleecing: the technique that allows to export a "snapshotted" state of
> the active disk with help of copy-before-write operations. We create a
> temporary image - target for copy-before-write operations, and provide an
> interface to the user to read the "snapshotted" state. And for read, we do
> read from temporary image the data which is already changed in original
> active disk, and we read unchanged data directly from active disk. The
> temporary image itself is also called "reverse delta" or "reversed delta".
>
>
>
> == Simple push backup ==
>
> Just use blockdev-backup, nothing new here. I just note some technical
> details, that are relatively new:
>
> 1. First, backup job inserts copy-before-write filter above source disk, to
> do copy-before-write operation.
> 2. Created copy-before-write filter shares internal block-copy state with
> backup job, so they work in collaboration, to not copy same things twice.
The simple case I’m aiming for matches a push backup. I’m OK w/ a snapshot.
Environment - macos 12.6 on arm processor, guest is aarch64 centos linux using
hvf accelerator. Qemu 7.2.
I assume what you describe w/ copy-before-write is behavior in qemu 7.2. I’m
fine if the Linux client needs to do a bit of log replay if I revert to a
backup.
In the docs I link above it talks as if a VM shutdown is recommended after the
job completes. Seems to ruin the whole point of an online backup. I tried
instead finishing w/ a blockdev-del and I see the backup file closed by qemu.
I’m guessing that’s an appropriate way to flush/complete the backup. In an
experiment, it seemed the generated backup worked as expected.
I’m hoping for confirmation or correction on my approach.
Specifically I’m doing the following QMP commands.
{"execute": "qmp_capabilities"}
{"execute":"blockdev-add",
"arguments":{"node-name":"backup-node", "driver":"qcow2",
"file":{"driver":"file", "filename":"backups/backup1.img"}}
}
{"execute":"blockdev-backup",
"arguments":{"device":"drive0", "job-id":"job0", "target":"backup-node",
"sync":"full"}
}
... watch many job state change events ...
{"execute":"blockdev-del",
"arguments": {"node-name":"backup-node"}
}
--
John Maline
[email protected]
Summary on new backup interfaces in QEMU
Hi all!
Here I want to summarize new interfaces and use cases for backup in QEMU.
TODO for me: convert this into good rst documentation in docs/.
OK, let's begin.
First, note that drive-backup qmp command is deprecated.
Next, some terminology:
push backup: the whole process is inside QEMU process, also may be called "internal
backup"
pull backup: QEMU only exports a kind of snapshot (for example by NBD), and third party
software reads this export and stores it somehow, also called "external backup"
copy-before-write operations: We usually do backup of active disk, guest is
running and may write to the disk during the process of backup. When guest
wants to rewrite data region which is not backed up yet, we must stop this
guest write, and copy original data to somewhere before continuing guest write.
That's a copy-before-write operation.
image-fleecing: the technique that allows to export a "snapshotted" state of the active disk with help of
copy-before-write operations. We create a temporary image - target for copy-before-write operations, and provide an
interface to the user to read the "snapshotted" state. And for read, we do read from temporary image the data
which is already changed in original active disk, and we read unchanged data directly from active disk. The temporary
image itself is also called "reverse delta" or "reversed delta".
== Simple push backup ==
Just use blockdev-backup, nothing new here. I just note some technical details,
that are relatively new:
1. First, backup job inserts copy-before-write filter above source disk, to do
copy-before-write operation.
2. Created copy-before-write filter shares internal block-copy state with
backup job, so they work in collaboration, to not copy same things twice.
== Full pull backup ==
Assume, we are going to do incremental backup in future, so we also need to
create a dirty bitmap, to track dirtiness of active disk since full backup.
1. Create empty temporary image for fleecing. It must be of the same size as
active disk. It's not necessary to be qcow2, and if it's a qcow2, you shouldn't
make the original active disk a backing file for the new temporary qcow2 image
(it was necessary in old fleecing scheme).
Example:
qemu-img create -f qcow2 temp.qcow2 64G
2. Initialize fleecing scheme and create dirty bitmap for future incremental
backup.
Assume, disk0 is an active disk, attached to qdev-id sda, to be backed up.
qmp: transaction [
block-dirty-bitmap-add {node: disk0, name: bitmap0, persistent: true}
blockdev-add* {node-name: tmp-protocol, driver: file, filename: temp.qcow2}
blockdev-add {node-name: tmp, driver: qcow2, file: tmp-protocol}
blockdev-add {node-name: cbw, driver: copy-before-write, file: disk0,
target: tmp}
blockdev-replace** {parent-type: qdev, qdev-id: sda, new-child: cbw}
blockdev-add {node-name: acc, driver: snapshot-access, file: cbw}
]
qmp: nbd-server-start {...}
qmp: nbd-server-add {device: acc, ...}
This way we create the following block-graph:
[guest] [NBD export]
||
| root | root
v file v
[copy-before-write]<--[snapshot-access]
| |
| file | target
v v
[active-disk] [temp.qcow2]
* "[PATCH 0/2] blockdev-add transaction" series needed for this
** "[PATCH v3 00/11] blockdev-replace" series needed for this
Note additional useful options for copy-before-write filter:
"[PATCH 0/3] block: copy-before-write: on-cbw-error behavior" provides
possibility of option on-cbw-error=break-snapshot, which means that on failure of CBW
operation we will not break guest write, but instead all further reads by NBD client will
fail, which formally means: break the backup process, not guest write.
"[PATCH 0/4] block: copy-before-write: cbw-timeout" provides an option
cbw-timeout, to set a timeout for CBW operations. That's very useful to avoid guest stuck.
3. Now third party backup tool can read data from NBD export
NBD_CMD_TRIM (discard) operation is supported on the export, it has the
following effects:
1. discard this data from temp image, if it is stored here
2. avoid further copy-before-write operation (guest is free to rewrite
corresponding data with no extra latency)
3. all further read requests from discarded areas by NBD client will fail
So, NBD client may discard regions that are already backed up to avoid extra
latency for guest writes and to free disk space on the host.
Possible TODO here is to implement NBD protocol extension, that allows to READ
& DISCARD in command. In this case we avoid extra command in the wire, but lose
possibility of retrying the READ operation if it failed.
4. After backup is complete, we should destroy the fleecing scheme:
qmp: nbd-server-stop
qmp: blockdev-del {node-name: acc}
qmp: blockdev-replace {parent-type: qdev, qdev-id: sda, new-child: disk0}
qmp: blockdev-del {node-name: cbw}
qmp: bl
