rp- opened a new pull request, #14731:
URL: https://github.com/apache/cloudstack/pull/14731
### Description
This PR lets the Veeam integration for KVM (#12991) back up and restore VMs
whose volumes are on LINSTOR primary
storage, including incremental backups.
The KVM backup export only works for file-based pools: the host reads the
volume files through a libvirt backup job,
and incremental backups need qcow2 dirty bitmaps. Volumes on block storage
like LINSTOR can't be read that way.
**Volumes exported by their storage driver.** Storage drivers can now export
the volumes of their pools for a backup
themselves, with new default methods on `PrimaryDataStoreDriver`. Drivers
that don't implement them work as before.
- `startBackupExport` takes one crash consistent storage snapshot of the
VM's volumes and makes each volume readable as
a raw device on a host that the driver chooses. `finishBackupExport`
releases it.
- `prepareVolumeForRestore` makes a volume writable on a host for a restore.
For such VMs, `KVMBackupExportServiceImpl` skips the libvirt backup job and
the hypervisor checkpoints. It serves each
volume with its own qemu-nbd on the driver's host, for running and stopped
VMs alike. VMs with volumes on both kinds of
pools are rejected.
**Incremental backups.** For each volume, the driver gets the change
tracking id of the volume's last backup at the
VM's active checkpoint, and can mark the export as incremental. It can also
name a source for the extents: the host
then writes the extents of the export to a file, flagged dirty (changed
since the last backup) or zero (reads as
zeros). The image server serves these as the extents of the transfer,
instead of a dirty bitmap. Without such a
source, the whole export counts as changed.
**LINSTOR** implements this with changed block tracking (CBT) snapshots of
DRBD (LINSTOR REST API 1.30).
- A backup takes one multi snapshot with the CBT flag and opens a CBT
session per volume on a node with a diskful
copy. The snapshot is deleted when the backup is finalized.
- An incremental backup reads only the extents that changed since the last
backup's CBT id. If DRBD did not track
the changes since then, for example after a failed backup, that volume is
backed up in full.
- A full backup reads only the extents that were ever written (needs
drbd-utils 9.36.0).
- The Veeam adapter reports LINSTOR pools as block storage domains. It
reports their disks as QCOW2 with incremental
backup, as Veeam only backs up QCOW2 disks incrementally. The data still
goes through the image transfers as raw.
- With a LINSTOR controller older than REST API 1.30, VMs on LINSTOR stay
unsupported for Veeam, as before.
**qcow2 uploads into these volumes.** Veeam uploads the image of its worker
VM as qcow2, which the image server
writes to a file. So deploying a worker onto LINSTOR failed with "Only raw
images can be uploaded".
- A driver can now provide a temporary raw device for such an upload
(`createUploadStaging`, `deleteUploadStaging`).
The image is written to it as is.
- On finalize, the host checks that it is a qcow2 image without a backing
file or data file, and converts it into
the volume with `qemu-img convert`.
- LINSTOR creates a temporary resource with one replica for it.
**Two fixes in the Veeam integration that apply to all storage:**
- The image server now brings up its HTTP listener before its control
socket. The agent takes the control socket as
the sign that the server is ready. Before, the first transfer after the
server started could get "connection
refused": binding the HTTP socket took 5 s on a host whose DNS could not
resolve its own name.
- The Veeam worker VM now gets the static IP that Veeam passes in its
cloud-init user data. Creating the VM did not
save that IP, so the worker's NIC got another IP of the network, and Veeam
waited at "obtaining IP address".
java-linstor goes from 0.7.0 to 0.9.0 (REST API 1.30).
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Feature/Enhancement Scale
- [x] Major
- [ ] Minor
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [ ] Major
- [ ] Minor
- [ ] Trivial
### Screenshots (if appropriate):
### How Has This Been Tested?
Unit tests, new and extended:
- `KVMBackupExportServiceImplTest`
- `LinstorPrimaryDataStoreDriverImplTest`, `LinstorStorageAdaptorTest`,
`LinstorUtilTest`
- the wrapper tests for creating and finalizing image transfers and for
starting the NBD server
- `ServerAdapterTest`, `VolumeJoinVOToDiskConverterTest`
- the image server's Python tests
Test setup:
- a 3-node KVM cluster on Ubuntu 24.04, running CloudStack 4.23 with this
change;
- LINSTOR primary storage on LVM-thin;
- a LINSTOR controller with REST API 1.30 (the CBT API that is in LINSTOR
1.36.0-beta.1);
- release candidates of DRBD 9.3.5 and drbd-utils 9.36.0.
With Veeam Backup & Replication 13 (oVirt KVM plug-in), in an advanced zone:
- Veeam deployed its worker VM onto LINSTOR (qcow2 upload) and reached it on
its static IP.
- A full backup of a VM with a 3.5 GiB root disk read 2.2 GB in about 2
minutes. The rest of the disk had never been
written.
- The incremental backup after it read 48 MB, of which 20.6 MB had changed.
- A restore of the whole VM to a new VM wrote 3.5 GiB in about 2 minutes.
The restored disk matched the source, except
for blocks that the still running source had written after the backup.
With a script that drives the same API as Veeam, comparing every result byte
for byte against the devices:
- full backups of running and stopped VMs;
- restores into a new disk and into an existing disk;
- a chain of a full and two incremental backups, restored;
- a CBT snapshot taken outside of the backups: the next backup of that disk
is full, while the other disks stay
incremental;
- deleting the active checkpoint: the next backup is full;
- a backup after a live migration of the VM stays incremental;
- no snapshots or resources left behind in LINSTOR.
#### How did you try to break this feature and the system with this change?
- Lost change tracking (a CBT snapshot outside of the backups, a deleted
checkpoint): the volume is backed up in full
rather than with wrong extents.
- A VM with volumes on LINSTOR and on a file-based pool: rejected (unit
test).
- A qcow2 upload with a backing file: rejected on finalize (unit test). A
cancelled upload, or one whose transfer
can't be set up on the host, deletes the temporary device without
converting (unit tests).
- A failing LINSTOR snapshot or snapshot delete: the backup fails without
leaving CBT sessions open, and the export is
still released (unit tests).
- Older LINSTOR controller: the version check treats it as unsupported, so
VMs on LINSTOR are not offered to Veeam
(unit tests).
Found while testing, not changed in this PR:
- The worker VM powers itself off after its first boot, and Veeam starts it
again at once. CloudStack doesn't release
the capacity of a VM that powered itself off until the next capacity check
(up to 5 minutes later), so the start can
fail with insufficient capacity on small hosts.
- The worker only configures its network from a config drive, so its network
has to offer the UserData service
through the ConfigDrive provider. The documentation only says that the
network needs UserData.
- `POST /disks/{id}/reduce` returns "not implemented". Veeam calls it for
QCOW2 disks after a restore and tolerates
the error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]