rp- opened a new pull request, #14731:
URL: https://github.com/apache/cloudstack/pull/14731

   ### Description
   
   This PR lets the Veeam integration for KVM (#12991) back up and restore VMs 
whose volumes are on LINSTOR primary
   storage, including incremental backups.
   
   The KVM backup export only works for file-based pools: the host reads the 
volume files through a libvirt backup job,
   and incremental backups need qcow2 dirty bitmaps. Volumes on block storage 
like LINSTOR can't be read that way.
   
   **Volumes exported by their storage driver.** Storage drivers can now export 
the volumes of their pools for a backup
   themselves, with new default methods on `PrimaryDataStoreDriver`. Drivers 
that don't implement them work as before.
   
   - `startBackupExport` takes one crash consistent storage snapshot of the 
VM's volumes and makes each volume readable as
     a raw device on a host that the driver chooses. `finishBackupExport` 
releases it.
   - `prepareVolumeForRestore` makes a volume writable on a host for a restore.
   
   For such VMs, `KVMBackupExportServiceImpl` skips the libvirt backup job and 
the hypervisor checkpoints. It serves each
   volume with its own qemu-nbd on the driver's host, for running and stopped 
VMs alike. VMs with volumes on both kinds of
   pools are rejected.
   
   **Incremental backups.** For each volume, the driver gets the change 
tracking id of the volume's last backup at the
   VM's active checkpoint, and can mark the export as incremental. It can also 
name a source for the extents: the host
   then writes the extents of the export to a file, flagged dirty (changed 
since the last backup) or zero (reads as
   zeros). The image server serves these as the extents of the transfer, 
instead of a dirty bitmap. Without such a
   source, the whole export counts as changed.
   
   **LINSTOR** implements this with changed block tracking (CBT) snapshots of 
DRBD (LINSTOR REST API 1.30).
   
   - A backup takes one multi snapshot with the CBT flag and opens a CBT 
session per volume on a node with a diskful
     copy. The snapshot is deleted when the backup is finalized.
   - An incremental backup reads only the extents that changed since the last 
backup's CBT id. If DRBD did not track
     the changes since then, for example after a failed backup, that volume is 
backed up in full.
   - A full backup reads only the extents that were ever written (needs 
drbd-utils 9.36.0).
   - The Veeam adapter reports LINSTOR pools as block storage domains. It 
reports their disks as QCOW2 with incremental
     backup, as Veeam only backs up QCOW2 disks incrementally. The data still 
goes through the image transfers as raw.
   - With a LINSTOR controller older than REST API 1.30, VMs on LINSTOR stay 
unsupported for Veeam, as before.
   
   **qcow2 uploads into these volumes.** Veeam uploads the image of its worker 
VM as qcow2, which the image server
   writes to a file. So deploying a worker onto LINSTOR failed with "Only raw 
images can be uploaded".
   
   - A driver can now provide a temporary raw device for such an upload 
(`createUploadStaging`, `deleteUploadStaging`).
     The image is written to it as is.
   - On finalize, the host checks that it is a qcow2 image without a backing 
file or data file, and converts it into
     the volume with `qemu-img convert`.
   - LINSTOR creates a temporary resource with one replica for it.
   
   **Two fixes in the Veeam integration that apply to all storage:**
   
   - The image server now brings up its HTTP listener before its control 
socket. The agent takes the control socket as
     the sign that the server is ready. Before, the first transfer after the 
server started could get "connection
     refused": binding the HTTP socket took 5 s on a host whose DNS could not 
resolve its own name.
   - The Veeam worker VM now gets the static IP that Veeam passes in its 
cloud-init user data. Creating the VM did not
     save that IP, so the worker's NIC got another IP of the network, and Veeam 
waited at "obtaining IP address".
   
   java-linstor goes from 0.7.0 to 0.9.0 (REST API 1.30).
   
   ### Types of changes
   
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to change)
   - [x] New feature (non-breaking change which adds functionality)
   - [ ] Bug fix (non-breaking change which fixes an issue)
   - [ ] Enhancement (improves an existing feature and functionality)
   - [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
   - [ ] Build/CI
   - [ ] Test (unit or integration test code)
   
   ### Feature/Enhancement Scale or Bug Severity
   
   #### Feature/Enhancement Scale
   
   - [x] Major
   - [ ] Minor
   
   #### Bug Severity
   
   - [ ] BLOCKER
   - [ ] Critical
   - [ ] Major
   - [ ] Minor
   - [ ] Trivial
   
   ### Screenshots (if appropriate):
   
   ### How Has This Been Tested?
   
   Unit tests, new and extended:
   
   - `KVMBackupExportServiceImplTest`
   - `LinstorPrimaryDataStoreDriverImplTest`, `LinstorStorageAdaptorTest`, 
`LinstorUtilTest`
   - the wrapper tests for creating and finalizing image transfers and for 
starting the NBD server
   - `ServerAdapterTest`, `VolumeJoinVOToDiskConverterTest`
   - the image server's Python tests
   
   Test setup:
   
   - a 3-node KVM cluster on Ubuntu 24.04, running CloudStack 4.23 with this 
change;
   - LINSTOR primary storage on LVM-thin;
   - a LINSTOR controller with REST API 1.30 (the CBT API that is in LINSTOR 
1.36.0-beta.1);
   - release candidates of DRBD 9.3.5 and drbd-utils 9.36.0.
   
   With Veeam Backup & Replication 13 (oVirt KVM plug-in), in an advanced zone:
   
   - Veeam deployed its worker VM onto LINSTOR (qcow2 upload) and reached it on 
its static IP.
   - A full backup of a VM with a 3.5 GiB root disk read 2.2 GB in about 2 
minutes. The rest of the disk had never been
     written.
   - The incremental backup after it read 48 MB, of which 20.6 MB had changed.
   - A restore of the whole VM to a new VM wrote 3.5 GiB in about 2 minutes. 
The restored disk matched the source, except
     for blocks that the still running source had written after the backup.
   
   With a script that drives the same API as Veeam, comparing every result byte 
for byte against the devices:
   
   - full backups of running and stopped VMs;
   - restores into a new disk and into an existing disk;
   - a chain of a full and two incremental backups, restored;
   - a CBT snapshot taken outside of the backups: the next backup of that disk 
is full, while the other disks stay
     incremental;
   - deleting the active checkpoint: the next backup is full;
   - a backup after a live migration of the VM stays incremental;
   - no snapshots or resources left behind in LINSTOR.
   
   #### How did you try to break this feature and the system with this change?
   
   - Lost change tracking (a CBT snapshot outside of the backups, a deleted 
checkpoint): the volume is backed up in full
     rather than with wrong extents.
   - A VM with volumes on LINSTOR and on a file-based pool: rejected (unit 
test).
   - A qcow2 upload with a backing file: rejected on finalize (unit test). A 
cancelled upload, or one whose transfer
     can't be set up on the host, deletes the temporary device without 
converting (unit tests).
   - A failing LINSTOR snapshot or snapshot delete: the backup fails without 
leaving CBT sessions open, and the export is
     still released (unit tests).
   - Older LINSTOR controller: the version check treats it as unsupported, so 
VMs on LINSTOR are not offered to Veeam
     (unit tests).
   
   Found while testing, not changed in this PR:
   
   - The worker VM powers itself off after its first boot, and Veeam starts it 
again at once. CloudStack doesn't release
     the capacity of a VM that powered itself off until the next capacity check 
(up to 5 minutes later), so the start can
     fail with insufficient capacity on small hosts.
   - The worker only configures its network from a config drive, so its network 
has to offer the UserData service
     through the ConfigDrive provider. The documentation only says that the 
network needs UserData.
   - `POST /disks/{id}/reduce` returns "not implemented". Veeam calls it for 
QCOW2 disks after a restore and tolerates
     the error.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to