rp- opened a new pull request, #14194: URL: https://github.com/apache/cloudstack/pull/14194
### Description This PR adds incremental NAS backup support for KVM VMs whose disks live on raw block-device storage such as LINSTOR/DRBD. #13877 (backport of #13074) implements incremental NAS backups with QEMU dirty bitmaps and libvirt checkpoints. Bitmap persistence is a qcow2-only feature, so `allVolumesOnCheckpointCapableStorage()` excludes raw block devices and every LINSTOR VM stays pinned to full-only backups regardless of the configured cadence. On top of that, the push-mode full backup of a raw device writes a fully allocated qcow2 which then needs a second `qemu-img convert` pass to reclaim space. This PR adds a **content-based backup path driven by libvirt pull mode** and routes VMs on content-diff-capable storage (currently LINSTOR) to it: **`nasbackup.sh` – two new modes** libvirt is asked for a pull-mode backup: it starts an NBD server exposing a point-in-time view of each disk, held consistent by copy-before-write into host-local fleecing scratch images. The backup is then derived from that export: - `content-full`: `qemu-img convert` from the NBD export straight into a sparse qcow2 in a single pass. This replaces the bloated push-mode output and its re-convert pass. - `content-incremental`: a qcow2 overlay backed by the NBD export, then a *safe* `qemu-img rebase` onto the parent backup. This merges in exactly the clusters that differ and yields a delta qcow2 backed by the parent, with a relative backing path. The chain has the same shape as a bitmap incremental, so the existing restore and cascade-delete logic applies unchanged. A missing or short parent list degrades to a full and emits `INCREMENTAL_FALLBACK=true`, mirroring the push-mode fallback. Also handles the raw `/dev/drbdN` device paths of long-running VMs the same way the push/stopped paths do (via `get_linstor_uuid_from_device`). The scratch dir is host-local and configurable with `-S` (default `/var/tmp`). It must not be on the NAS share, since copy-before-write runs inline with guest writes. NBD socket paths are validated against the kernel's 108-byte UNIX socket limit up front instead of failing opaquely in `qemu-img` later. **`NASBackupProvider` – routing and cadence** - New `decideContentChain()` mirrors the bitmap cadence logic (incremental master switch, stopped VMs, `nas.backup.full.every`, chain length) but anchors on backup history alone. There is no host-side bitmap that can be invalidated, so a VM restart, live migration or restore cannot break the chain; content comparison is stateless and self-correcting. - The content path is taken even when `nas.backup.incremental.enabled` is off, because a `content-full` is still strictly better than the fully allocated push output it replaces. Incrementals themselves remain gated by the master switch. - `ChainDecision` carries the parent backup uuid directly, since a content chain has no bitmap name to resolve the parent by. - Mixed-storage VMs (e.g. root on LINSTOR, data disk on NFS) stay off the content path: one backup run uses one mode. **`LibvirtTakeBackupCommandWrapper` / agent** Accepts the two new modes (`content-incremental` requires only its per-disk parent paths, no bitmap) and passes the scratch directory from the new `agent.properties` key `nas.backup.pull.scratch.dir` (default `/var/tmp`). Behaviour for VMs on qcow2/NFS/Ceph storage is unchanged; they keep using the bitmap path from #13877. ### Types of changes - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] New feature (non-breaking change which adds functionality) - [ ] Bug fix (non-breaking change which fixes an issue) - [x] Enhancement (improves an existing feature and functionality) - [ ] Cleanup (Code refactoring and cleanup, that may add test cases) - [ ] Build/CI - [ ] Test (unit or integration test code) ### Feature/Enhancement Scale or Bug Severity #### Feature/Enhancement Scale - [ ] Major - [x] Minor #### Bug Severity - [ ] BLOCKER - [ ] Critical - [ ] Major - [ ] Minor - [ ] Trivial ### Screenshots (if appropriate): ### How Has This Been Tested? **Unit tests** Six new `NASBackupProviderTest` cases cover the routing and cadence decisions: - `decideChainReturnsContentFullForLinstorWhenIncrementalDisabled` - `decideChainReturnsContentFullForStoppedLinstorVm` - `decideChainReturnsContentIncrementalForLinstorWithValidParent` - `decideChainReturnsContentFullForLinstorAtChainEnd` - `allVolumesOnContentDiffCapableStorageFalseForMixedStorage` - `allVolumesOnContentDiffCapableStorageFalseForNoVolumes` All pre-existing `NASBackupProviderTest` and KVM backup wrapper tests still pass (27/27 and 20/20 after rebasing onto current 4.22). **End-to-end on a 3-host KVM/LINSTOR cluster** (management server + 3 agents, NFS backup repository, `nas.backup.incremental.enabled=true`, `nas.backup.full.every=3`): | # | Test case | Expected | Result | |---|-----------|----------|--------| | 1 | Backup running LINSTOR VM with feature disabled | `content-full`: single sparse qcow2 per disk, no bitmap/checkpoint, restore works | Pass | | 2 | Four consecutive backups of a running LINSTOR VM with `full.every=3` | FULL / INCREMENTAL / INCREMENTAL / FULL recorded in DB | Pass | | 3 | Backup size | 8 GiB volume: ~1.9 GiB sparse full; deltas of a few MiB after small guest writes | Pass | | 4 | Chain layout | Incrementals have a *relative* backing path to the parent; 3-level chain clean under `qemu-img check` | Pass | | 5 | Restore from the tail incremental | VM boots and answers on the network | Pass | | 6 | VM on qcow2/NFS storage | Still takes the bitmap path from #13877, no behaviour change | Pass | #### How did you try to break this feature and the system with this change? - **Long scratch path**: pointed `nas.backup.pull.scratch.dir` at a deep directory so the NBD socket path exceeded 108 bytes. The script fails up front with a clear message and a hint to shorten the dir, instead of an opaque `qemu-img` connection error mid-backup; cleanup runs and no partial backup dir is left behind. - **Scratch on the NAS share**: verified guest writes stall during the backup when copy-before-write goes to the remote share. This is why the scratch dir is host-local by default and documented as such. - **Missing parent**: deleted the parent backup files and requested an incremental. The script falls back to a full, signals `INCREMENTAL_FALLBACK=true`, and the DB records a full. - **VM restart / live migration between backups**: since no host-side bitmap exists, the next backup is still incremental and the delta is correct. This is the case where the bitmap chain has to restart. - **Mixed storage**: VM with a LINSTOR root disk and an NFS data disk stays on the legacy full-only path; no half-content, half-bitmap chain is produced. - **Failure mid-loop** (killed `qemu-img` during the pull): the backup job is aborted, scratch dir and partial output removed, mount released; the next backup succeeds. - **Legacy DRBD device paths** (`/dev/drbdN` instead of `/dev/drbd/by-res/…` in live XML): resolved via udev like the push path; if the UUID cannot be derived the backup fails hard rather than producing a file restore cannot find. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
