calvix opened a new pull request, #13556:
URL: https://github.com/apache/cloudstack/pull/13556
Hello, this PR adds a feature to support an encrypted RBD volume for
CloudStack. As this is a necessary feature we need for infrastructure in our
company and we currently run custom build with this and we would like to have
it in the CloudStack upstream.
### Description
This PR adds Ceph-native (librbd) LUKS2 encryption for RBD volumes on the
KVM, covering both **data disks and root disks**. Encryption/decryption is done
transparently by qemu+librbd via `<encryption format='luks2' engine='librbd'>`.
The guest sees a normal virtio disk.
It reuses CloudStack's **existing passphrase escrow** , the same mechanism
as the current qemu-native volume encryption. RBD's `StoragePoolType`
encryption support is flipped from `Unsupported` to `Hypervisor`, and the KVM
agent gains `engine='librbd'` path for the create, boot, attach, and resize
flows.
The `rbd`/`qemu-img` CLI dependency is used because `rados-java` 0.x does
not expose the `rbd_encryption_format` API; but as both CLIs are already used
in the code, it's not a ned dependency.
#### Operations covered
| Operation | How it works | Provisioning |
|---|---|---|
| Create encrypted data volume (blank) | raw RBD image → `rbd encryption
format luks2` (standard `LUKS` header) | thin |
| Deploy encrypted root, template on the same Ceph cluster (thin CoW clone)
| reserve 16 MiB header on a protected template snapshot - `rbd clone` - LUKS2
format - encryption-aware resize | thin CoW (only header + CoW deltas written) |
| Deploy encrypted root, template elsewhere / first use from secondary (full
copy) | create empty image - LUKS2 format - `qemu-img convert -n` template
through the encryption layer | full copy |
| Attach / detach encrypted data disk (hot-plug) | libvirt disk gains
`<encryption format='luks2' engine='librbd'>` | - |
| Offline resize | `rbd resize --size … --encryption-passphrase-file`
(encryption-aware; grows the payload, keeps the header consistent) | — |
| Online resize | routed through `virsh blockresize` (librbd handles the
layout) | - |
| Boot / runtime decrypt | qemu + librbd decrypt in-process via
`engine='librbd'`; guest sees plaintext | - |
- Data-disk sizing: follows the existing block-encryption convention (cf.
ScaleIO `getUsableBytesFromRawBytes`) — the guest sees raw − header; the root
path additionally reserves the header so the OS image always fits.
- At rest: the raw RBD image is a LUKS2 container; data written inside the
guest does not appear in the raw Ceph objects (verified below). At runtime,
qemu+librbd decrypts transparently, so the guest and host see no crypto device.
#### Requirements/caveats
- A qemu/librbd build that supports `<encryption format='luks2'
engine='librbd'>`.
- **Data-disk hot-plug (attach to a running VM) needs libvirt ≥ 10.1.0.**
libvirt 10.0.0 has an object apply-order bug in `qemu_block.c` that emits the
encryption secret after the rbd blockdev, so a hot-plug fails with `No secret
with id 'xxxxxx'`.
This is fixed in version 10.1.0+. This bug does not affect booting from an
encrypted volume. The agent refuses this attach on older libvirt with a
corresponding error.
- RBD encryption uses librbd (`format='luks2' engine='librbd'`), a separate
stack from CloudStack's existing qemu-native LUKS1 encryption. librbd was
chosen because it can encrypt a **thin CoW clone of an unencrypted template
with a per-volume key** — qemu-native LUKS treats the whole image as one opaque
LUKS container, so it would force a **full, non-thin copy of the template for
every encrypted VM** (or share one key across all clones) — and because it's
LUKS2 and keeps RBD's native snapshot/clone/resize working. A volume is
decrypted by the same engine that encrypted it; the two are not mixed on a
single volume.
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Feature/Enhancement Scale
- [x] Major
- [ ] Minor
### How Has This Been Tested?
**Environment:** 9-node KVM + Ceph cluster - hosts on Ubuntu 26.04 (libvirt
12.0.0), Ceph 20.2 (dedicated ceph cluster + `ceph-common` on hosts),
management server 4.22.1.0.
**Unit tests** — new `RbdEncryptionTest` (9 tests, all green) asserts the
exact `rbd`/`qemu-img` argv built for `format`, `resize` and
convert-through-encryption (RBD and file sources), and that empty/null
passphrases are rejected. Command construction is verified without a live Ceph
cluster.
**End-to-end through the CloudStack API** on the CS cluster:
- Encrypted data disk: create - attach - in-guest `mkfs.ext4` + write +
remount + read-back - online resize - detach. Similar for offline resize.
- Encrypted root disk: deploy from template via thin CoW or full copy - both
boot to Ubuntu 24.04 (verified over the guest agent and a real SSH login).
- Raw image inspected with `cryptsetup luksDump`: LUKS2, `aes-xts-plain64`,
16 MiB data offset.
#### test to ensure wrong values/behavior will not cause unexpected or
hidden issues
- Empty / null passphrase on `format`/`resize`/`importTemplate` → guarded
with a clear error.
- Non-MiB-aligned resize request → `--size` rounds up so the volume never
ends up below the request.
- Same-cluster vs cross-cluster template → exercises Option A vs Option B.
- Root size larger than the template → encryption-aware grow of the clone.
- Repeated deploys from the same template → idempotent check on the shared
`-luks` snapshot.
- Ceph errors mid-operation → Rados/IoCTX/RbdImage handles released in
`finally`; temp cephx conf/keyring (0600) cleaned up on all paths.
- Non-encrypted volumes and the qemu-native encryption path are unchanged
(RBD stays on the librbd engine; other pool types keep their existing
behaviour).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
