This is an automated email from the ASF dual-hosted git repository. weizhouapache pushed a commit to branch 4.23.0-ceph in repository https://gitbox.apache.org/repos/asf/cloudstack.git
commit af2ca91bdfb5e5ca20e2ec69d23f5bb05dcd76fd Author: calvix <[email protected]> AuthorDate: Wed Sep 16 08:10:05 2026 +0200 storage: KVM - enable RBD/Ceph volume encryption support (#13556) * storage: enable RBD/Ceph volume encryption support (shared base) Flip StoragePoolType.RBD from EncryptionSupport.Unsupported to Hypervisor so the existing encryption control plane (allocator, endpoint selector, offerings) treats RBD pools as encryption-capable. The agent-side encrypted RBD create path is not implemented yet; it is delivered by two follow-up tracks (qemu-native engine='qemu' and ceph-native engine='librbd'). Until then, fail closed at the two RBD create chokepoints in LibvirtStorageAdaptor (createPhysicalDisk and createDiskFromTemplate) when a passphrase is present, so we never silently produce a plaintext volume that the control plane believes is encrypted. No change for existing unencrypted RBD volumes (guards only fire when a passphrase is set; supportsEncryption() only affects volumes that require encryption). * kvm: Ceph-native LUKS2 encryption for RBD volumes (engine='librbd') Implements encrypted RBD data and root disks using librbd's native LUKS2 encryption, decrypted at runtime by libvirt/qemu via <encryption engine='librbd'>. CloudStack manages the passphrase (existing model). - RbdEncryption: isolated helper wrapping `rbd encryption format luks2`, cephx via --id + keyfile (secret not on the command line), LUKS passphrase via KeyFile. Kept separate so the CLI can later be swapped for a JNA binding (rados-java has no rbd_encryption_format API). - LibvirtStorageAdaptor: create/clone the raw RBD image, then apply `rbd encryption format luks2`; mark the disk LUKS2 so encrypt_format propagates to the volume. Replaces the fail-closed guards. - QemuObject.EncryptFormat: add LUKS2. - LibvirtVMDef: render <encryption format='luks2' engine='librbd'>; the encrypt details now carry an optional engine. - attach (KVMStorageProcessor) and boot (LibvirtComputingResource): set engine='librbd' for RBD-backed encrypted volumes. NOTE: the CoW-clone-then-format path (encrypted root from an unencrypted template) needs live-cluster validation for the parent-grow / usable-size behaviour described in the Ceph image-encryption docs. Builds: api + plugins/hypervisors/kvm (JDK11). * kvm: gate host encryption probe on librbd support for RBD hostSupportsVolumeEncryption() now advertises encryption capability if the host supports EITHER qemu-native LUKS (qemu-img LUKS + cryptsetup) OR librbd native encryption (rbd CLI with the encryption subcommand). Previously a Ceph-only host that lacked cryptsetup would not advertise encryption even though librbd can encrypt RBD volumes. Split into hostSupportsQemuNativeVolumeEncryption() and hostSupportsRbdVolumeEncryption(); kept HOST_VOLUME_ENCRYPTION as the single host-wide flag (documented limitation: not per-pool). * kvm: resize support for librbd-encrypted RBD volumes (#5) Encrypted RBD volumes are encrypted natively by librbd and must be resized with `rbd resize --encryption-passphrase-file` so librbd grows the encrypted payload and keeps the LUKS header consistent. The existing encrypted-resize path (resizeEncryptedQcowFile) uses qemu-img --object secret, which is for qemu-native LUKS and does not fit the librbd LUKS2 layout. - RbdEncryption.resize(): new `rbd resize` wrapper (cephx via --id + keyfile, passphrase via KeyFile, optional --allow-shrink). - LibvirtResizeVolumeCommandWrapper: detect encrypted RBD and route to the rbd resize path, bypassing the libvirt v.resize and qemu-img paths. Snapshot/revert, RBD<->RBD copy, and migration of encrypted RBD volumes need no code changes: they operate on the raw (LUKS-containing) image at the block level, and the destination passphrase secret is already created engine-agnostic in LibvirtPrepareForMigrationCommandWrapper. These still require live validation. Builds: plugins/hypervisors/kvm (JDK11). * kvm: route online resize of encrypted RBD through virsh blockresize For a running VM, an librbd-encrypted RBD volume must be resized in-band by qemu/librbd, not out-of-band by the rbd CLI. Gate the CLI rbd-resize path on !vmIsRunning so: - offline -> `rbd resize --encryption-passphrase-file` (librbd-aware), and - online -> existing NOTIFYONLY path -> virsh blockresize, where qemu's block_resize delegates to librbd to grow the encrypted payload and notify the guest in one step (no passphrase needed; qemu holds the secret). This avoids notify-less out-of-band growth and qemu/librbd size divergence while the image is open. Online behaviour still needs live validation that blockresize resizes the encrypted payload for engine='librbd' disks. * kvm: encrypted RBD root disks (thin CoW clone + full-copy fallback) Root disks could not be encrypted: cloning a plaintext template and then `rbd encryption format`ing the clone leaves the inherited OS data unreadable (the LUKS header offsets it), so the guest could not mount root. Fix, in createDiskFromTemplateOnRBD, with two paths: - Option A (same-cluster cached RBD template): grow the template base to reserve LUKS2 header space, snapshot+protect it (cloudstack-base-snap-luks), clone from it, apply the LUKS2 header, resize the clone to the requested size. Inherited template data stays readable through the clone's encryption and the clone is a thin CoW image (only the header is written). - Option B (first-use / non-RBD template): create an empty image, apply a LUKS2 header, then import the template THROUGH the encryption layer via RbdEncryption.importTemplate (qemu-img convert -n into encrypt.key-secret). Correct but a full copy. Validated end-to-end on Ubuntu 26.04 / libvirt 12.0.0: both boot; A is thin (3.5 GiB provisioned, ~120 MiB used); LUKS2 verified at rest on Ceph. * kvm: harden and align librbd-encrypted RBD volume code Review pass over the librbd LUKS2 encryption feature to fix latent issues and bring it in line with CloudStack conventions: - RbdEncryption: reject empty/null passphrase with a clear error; round rbd --size up to MiB so a non-aligned request never shrinks the volume below what was asked for; create the temporary cephx conf/keyring 0600 explicitly instead of relying on the umask. - LibvirtStorageAdaptor: close Rados/IoCTX/RbdImage in a finally block on the encrypted-root paths (mirrors deleteVolume) so handles are not leaked on exceptions; use parameterized log messages instead of string concatenation; extract the encrypted-root Option A/B logic into createEncryptedRootCoWClone / createEncryptedRootFullCopy. - RbdEncryption: use an instance logger (matching the plugin convention) and split argv construction into build{Format,Resize,Convert}Script so the generated commands can be unit-tested. * kvm: add RbdEncryption unit tests Assert the rbd/qemu-img argv built for format, resize and convert-through-encryption (RBD and file sources), and that empty/null passphrases are rejected. Command construction is verified without a live Ceph cluster. * kvm: refuse encrypted RBD hot-plug on libvirt < 10.1.0 libvirt 10.0.0 has an object apply-order bug (fixed in 10.1.0) that breaks hot-plug of an encrypted rbd blockdev: on attach the disk is opened before its LUKS secret object is defined, so the attach fails with "No secret with id '...-format-encryption-secret0'". Booting a VM from an encrypted RBD disk is unaffected (the QEMU command line resolves all -object before -blockdev). Refuse the attach up front with a clear error (mirroring the existing openvswitch/io_uring libvirt-version gates) instead of letting libvirt fail opaquely. Only the RBD hot-plug path is gated; boot/root/detach are untouched. * docs: add PendingReleaseNotes entry for librbd-encrypted RBD volumes * kvm: route the encrypted RBD template import through QemuImg RbdEncryption built its own 'qemu-img convert' command line, which duplicated qemu-img knowledge outside of QemuImg. QemuImg could only write to a plain filename destination, so importing a template through the librbd encryption layer was not expressible with it. QemuImg now supports a destination described by image options (--target-image-opts, with -n implied since such a target always exists already), exposed as convertIntoExistingTarget(). QemuImageOptions can render its parameters under either image-opts flag. RbdEncryption.importTemplate now composes QemuImageOptions and a QemuObject secret and delegates to QemuImg; its hand-built convert script is removed. The rbd CLI calls (encryption format, resize, support probe) stay, as qemu-img cannot perform them. No functional change to the generated command. * kvm: use readable variable names in the encrypted RBD root helpers Review feedback: single-letter and abbreviated names are hard to read. Renamed in the two methods added by this PR only (renaming the rest of the class is out of scope here): r -> radosConnection, io -> ioContext, rbd -> rbdClient, base -> templateImage, s -> snapshotInfo, encSnap -> luksReservedSnapshotName, haveEncSnap -> luksSnapshotExists, createSize -> imageSizeWithLuksHeader, srcIsRbd -> sourceIsRbdPool. No functional change. * server, kvm: report and require RBD volume encryption separately Review feedback: distinguish the two volume encryption mechanisms instead of advertising them under one host flag. host.volume.encryption goes back to meaning qemu-native LUKS only (qemu-img LUKS + cryptsetup), as it did before this PR, and hosts now additionally report host.volume.encryption.rbd for librbd encryption (rbd encryption format). The deployment planner requires the flag matching the pool type of each encrypted volume - librbd for volumes on RBD pools, qemu-native for any other pool type - at all three places it validated encryption support before. The pool is taken from the pools proposed alongside the host when present, so first deployments are matched accurately too; an encrypted volume with no pool yet accepts either mechanism and the storage pool allocator picks a pool the host can serve. This also stops a host whose librbd is too old for 'rbd encryption format' from being selected for encrypted RBD volumes; it previously advertised encryption through the qemu stack and the VM failed to start. --------- Co-authored-by: Václav Rozsypálek <[email protected]> Co-authored-by: calvix <[email protected]> --- PendingReleaseNotes | 9 + api/src/main/java/com/cloud/host/Host.java | 1 + api/src/main/java/com/cloud/storage/Storage.java | 2 +- .../kvm/resource/LibvirtComputingResource.java | 19 +- .../hypervisor/kvm/resource/LibvirtVMDef.java | 13 +- .../wrapper/LibvirtResizeVolumeCommandWrapper.java | 32 ++- .../kvm/storage/KVMStorageProcessor.java | 27 +- .../kvm/storage/LibvirtStorageAdaptor.java | 150 ++++++++++- .../cloudstack/utils/qemu/QemuImageOptions.java | 11 +- .../org/apache/cloudstack/utils/qemu/QemuImg.java | 56 +++- .../apache/cloudstack/utils/qemu/QemuObject.java | 1 + .../apache/cloudstack/utils/rbd/RbdEncryption.java | 294 +++++++++++++++++++++ .../kvm/storage/KVMStorageProcessorTest.java | 22 ++ .../cloudstack/utils/rbd/RbdEncryptionTest.java | 166 ++++++++++++ .../deploy/DeploymentPlanningManagerImpl.java | 59 ++++- .../deploy/DeploymentPlanningManagerImplTest.java | 73 +++++ 16 files changed, 907 insertions(+), 28 deletions(-) diff --git a/PendingReleaseNotes b/PendingReleaseNotes index 9670b6e7c13..c5f68918fd5 100644 --- a/PendingReleaseNotes +++ b/PendingReleaseNotes @@ -39,3 +39,12 @@ example.ver.1 > example.ver.2: which can now be attached to Instances. This is to prevent the Secondary Storage to grow to enormous sizes as Linux Distributions keep growing in size while a stripped down Linux should fit on a 2.88MB floppy. + +4.23.0.0 > 24.0.0: + * KVM/Ceph: RBD volumes can now be encrypted at rest using native librbd + LUKS2 (<encryption format='luks2' engine='librbd'>), for both data disks + and root disks. Encryption is transparent to the guest and reuses the + existing CloudStack volume-encryption passphrase handling, so no + additional key store is required. Note: attaching an encrypted RBD volume + to a running Instance requires libvirt >= 10.1.0; booting an Instance from + an encrypted RBD root disk works on older libvirt. diff --git a/api/src/main/java/com/cloud/host/Host.java b/api/src/main/java/com/cloud/host/Host.java index c110e4ca94e..cca2edd70c6 100644 --- a/api/src/main/java/com/cloud/host/Host.java +++ b/api/src/main/java/com/cloud/host/Host.java @@ -56,6 +56,7 @@ public interface Host extends StateObject<Status>, Identity, Partition, HAResour String HOST_UEFI_ENABLE = "host.uefi.enable"; String HOST_VOLUME_ENCRYPTION = "host.volume.encryption"; + String HOST_RBD_VOLUME_ENCRYPTION = "host.volume.encryption.rbd"; String HOST_INSTANCE_CONVERSION = "host.instance.conversion"; String HOST_VDDK_SUPPORT = "host.vddk.support"; String HOST_VDDK_LIB_DIR = "vddk.lib.dir"; diff --git a/api/src/main/java/com/cloud/storage/Storage.java b/api/src/main/java/com/cloud/storage/Storage.java index 3511b4e88cb..275f6d25268 100644 --- a/api/src/main/java/com/cloud/storage/Storage.java +++ b/api/src/main/java/com/cloud/storage/Storage.java @@ -172,7 +172,7 @@ public class Storage { LVM(false, false, EncryptionSupport.Unsupported), // XenServer local LVM SR CLVM(true, false, EncryptionSupport.Unsupported), CLVM_NG(true, false, EncryptionSupport.Hypervisor), - RBD(true, true, EncryptionSupport.Unsupported), // http://libvirt.org/storage.html#StorageBackendRBD + RBD(true, true, EncryptionSupport.Hypervisor), // http://libvirt.org/storage.html#StorageBackendRBD ; encrypted natively by librbd (LUKS2, engine='librbd') SharedMountPoint(true, true, EncryptionSupport.Hypervisor), VMFS(true, true, EncryptionSupport.Unsupported), // VMware VMFS storage PreSetup(true, true, EncryptionSupport.Unsupported), // for XenServer, Storage Pool is set up by customers. diff --git a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java index 4281036d945..9946c85cfd9 100644 --- a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java +++ b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java @@ -19,6 +19,7 @@ package com.cloud.hypervisor.kvm.resource; import static com.cloud.host.Host.HOST_CDROM_MAX_COUNT; import static com.cloud.host.Host.HOST_INSTANCE_CONVERSION; import static com.cloud.host.Host.HOST_OVFTOOL_VERSION; +import static com.cloud.host.Host.HOST_RBD_VOLUME_ENCRYPTION; import static com.cloud.host.Host.HOST_VDDK_LIB_DIR; import static com.cloud.host.Host.HOST_VDDK_SUPPORT; import static com.cloud.host.Host.HOST_VDDK_VERSION; @@ -91,6 +92,7 @@ import org.apache.cloudstack.storage.to.VolumeObjectTO; import org.apache.cloudstack.storage.volume.VolumeOnStorageTO; import org.apache.cloudstack.utils.bytescale.ByteScaleUtils; import org.apache.cloudstack.utils.cryptsetup.CryptSetup; +import org.apache.cloudstack.utils.rbd.RbdEncryption; import org.apache.cloudstack.utils.hypervisor.HypervisorUtils; import org.apache.cloudstack.utils.linux.CPUStat; import org.apache.cloudstack.utils.linux.KVMHostInfo; @@ -3882,7 +3884,9 @@ public class LibvirtComputingResource extends ServerResourceBase implements Serv if (volumeObjectTO.requiresEncryption() && pool.getType().encryptionSupportMode() == Storage.EncryptionSupport.Hypervisor ) { String secretUuid = createLibvirtVolumeSecret(conn, volumeObjectTO.getPath(), volumeObjectTO.getPassphrase()); - DiskDef.LibvirtDiskEncryptDetails encryptDetails = new DiskDef.LibvirtDiskEncryptDetails(secretUuid, QemuObject.EncryptFormat.enumValue(volumeObjectTO.getEncryptFormat())); + // RBD volumes are encrypted natively by librbd, so request the librbd encryption engine. + String encryptEngine = (pool.getType() == StoragePoolType.RBD) ? "librbd" : null; + DiskDef.LibvirtDiskEncryptDetails encryptDetails = new DiskDef.LibvirtDiskEncryptDetails(secretUuid, QemuObject.EncryptFormat.enumValue(volumeObjectTO.getEncryptFormat()), encryptEngine); disk.setLibvirtDiskEncryptDetails(encryptDetails); } } @@ -4410,6 +4414,7 @@ public class LibvirtComputingResource extends ServerResourceBase implements Serv cmd.setGatewayIpAddress(localGateway); cmd.setIqn(getIqn()); cmd.getHostDetails().put(HOST_VOLUME_ENCRYPTION, String.valueOf(hostSupportsVolumeEncryption())); + cmd.getHostDetails().put(HOST_RBD_VOLUME_ENCRYPTION, String.valueOf(hostSupportsRbdVolumeEncryption())); cmd.setHostTags(getHostTags()); boolean instanceConversionSupported = hostSupportsInstanceConversion(); cmd.getHostDetails().put(HOST_INSTANCE_CONVERSION, String.valueOf(instanceConversionSupported)); @@ -6195,7 +6200,10 @@ public class LibvirtComputingResource extends ServerResourceBase implements Serv } /** - * Test host for volume encryption support + * Test host for qemu-native LUKS volume encryption (qemu-img LUKS support + cryptsetup), + * reported as {@code host.volume.encryption}. RBD/librbd encryption support is a separate + * capability, reported as {@code host.volume.encryption.rbd} + * (see {@link #hostSupportsRbdVolumeEncryption()}). * @return boolean */ public boolean hostSupportsVolumeEncryption() { @@ -6220,6 +6228,13 @@ public class LibvirtComputingResource extends ServerResourceBase implements Serv return true; } + /** + * Test host for librbd native LUKS encryption support (rbd CLI with the encryption subcommand). + */ + public boolean hostSupportsRbdVolumeEncryption() { + return new RbdEncryption().isSupported(); + } + public boolean isSecureMode(String bootMode) { if (StringUtils.isNotBlank(bootMode) && "secure".equalsIgnoreCase(bootMode)) { return true; diff --git a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtVMDef.java b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtVMDef.java index 74529d9d5fa..439e4f66341 100644 --- a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtVMDef.java +++ b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtVMDef.java @@ -788,14 +788,21 @@ public class LibvirtVMDef { public static class LibvirtDiskEncryptDetails { String passphraseUuid; QemuObject.EncryptFormat encryptFormat; + String engine; // optional libvirt encryption engine (e.g. "librbd"); null => libvirt/qemu default public LibvirtDiskEncryptDetails(String passphraseUuid, QemuObject.EncryptFormat encryptFormat) { + this(passphraseUuid, encryptFormat, null); + } + + public LibvirtDiskEncryptDetails(String passphraseUuid, QemuObject.EncryptFormat encryptFormat, String engine) { this.passphraseUuid = passphraseUuid; this.encryptFormat = encryptFormat; + this.engine = engine; } public String getPassphraseUuid() { return this.passphraseUuid; } public QemuObject.EncryptFormat getEncryptFormat() { return this.encryptFormat; } + public String getEngine() { return this.engine; } } public static class DiskGeometry { @@ -1446,7 +1453,11 @@ public class LibvirtVMDef { } if (encryptDetails != null) { - diskBuilder.append("<encryption format='" + encryptDetails.encryptFormat + "'>\n"); + diskBuilder.append("<encryption format='" + encryptDetails.encryptFormat + "'"); + if (encryptDetails.engine != null) { + diskBuilder.append(" engine='" + encryptDetails.engine + "'"); + } + diskBuilder.append(">\n"); diskBuilder.append("<secret type='passphrase' uuid='" + encryptDetails.passphraseUuid + "' />\n"); diskBuilder.append("</encryption>\n"); } diff --git a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtResizeVolumeCommandWrapper.java b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtResizeVolumeCommandWrapper.java index a43b584dd6d..20e3891478b 100644 --- a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtResizeVolumeCommandWrapper.java +++ b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtResizeVolumeCommandWrapper.java @@ -33,6 +33,7 @@ import org.apache.cloudstack.utils.qemu.QemuImg; import org.apache.cloudstack.utils.qemu.QemuImg.PhysicalDiskFormat; import org.apache.cloudstack.utils.qemu.QemuImgException; import org.apache.cloudstack.utils.qemu.QemuObject; +import org.apache.cloudstack.utils.rbd.RbdEncryption; import org.libvirt.Connect; import org.libvirt.Domain; import org.libvirt.DomainInfo; @@ -93,6 +94,13 @@ public final class LibvirtResizeVolumeCommandWrapper extends CommandWrapper<Resi final String path = vol.getPath(); String type = notifyOnlyType; + // Encrypted RBD volumes are encrypted natively by librbd; they must be resized via + // `rbd resize --encryption-passphrase-file` so librbd grows the encrypted payload and keeps + // the LUKS header consistent. The libvirt/qemu-img resize paths below are for qemu-native + // encryption and would not handle the librbd LUKS2 layout. + final boolean rbdEncrypted = pool.getType() == StoragePoolType.RBD + && command.getPassphrase() != null && command.getPassphrase().length > 0; + if (spool.getType().equals(StoragePoolType.PowerFlex) && vol.getFormat().equals(PhysicalDiskFormat.QCOW2)) { // PowerFlex QCOW2 sizing needs to consider overhead. newSize = ScaleIOStorageAdaptor.getUsableBytesFromRawBytes(newSize); @@ -115,7 +123,7 @@ public final class LibvirtResizeVolumeCommandWrapper extends CommandWrapper<Resi /* libvirt doesn't support resizing (C)LVM devices, and corrupts QCOW2 in some scenarios, so we have to do these via qemu-img */ if (pool.getType() != StoragePoolType.CLVM && pool.getType() != StoragePoolType.CLVM_NG && pool.getType() != StoragePoolType.Linstor && pool.getType() != StoragePoolType.PowerFlex - && vol.getFormat() != PhysicalDiskFormat.QCOW2) { + && vol.getFormat() != PhysicalDiskFormat.QCOW2 && !rbdEncrypted) { logger.debug("Volume " + path + " can be resized by libvirt. Asking libvirt to resize the volume."); try { final LibvirtUtilitiesHelper libvirtUtilitiesHelper = libvirtComputingResource.getLibvirtUtilitiesHelper(); @@ -139,11 +147,15 @@ public final class LibvirtResizeVolumeCommandWrapper extends CommandWrapper<Resi boolean vmIsRunning = isVmRunning(vmInstanceName, libvirtComputingResource); - /* when VM is offline, we use qemu-img directly to resize encrypted volumes. - If VM is online, the existing resize script will call virsh blockresize which works - with both encrypted and non-encrypted volumes. + /* when VM is offline, we use qemu-img (or rbd, for librbd-encrypted RBD) directly to resize + encrypted volumes. If VM is online, the existing resize script calls virsh blockresize, + which for an librbd-encrypted RBD disk lets qemu/librbd grow the encrypted payload and + notify the guest in one step (no passphrase needed, qemu already has the secret loaded). */ - if (!vmIsRunning && command.getPassphrase() != null && command.getPassphrase().length > 0 ) { + if (rbdEncrypted && !vmIsRunning) { + logger.debug("Invoking rbd to resize an offline, encrypted (librbd) RBD volume"); + resizeRbdEncryptedVolume(pool, vol, newSize, shrinkOk, command.getPassphrase()); + } else if (!vmIsRunning && command.getPassphrase() != null && command.getPassphrase().length > 0 ) { logger.debug("Invoking qemu-img to resize an offline, encrypted volume"); QemuObject.EncryptFormat encryptFormat = QemuObject.EncryptFormat.enumValue(command.getEncryptFormat()); resizeEncryptedQcowFile(vol, encryptFormat,newSize, command.getPassphrase(), libvirtComputingResource); @@ -213,6 +225,16 @@ public final class LibvirtResizeVolumeCommandWrapper extends CommandWrapper<Resi } } + private void resizeRbdEncryptedVolume(final KVMStoragePool pool, final KVMPhysicalDisk vol, long newSize, + boolean shrinkOk, byte[] passphrase) throws CloudRuntimeException { + try { + new RbdEncryption().resize(pool.getSourceHost(), pool.getSourcePort(), pool.getAuthUserName(), + pool.getAuthSecret(), pool.getSourceDir(), vol.getName(), newSize, shrinkOk, passphrase); + } finally { + Arrays.fill(passphrase, (byte) 0); + } + } + private Answer handleMultipathSCSIResize(ResizeVolumeCommand command, KVMStoragePool pool) { ((MultipathSCSIPool)pool).resize(command.getPath(), command.getInstanceName(), command.getNewSize()); return new ResizeVolumeAnswer(command, true, ""); diff --git a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java index fe297adb327..b8a01b84834 100644 --- a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java +++ b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java @@ -182,6 +182,12 @@ public class KVMStorageProcessor implements StorageProcessor { private static final String CEPH_AUTH_KEY = "key"; private static final String CEPH_CLIENT_MOUNT_TIMEOUT = "client_mount_timeout"; private static final String CEPH_DEFAULT_MOUNT_TIMEOUT = "30"; + + // libvirt < 10.1.0 has an object apply-order bug (fixed in 10.1.0) that breaks hot-plug of an encrypted + // blockdev: on attach the disk is opened before its LUKS secret object is defined, so the attach fails with + // "No secret with id '...-format-encryption-secret0'". Booting a VM from an encrypted disk is unaffected (the + // QEMU command line resolves all -object before -blockdev). See qemuBlockStorageSourceAttachApply() in libvirt. + private static final long MIN_LIBVIRT_VERSION_FOR_RBD_ENCRYPTED_HOTPLUG = 10001000L; // libvirt 10.1.0 /** * Time interval before rechecking virsh commands */ @@ -1795,6 +1801,20 @@ public class KVMStorageProcessor implements StorageProcessor { return DiskDef.DiskBus.VIRTIO; } + /** + * libvirt < 10.1.0 cannot hot-plug an encrypted rbd blockdev (the LUKS secret is applied after the disk is + * opened), so refuse the attach with a clear message rather than letting libvirt fail with an opaque + * "No secret with id ..." error. Only the RBD hot-plug path is affected; booting a VM from an encrypted RBD + * disk works on older libvirt, so this does not gate the boot/root path. + */ + protected void ensureLibvirtSupportsEncryptedRbdHotplug(StoragePoolType poolType) { + if (poolType == StoragePoolType.RBD + && resource.getHypervisorLibvirtVersion() < MIN_LIBVIRT_VERSION_FOR_RBD_ENCRYPTED_HOTPLUG) { + throw new CloudRuntimeException("Libvirt version 10.1.0 required to attach an encrypted RBD volume to a running VM, but version " + + resource.getHypervisorLibvirtVersion() + " detected. Booting a VM from an encrypted RBD disk is not affected."); + } + } + @Override public Answer attachVolume(final AttachCommand cmd) { final DiskTO disk = cmd.getDisk(); @@ -1807,8 +1827,13 @@ public class KVMStorageProcessor implements StorageProcessor { final Connect conn = LibvirtConnection.getConnectionByVmName(vmName); DiskDef.LibvirtDiskEncryptDetails encryptDetails = null; if (vol.requiresEncryption()) { + // Encrypted RBD is decrypted by librbd inside qemu; hot-plugging it needs a libvirt new enough to + // emit the LUKS secret before the rbd blockdev. Booting from an encrypted RBD disk is unaffected. + ensureLibvirtSupportsEncryptedRbdHotplug(primaryStore.getPoolType()); String secretUuid = resource.createLibvirtVolumeSecret(conn, vol.getPath(), vol.getPassphrase()); - encryptDetails = new DiskDef.LibvirtDiskEncryptDetails(secretUuid, QemuObject.EncryptFormat.enumValue(vol.getEncryptFormat())); + // RBD volumes are encrypted natively by librbd, so request the librbd encryption engine. + String encryptEngine = (primaryStore.getPoolType() == StoragePoolType.RBD) ? "librbd" : null; + encryptDetails = new DiskDef.LibvirtDiskEncryptDetails(secretUuid, QemuObject.EncryptFormat.enumValue(vol.getEncryptFormat()), encryptEngine); vol.clearPassphrase(); } diff --git a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java index 01e8fdcf2ca..8a1a7b5bbe2 100644 --- a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java +++ b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java @@ -34,7 +34,9 @@ import java.util.stream.Collectors; import com.cloud.agent.properties.AgentProperties; import com.cloud.agent.properties.AgentPropertiesFileHandler; import org.apache.cloudstack.api.ApiConstants; +import org.apache.cloudstack.utils.cryptsetup.CryptSetup; import org.apache.cloudstack.utils.cryptsetup.KeyFile; +import org.apache.cloudstack.utils.rbd.RbdEncryption; import org.apache.cloudstack.utils.qemu.QemuImageOptions; import org.apache.cloudstack.utils.qemu.QemuImg; import org.apache.cloudstack.utils.qemu.QemuImg.PhysicalDiskFormat; @@ -94,6 +96,9 @@ public class LibvirtStorageAdaptor implements StorageAdaptor { private static final int RBD_FEATURE_DEEP_FLATTEN = 32; public static final int RBD_FEATURES = RBD_FEATURE_LAYERING + RBD_FEATURE_EXCLUSIVE_LOCK + RBD_FEATURE_OBJECT_MAP + RBD_FEATURE_FAST_DIFF + RBD_FEATURE_DEEP_FLATTEN; private int rbdOrder = 0; /* Order 0 means 4MB blocks (the default) */ + /* Space reserved at the front of an encrypted RBD image for the LUKS2 header/keyslots so the + usable (decrypted) size still matches the requested volume size. */ + private static final long LUKS2_HEADER_RESERVE_BYTES = 16L << 20; // 16 MiB /* libvirt's VIR_STORAGE_VOL_DELETE_WITH_SNAPSHOTS, not exposed as a constant by libvirt-java */ private static final int VIR_STORAGE_VOL_DELETE_WITH_SNAPSHOTS = 2; @@ -989,8 +994,18 @@ public class LibvirtStorageAdaptor implements StorageAdaptor { Map<String, String> details = pool.getDetails(); String dataPool = (details == null) ? null : details.get(KVMPhysicalDisk.RBD_DEFAULT_DATA_POOL); - return (dataPool == null) ? createPhysicalDiskByLibVirt(name, pool, PhysicalDiskFormat.RAW, provisioningType, size) : - createPhysicalDiskByQemuImg(name, pool, PhysicalDiskFormat.RAW, provisioningType, size, passphrase); + // Create the raw RBD image first. For encrypted volumes we apply a native librbd LUKS header + // afterwards via `rbd encryption format` (engine='librbd'). We deliberately do NOT hand the + // passphrase to qemu-img, which would instead produce a qemu-native LUKS container. + KVMPhysicalDisk disk = (dataPool == null) ? + createPhysicalDiskByLibVirt(name, pool, PhysicalDiskFormat.RAW, provisioningType, size) : + createPhysicalDiskByQemuImg(name, pool, PhysicalDiskFormat.RAW, provisioningType, size, null); + + if (passphrase != null && passphrase.length > 0) { + formatRbdImageEncryption(pool, name, passphrase); + disk.setQemuEncryptFormat(QemuObject.EncryptFormat.LUKS2); + } + return disk; } else if (QEMU_IMG_MANAGED_POOL_TYPES.contains(poolType)) { switch (format) { case QCOW2: @@ -1190,7 +1205,7 @@ public class LibvirtStorageAdaptor implements StorageAdaptor { KVMPhysicalDisk disk = null; if (destPool.getType() == StoragePoolType.RBD) { - disk = createDiskFromTemplateOnRBD(template, name, format, provisioningType, size, destPool, timeout); + disk = createDiskFromTemplateOnRBD(template, name, format, provisioningType, size, destPool, timeout, passphrase); } else { try (KeyFile keyFile = new KeyFile(passphrase)){ String newUuid = name; @@ -1270,7 +1285,7 @@ public class LibvirtStorageAdaptor implements StorageAdaptor { } private KVMPhysicalDisk createDiskFromTemplateOnRBD(KVMPhysicalDisk template, - String name, PhysicalDiskFormat format, Storage.ProvisioningType provisioningType, long size, KVMStoragePool destPool, int timeout){ + String name, PhysicalDiskFormat format, Storage.ProvisioningType provisioningType, long size, KVMStoragePool destPool, int timeout, byte[] passphrase){ /* With RBD you can't run qemu-img convert with an existing RBD image as destination @@ -1299,6 +1314,16 @@ public class LibvirtStorageAdaptor implements StorageAdaptor { } + if (passphrase != null && passphrase.length > 0) { + boolean sameClusterRbd = srcPool.getType() == StoragePoolType.RBD + && srcPool.getSourceHost().equals(destPool.getSourceHost()) + && srcPool.getSourceDir().equals(destPool.getSourceDir()); + if (sameClusterRbd) { + return createEncryptedRootCoWClone(template, destPool, newUuid, disk, passphrase); + } + return createEncryptedRootFullCopy(srcPool, template, destPool, newUuid, disk, passphrase); + } + QemuImgFile srcFile; QemuImgFile destFile = new QemuImgFile(KVMPhysicalDisk.RBDStringBuilder(destPool, disk.getPath())); destFile.setFormat(format); @@ -1449,9 +1474,126 @@ public class LibvirtStorageAdaptor implements StorageAdaptor { disk = null; } } + + // Encrypted volumes are handled by the early return above (create empty -> luks2 format -> + // import template through encryption); the clone/convert path here is for plaintext volumes. + return disk; + } + + /** + * Option A (thin CoW encrypted root), used when the template already lives on the same RBD cluster + * as the destination pool. Per the Ceph "Image Encryption" clone recipe: grow the template base to + * reserve LUKS2-header space, snapshot+protect that grown state, clone from it, apply a LUKS2 header, + * then resize the clone to the requested size. The inherited (plaintext) template data stays readable + * through the clone's encryption, and the clone is a thin CoW image (only the header is written). + * + * @return the encrypted CoW clone, or {@code null} if the Ceph operations failed + */ + private KVMPhysicalDisk createEncryptedRootCoWClone(KVMPhysicalDisk template, KVMStoragePool destPool, + String newUuid, KVMPhysicalDisk disk, byte[] passphrase) { + String luksReservedSnapshotName = rbdTemplateSnapName + "-luks"; + Rados radosConnection = null; + IoCTX ioContext = null; + Rbd rbdClient = null; + RbdImage templateImage = null; + try { + radosConnection = new Rados(destPool.getAuthUserName()); + radosConnection.confSet("mon_host", destPool.getSourceHost() + ":" + destPool.getSourcePort()); + radosConnection.confSet("key", destPool.getAuthSecret()); + radosConnection.confSet("client_mount_timeout", "30"); + radosConnection.connect(); + ioContext = radosConnection.ioCtxCreate(destPool.getSourceDir()); + rbdClient = new Rbd(ioContext); + templateImage = rbdClient.open(template.getName()); + boolean luksSnapshotExists = false; + for (RbdSnapInfo snapshotInfo : templateImage.snapList()) { + if (luksReservedSnapshotName.equals(snapshotInfo.name)) { + luksSnapshotExists = true; + break; + } + } + if (!luksSnapshotExists) { + templateImage.resize(template.getVirtualSize() + LUKS2_HEADER_RESERVE_BYTES); + templateImage.snapCreate(luksReservedSnapshotName); + templateImage.snapProtect(luksReservedSnapshotName); + logger.debug("Prepared LUKS-reserved template snapshot {}@{}", template.getName(), luksReservedSnapshotName); + } + rbdClient.clone(template.getName(), luksReservedSnapshotName, ioContext, newUuid, RBD_FEATURES, rbdOrder); + } catch (RadosException | RbdException e) { + logger.error("Failed to create encrypted CoW clone {}: {}", newUuid, e.getMessage()); + return null; + } finally { + if (rbdClient != null && templateImage != null) { + try { + rbdClient.close(templateImage); + } catch (RbdException ignored) { + // best-effort close of the template handle + } + } + if (radosConnection != null && ioContext != null) { + radosConnection.ioCtxDestroy(ioContext); + } + } + formatRbdImageEncryption(destPool, newUuid, passphrase); + if (disk.getVirtualSize() > template.getVirtualSize()) { + // grow the clone to the requested root size (encryption-aware) + new RbdEncryption().resize(destPool.getSourceHost(), destPool.getSourcePort(), + destPool.getAuthUserName(), destPool.getAuthSecret(), destPool.getSourceDir(), + newUuid, disk.getVirtualSize(), false, passphrase); + } + disk.setQemuEncryptFormat(QemuObject.EncryptFormat.LUKS2); return disk; } + /** + * Option B (full-copy encrypted root), used when the template is not on the same RBD cluster (e.g. first + * use from secondary storage). Create an empty image, apply a LUKS2 header, then import the template + * THROUGH the encryption layer (qemu-img convert -n). Correct but not thin (no CoW). + * + * @return the encrypted image, or {@code null} if the Ceph operations failed + */ + private KVMPhysicalDisk createEncryptedRootFullCopy(KVMStoragePool srcPool, KVMPhysicalDisk template, + KVMStoragePool destPool, String newUuid, KVMPhysicalDisk disk, byte[] passphrase) { + long imageSizeWithLuksHeader = disk.getVirtualSize() + LUKS2_HEADER_RESERVE_BYTES; + Rados radosConnection = null; + IoCTX ioContext = null; + try { + radosConnection = new Rados(destPool.getAuthUserName()); + radosConnection.confSet("mon_host", destPool.getSourceHost() + ":" + destPool.getSourcePort()); + radosConnection.confSet("key", destPool.getAuthSecret()); + radosConnection.confSet("client_mount_timeout", "30"); + radosConnection.connect(); + ioContext = radosConnection.ioCtxCreate(destPool.getSourceDir()); + Rbd rbdClient = new Rbd(ioContext); + rbdClient.create(newUuid, imageSizeWithLuksHeader, RBD_FEATURES, rbdOrder); + } catch (RadosException | RbdException e) { + logger.error("Failed to create encrypted RBD image {}: {}", newUuid, e.getMessage()); + return null; + } finally { + if (radosConnection != null && ioContext != null) { + radosConnection.ioCtxDestroy(ioContext); + } + } + formatRbdImageEncryption(destPool, newUuid, passphrase); + boolean sourceIsRbdPool = srcPool.getType() == StoragePoolType.RBD; + new RbdEncryption().importTemplate( + sourceIsRbdPool ? srcPool.getSourceDir() : null, sourceIsRbdPool ? template.getName() : null, + sourceIsRbdPool ? null : template.getPath(), sourceIsRbdPool ? null : template.getFormat().toString(), + destPool.getSourceHost(), destPool.getSourcePort(), destPool.getAuthUserName(), destPool.getAuthSecret(), + destPool.getSourceDir(), newUuid, passphrase, CryptSetup.LuksType.LUKS2); + disk.setQemuEncryptFormat(QemuObject.EncryptFormat.LUKS2); + return disk; + } + + /** + * Apply native librbd LUKS encryption to an existing RBD image via the rbd CLI. + * Isolated here so the CLI dependency can later be swapped for a native (JNA) librbd binding. + */ + private void formatRbdImageEncryption(KVMStoragePool pool, String image, byte[] passphrase) { + new RbdEncryption().format(pool.getSourceHost(), pool.getSourcePort(), pool.getAuthUserName(), + pool.getAuthSecret(), pool.getSourceDir(), image, passphrase, CryptSetup.LuksType.LUKS2); + } + @Override public KVMPhysicalDisk createTemplateFromDisk(KVMPhysicalDisk disk, String name, PhysicalDiskFormat format, long size, KVMStoragePool destPool) { return null; diff --git a/plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/utils/qemu/QemuImageOptions.java b/plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/utils/qemu/QemuImageOptions.java index 4a577ef3400..10e39cb5ffc 100644 --- a/plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/utils/qemu/QemuImageOptions.java +++ b/plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/utils/qemu/QemuImageOptions.java @@ -86,8 +86,17 @@ public class QemuImageOptions { return new String[] { params.get(FILENAME_PARAM_KEY) }; } } + return toCommandFlag(QemuImg.IMAGE_OPTS_FLAG); + } + + /** + * Converts QemuImageOptions into the command strings under the given qemu-img flag, + * e.g. {@link QemuImg#TARGET_IMAGE_OPTS_FLAG} for a convert destination. + * @return array of strings representing the flag and its options value + */ + public String[] toCommandFlag(String flagName) { Map<String, String> sorted = new TreeMap<>(params); String paramString = Joiner.on(",").withKeyValueSeparator("=").join(sorted); - return new String[] {"--image-opts", paramString}; + return new String[] {flagName, paramString}; } } diff --git a/plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/utils/qemu/QemuImg.java b/plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/utils/qemu/QemuImg.java index cae6832999e..49f531ed7c1 100644 --- a/plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/utils/qemu/QemuImg.java +++ b/plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/utils/qemu/QemuImg.java @@ -52,6 +52,8 @@ public class QemuImg { public static final String ENCRYPT_FORMAT = "encrypt.format"; public static final String ENCRYPT_KEY_SECRET = "encrypt.key-secret"; public static final String TARGET_ZERO_FLAG = "--target-is-zero"; + public static final String IMAGE_OPTS_FLAG = "--image-opts"; + public static final String TARGET_IMAGE_OPTS_FLAG = "--target-image-opts"; public static final String PREALLOCATION = "preallocation"; public static final long QEMU_2_10 = 2010000; public static final long QEMU_5_1 = 5001000; @@ -402,6 +404,21 @@ public class QemuImg { convert(srcFile, destFile, null, options, qemuObjects, srcImageOpts, snapshotName, forceSourceFormat, false, false, false, null, null); } + /** + * Converts an image into an existing destination that is described by explicit image options + * ({@code --target-image-opts}) instead of a plain filename - for example an RBD image written + * through librbd encryption ({@code driver=rbd,...,encrypt.format=...}). The destination is + * never created ({@code -n} is implied) and must already exist with the wanted size and format. + * + * @param destImageOpts + * image options describing the existing destination; passed as --target-image-opts. + */ + public void convertIntoExistingTarget(final QemuImgFile srcFile, final Map<String, String> options, + final List<QemuObject> qemuObjects, final QemuImageOptions srcImageOpts, final QemuImageOptions destImageOpts, + final boolean forceSourceFormat) throws QemuImgException { + convert(srcFile, null, null, options, qemuObjects, srcImageOpts, destImageOpts, null, forceSourceFormat, false, false, false, null, null); + } + protected Map<String, String> getResizeOptionsFromConvertOptions(final Map<String, String> options) { if (MapUtils.isEmpty(options)) { return null; @@ -450,6 +467,25 @@ public class QemuImg { public void convert(final QemuImgFile srcFile, final QemuImgFile destFile, QemuImgFile backingFile, final Map<String, String> options, final List<QemuObject> qemuObjects, final QemuImageOptions srcImageOpts, final String snapshotName, final boolean forceSourceFormat, boolean keepBitmaps, boolean outOfOrderWrites, boolean compress, Integer coroutines, Integer rateLimit) throws QemuImgException { + convert(srcFile, destFile, backingFile, options, qemuObjects, srcImageOpts, null, snapshotName, forceSourceFormat, keepBitmaps, outOfOrderWrites, compress, coroutines, + rateLimit); + } + + /** + * Converts an image from source to destination, optionally into an existing destination described by explicit image options + * ({@code --target-image-opts}) instead of a plain filename; see {@link #convertIntoExistingTarget}. All other parameters + * behave as documented above. + * + * @param destImageOpts + * If not null, the destination is described by these image options and {@code destFile} is unused. + */ + public void convert(final QemuImgFile srcFile, final QemuImgFile destFile, QemuImgFile backingFile, final Map<String, String> options, final List<QemuObject> qemuObjects, + final QemuImageOptions srcImageOpts, final QemuImageOptions destImageOpts, final String snapshotName, final boolean forceSourceFormat, boolean keepBitmaps, + boolean outOfOrderWrites, boolean compress, Integer coroutines, Integer rateLimit) throws QemuImgException { + if (destImageOpts != null && this.version < QEMU_2_10) { + throw new QemuImgException(String.format("qemu >= 2.10 is required to convert into a destination described by %s", TARGET_IMAGE_OPTS_FLAG)); + } + Script script = new Script(_qemuImgPath, timeout); if (StringUtils.isNotBlank(snapshotName)) { String qemuPath = Script.runSimpleBashScript(getQemuImgPathScript); @@ -458,7 +494,10 @@ public class QemuImg { script.add("convert"); - if (skipZero && Files.exists(Paths.get(destFile.getFileName()))) { + if (destImageOpts != null) { + // a destination described by image options always exists already; qemu-img requires -n with --target-image-opts + script.add("-n"); + } else if (skipZero && Files.exists(Paths.get(destFile.getFileName()))) { script.add("-n"); script.add(TARGET_ZERO_FLAG); script.add("-W"); @@ -469,8 +508,10 @@ public class QemuImg { script.add("-n"); } - script.add("-O"); - script.add(destFile.getFormat().toString()); + if (destImageOpts == null) { + script.add("-O"); + script.add(destFile.getFormat().toString()); + } addBackingFileToConvertCommand(script, backingFile); addScriptOptionsFromMap(options, script); @@ -524,14 +565,19 @@ public class QemuImg { script.add("--bitmaps"); } - script.add(destFile.getFileName()); + if (destImageOpts != null) { + script.add(destImageOpts.toCommandFlag(TARGET_IMAGE_OPTS_FLAG)); + } else { + script.add(destFile.getFileName()); + } final String result = script.execute(); if (result != null) { throw new QemuImgException(result); } - if (srcFile.getSize() < destFile.getSize()) { + // an image-options destination already exists with its final size; 'qemu-img resize' cannot address it by filename + if (destImageOpts == null && srcFile.getSize() < destFile.getSize()) { this.resize(destFile, destFile.getSize(), getResizeOptionsFromConvertOptions(options)); } } diff --git a/plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/utils/qemu/QemuObject.java b/plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/utils/qemu/QemuObject.java index efeee04cb90..511e9107496 100644 --- a/plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/utils/qemu/QemuObject.java +++ b/plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/utils/qemu/QemuObject.java @@ -54,6 +54,7 @@ public class QemuObject { */ public enum EncryptFormat { LUKS("luks"), + LUKS2("luks2"), AES("aes"); private final String format; diff --git a/plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/utils/rbd/RbdEncryption.java b/plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/utils/rbd/RbdEncryption.java new file mode 100644 index 00000000000..a23ce93d2b8 --- /dev/null +++ b/plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/utils/rbd/RbdEncryption.java @@ -0,0 +1,294 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. +package org.apache.cloudstack.utils.rbd; + +import com.cloud.utils.exception.CloudRuntimeException; +import com.cloud.utils.script.Script; +import org.apache.cloudstack.utils.cryptsetup.CryptSetup; +import org.apache.cloudstack.utils.cryptsetup.KeyFile; +import org.apache.cloudstack.utils.qemu.QemuImageOptions; +import org.apache.cloudstack.utils.qemu.QemuImg; +import org.apache.cloudstack.utils.qemu.QemuImgException; +import org.apache.cloudstack.utils.qemu.QemuImgFile; +import org.apache.cloudstack.utils.qemu.QemuObject; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; +import org.libvirt.LibvirtException; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.attribute.FileAttribute; +import java.nio.file.attribute.PosixFilePermissions; +import java.util.EnumMap; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * Thin wrapper around the {@code rbd} CLI to apply native librbd LUKS encryption to an + * RBD image via {@code rbd encryption format}. This is only used at volume create time; + * runtime decryption is handled by libvirt/qemu through {@code <encryption engine='librbd'>}. + * + * The CLI dependency is intentionally isolated in this class so it can later be replaced + * by a native librbd (JNA) binding without touching callers. rados-java (0.x) does not + * expose the rbd_encryption_format API, hence the CLI for now. + * + * The command builders ({@code build*Script}) are separated from execution so the generated + * argv can be unit-tested without a live Ceph cluster (see {@code RbdEncryptionTest}). + */ +public class RbdEncryption { + protected Logger logger = LogManager.getLogger(getClass()); + + protected String commandPath = "rbd"; + + /** qemu secret id used to hand the LUKS passphrase to qemu-img during template import */ + protected static final String LUKS_SECRET_ID = "luks0"; + + public RbdEncryption() {} + + public RbdEncryption(String commandPath) { + this.commandPath = commandPath; + } + + private static String monSpec(String monHost, int monPort) { + return monPort > 0 ? monHost + ":" + monPort : monHost; + } + + /** + * Apply a LUKS header to an existing RBD image so librbd can transparently encrypt it. + * <p> + * cephx authentication is supplied via {@code --id} plus a temporary keyfile so the secret + * never appears on the command line. The LUKS passphrase is supplied via a temporary file + * ({@link KeyFile}). Both temp files are deleted when this method returns. + * + * @param monHost ceph monitor host + * @param monPort ceph monitor port (0 to omit) + * @param authUser cephx user (e.g. "cloudstack"); null to skip --id + * @param authSecret cephx secret/key (as used for ceph "key" config); null to skip --keyfile + * @param cephPool ceph pool name + * @param image rbd image name + * @param passphrase LUKS passphrase + * @param luksType LUKS1/LUKS2 (librbd engine supports both; LUKS2 recommended) + */ + public void format(String monHost, int monPort, String authUser, String authSecret, + String cephPool, String image, byte[] passphrase, CryptSetup.LuksType luksType) { + final String imageSpec = cephPool + "/" + image; + if (passphrase == null || passphrase.length == 0) { + throw new CloudRuntimeException("Cannot LUKS-format RBD image " + imageSpec + ": empty passphrase"); + } + try (KeyFile passFile = new KeyFile(passphrase); + KeyFile cephKeyFile = new KeyFile(authSecret == null ? null : authSecret.getBytes(StandardCharsets.UTF_8))) { + final Script script = buildFormatScript(imageSpec, luksType, passFile.toString(), + monSpec(monHost, monPort), authUser, cephKeyFile.isSet() ? cephKeyFile.toString() : null); + final String result = script.execute(); + if (result != null) { + throw new CloudRuntimeException(String.format("Failed to apply librbd %s encryption to %s: %s", luksType, imageSpec, result)); + } + logger.debug("Applied {} encryption to RBD image {}", luksType, imageSpec); + } catch (IOException ex) { + throw new CloudRuntimeException(String.format("Failed to apply librbd %s encryption to %s", luksType, imageSpec), ex); + } + } + + protected Script buildFormatScript(String imageSpec, CryptSetup.LuksType luksType, String passFilePath, + String monSpec, String authUser, String cephKeyFilePath) { + final Script script = new Script(commandPath); + script.add("encryption"); + script.add("format"); + script.add(imageSpec); + script.add(luksType.toString()); + script.add(passFilePath); + script.add("--mon-host"); + script.add(monSpec); + if (authUser != null) { + script.add("--id"); + script.add(authUser); + } + if (cephKeyFilePath != null) { + script.add("--keyfile"); + script.add(cephKeyFilePath); + } + return script; + } + + /** + * Resize an encrypted RBD image. librbd needs the passphrase so it can resize the encrypted + * payload (not just the raw image) and keep the LUKS header consistent. {@code newSizeBytes} is + * the usable (decrypted) size requested; rbd {@code --size} is expressed in MiB. + * + * @param allowShrink pass --allow-shrink when shrinking is permitted + */ + public void resize(String monHost, int monPort, String authUser, String authSecret, + String cephPool, String image, long newSizeBytes, boolean allowShrink, byte[] passphrase) { + final String imageSpec = cephPool + "/" + image; + if (passphrase == null || passphrase.length == 0) { + throw new CloudRuntimeException("Cannot resize encrypted RBD image " + imageSpec + ": empty passphrase"); + } + // rbd --size is in MiB; round up so a non-MiB-aligned request never shrinks the volume below what was asked for. + final long sizeMiB = (newSizeBytes + (1024L * 1024L) - 1) / (1024L * 1024L); + try (KeyFile passFile = new KeyFile(passphrase); + KeyFile cephKeyFile = new KeyFile(authSecret == null ? null : authSecret.getBytes(StandardCharsets.UTF_8))) { + final Script script = buildResizeScript(imageSpec, sizeMiB, passFile.toString(), allowShrink, + monSpec(monHost, monPort), authUser, cephKeyFile.isSet() ? cephKeyFile.toString() : null); + final String result = script.execute(); + if (result != null) { + throw new CloudRuntimeException(String.format("Failed to resize encrypted RBD image %s to %d MiB: %s", imageSpec, sizeMiB, result)); + } + logger.debug("Resized encrypted RBD image {} to {} MiB", imageSpec, sizeMiB); + } catch (IOException ex) { + throw new CloudRuntimeException(String.format("Failed to resize encrypted RBD image %s", imageSpec), ex); + } + } + + protected Script buildResizeScript(String imageSpec, long sizeMiB, String passFilePath, boolean allowShrink, + String monSpec, String authUser, String cephKeyFilePath) { + final Script script = new Script(commandPath); + script.add("resize"); + script.add("--size"); + script.add(String.valueOf(sizeMiB)); + script.add(imageSpec); + script.add("--encryption-passphrase-file"); + script.add(passFilePath); + if (allowShrink) { + script.add("--allow-shrink"); + } + script.add("--mon-host"); + script.add(monSpec); + if (authUser != null) { + script.add("--id"); + script.add(authUser); + } + if (cephKeyFilePath != null) { + script.add("--keyfile"); + script.add(cephKeyFilePath); + } + return script; + } + + /** + * Import a template into an already-created, already-LUKS-formatted RBD image by writing it + * THROUGH the librbd encryption layer with {@code qemu-img convert -n} (so the data lands + * encrypted). This is how encrypted root disks are populated: we never clone-then-format a + * plaintext template (that leaves the inherited OS data unreadable) — instead we format an + * empty image and convert the template into it. + * + * Exactly one source must be given: an RBD image ({@code srcRbdPool}+{@code srcRbdImage}) or a + * local file ({@code srcFilePath}[+{@code srcFileFormat}]). cephx auth is provided to qemu-img + * via a temporary ceph.conf + keyring (deleted on return). The conversion itself goes through + * {@link QemuImg#convertIntoExistingTarget}. + */ + public void importTemplate(String srcRbdPool, String srcRbdImage, + String srcFilePath, String srcFileFormat, + String monHost, int monPort, String authUser, String authSecret, + String cephPool, String destImage, byte[] passphrase, CryptSetup.LuksType luksType) { + final String imageSpec = cephPool + "/" + destImage; + if (passphrase == null || passphrase.length == 0) { + throw new CloudRuntimeException("Cannot import template into encrypted RBD image " + imageSpec + ": empty passphrase"); + } + Path conf = null; + Path keyring = null; + try (KeyFile passFile = new KeyFile(passphrase)) { + // These temp files hold the cephx secret; create them 0600 up front (matching KeyFile) rather than relying on the umask. + final FileAttribute<?> ownerOnly = PosixFilePermissions.asFileAttribute(PosixFilePermissions.fromString("rw-------")); + keyring = Files.createTempFile("cs-ceph-", ".keyring", ownerOnly); + Files.writeString(keyring, "[client." + authUser + "]\n\tkey = " + authSecret + "\n"); + conf = Files.createTempFile("cs-ceph-", ".conf", ownerOnly); + Files.writeString(conf, "[global]\nmon_host = " + monSpec(monHost, monPort) + "\nkeyring = " + keyring + "\n"); + + QemuImgFile srcQemuFile; + QemuImageOptions srcImageOpts; + boolean forceSourceFormat = false; + if (srcRbdImage != null) { + srcQemuFile = new QemuImgFile(srcRbdPool + "/" + srcRbdImage, QemuImg.PhysicalDiskFormat.RAW); + srcImageOpts = new QemuImageOptions(rbdImageOptions(srcRbdPool, srcRbdImage, conf.toString(), authUser)); + srcImageOpts.setImageOptsFlag(true); + } else { + QemuImg.PhysicalDiskFormat srcFormat = srcFileFormat != null ? QemuImg.PhysicalDiskFormat.valueOf(srcFileFormat.toUpperCase()) : null; + srcQemuFile = srcFormat != null ? new QemuImgFile(srcFilePath, srcFormat) : new QemuImgFile(srcFilePath); + srcImageOpts = new QemuImageOptions(srcFilePath); + if (srcFormat != null) { + // emit the source as --image-opts driver=<format>,file.filename=<path> (the -f equivalent) + srcImageOpts.setImageOptsFlag(true); + forceSourceFormat = true; + } + } + + Map<String, String> destParams = rbdImageOptions(cephPool, destImage, conf.toString(), authUser); + destParams.put("encrypt.format", luksType.toString()); + destParams.put("encrypt.key-secret", LUKS_SECRET_ID); + QemuImageOptions destImageOpts = new QemuImageOptions(destParams); + + EnumMap<QemuObject.ObjectParameter, String> secretParams = new EnumMap<>(QemuObject.ObjectParameter.class); + secretParams.put(QemuObject.ObjectParameter.ID, LUKS_SECRET_ID); + secretParams.put(QemuObject.ObjectParameter.FILE, passFile.toString()); + QemuObject luksSecret = new QemuObject(QemuObject.ObjectType.SECRET, secretParams); + + QemuImg qemu = createQemuImg(); + qemu.convertIntoExistingTarget(srcQemuFile, null, List.of(luksSecret), srcImageOpts, destImageOpts, forceSourceFormat); + logger.debug("Imported template into encrypted RBD image {}", imageSpec); + } catch (IOException | QemuImgException | LibvirtException ex) { + throw new CloudRuntimeException(String.format("Failed to import template into encrypted RBD image %s", imageSpec), ex); + } finally { + deleteQuietly(conf); + deleteQuietly(keyring); + } + } + + /** + * Seam for unit tests; {@link QemuImg} probes the qemu version through libvirt on construction. + */ + protected QemuImg createQemuImg() throws QemuImgException, LibvirtException { + return new QemuImg(0); + } + + /** qemu image options addressing an RBD image (as used with --image-opts / --target-image-opts). */ + private static Map<String, String> rbdImageOptions(String cephPool, String image, String confPath, String authUser) { + Map<String, String> opts = new HashMap<>(); + opts.put("driver", "rbd"); + opts.put("pool", cephPool); + opts.put("image", image); + opts.put("conf", confPath); + if (authUser != null) { + opts.put("user", authUser); + } + return opts; + } + + private static void deleteQuietly(Path p) { + if (p == null) { + return; + } + try { + Files.deleteIfExists(p); + } catch (IOException ignored) { + // best-effort cleanup of the temporary ceph auth files + } + } + + /** + * Best-effort probe that the local rbd CLI supports the encryption subcommand. + */ + public boolean isSupported() { + final Script script = new Script(commandPath); + script.add("help"); + script.add("encryption"); + script.add("format"); + return script.execute() == null; + } +} diff --git a/plugins/hypervisors/kvm/src/test/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessorTest.java b/plugins/hypervisors/kvm/src/test/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessorTest.java index daa8792ed8f..4a2bf5cf332 100644 --- a/plugins/hypervisors/kvm/src/test/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessorTest.java +++ b/plugins/hypervisors/kvm/src/test/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessorTest.java @@ -26,6 +26,7 @@ import com.ceph.rbd.RbdImage; import com.cloud.exception.InternalErrorException; import com.cloud.hypervisor.kvm.resource.LibvirtComputingResource; import com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser; +import com.cloud.storage.Storage.StoragePoolType; import com.cloud.hypervisor.kvm.resource.LibvirtVMDef; import com.cloud.storage.Storage; import com.cloud.storage.template.TemplateConstants; @@ -743,4 +744,25 @@ public class KVMStorageProcessorTest { Mockito.verify(radosMock).ioCtxDestroy(ioCtxMock); } } + + @Test + public void ensureLibvirtSupportsEncryptedRbdHotplugRejectsOldLibvirt() { + Mockito.when(resource.getHypervisorLibvirtVersion()).thenReturn(10000000L); // libvirt 10.0.0 + CloudRuntimeException ex = Assert.assertThrows(CloudRuntimeException.class, + () -> storageProcessor.ensureLibvirtSupportsEncryptedRbdHotplug(StoragePoolType.RBD)); + Assert.assertTrue(ex.getMessage(), ex.getMessage().contains("Libvirt version 10.1.0 required")); + } + + @Test + public void ensureLibvirtSupportsEncryptedRbdHotplugAllowsNewLibvirt() { + Mockito.when(resource.getHypervisorLibvirtVersion()).thenReturn(10001000L); // libvirt 10.1.0 + storageProcessor.ensureLibvirtSupportsEncryptedRbdHotplug(StoragePoolType.RBD); // must not throw + } + + @Test + public void ensureLibvirtSupportsEncryptedRbdHotplugIgnoresNonRbd() { + // The libvirt hot-plug apply-order bug is RBD-specific; other pool types are not gated, and the libvirt + // version is not even consulted for them. + storageProcessor.ensureLibvirtSupportsEncryptedRbdHotplug(StoragePoolType.NetworkFilesystem); + } } diff --git a/plugins/hypervisors/kvm/src/test/java/org/apache/cloudstack/utils/rbd/RbdEncryptionTest.java b/plugins/hypervisors/kvm/src/test/java/org/apache/cloudstack/utils/rbd/RbdEncryptionTest.java new file mode 100644 index 00000000000..e4c977c676f --- /dev/null +++ b/plugins/hypervisors/kvm/src/test/java/org/apache/cloudstack/utils/rbd/RbdEncryptionTest.java @@ -0,0 +1,166 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.cloudstack.utils.rbd; + +import com.cloud.utils.exception.CloudRuntimeException; +import com.cloud.utils.script.Script; +import org.apache.cloudstack.utils.cryptsetup.CryptSetup; +import org.apache.cloudstack.utils.qemu.QemuImageOptions; +import org.apache.cloudstack.utils.qemu.QemuImg; +import org.apache.cloudstack.utils.qemu.QemuImgFile; +import org.apache.cloudstack.utils.qemu.QemuObject; +import org.junit.Assert; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.ArgumentCaptor; +import org.mockito.Mockito; +import org.mockito.junit.MockitoJUnitRunner; + +import java.nio.charset.StandardCharsets; +import java.util.List; +import java.util.stream.Collectors; + +/** + * Unit tests for {@link RbdEncryption}. These assert the {@code rbd} argv that would be handed to + * {@link Script} and the image options handed to {@link QemuImg}, so the command construction is + * verified without a live Ceph cluster (the actual execution needs a real cluster and is covered + * by end-to-end testing). + */ +@RunWith(MockitoJUnitRunner.class) +public class RbdEncryptionTest { + + private final RbdEncryption rbdEncryption = new RbdEncryption(); + + @Test + public void buildFormatScriptWithCephxAuth() { + Script script = rbdEncryption.buildFormatScript("cloudstack/img", CryptSetup.LuksType.LUKS2, + "/tmp/pass", "1.2.3.4:6789", "cloudstack", "/tmp/key"); + String cmd = script.toString(); + Assert.assertTrue(cmd, cmd.contains("rbd encryption format cloudstack/img luks2 /tmp/pass")); + Assert.assertTrue(cmd, cmd.contains("--mon-host 1.2.3.4:6789")); + Assert.assertTrue(cmd, cmd.contains("--id cloudstack")); + Assert.assertTrue(cmd, cmd.contains("--keyfile /tmp/key")); + } + + @Test + public void buildFormatScriptWithoutCephxAuth() { + // authUser == null and cephKeyFilePath == null (e.g. auth-less cluster): no --id / --keyfile. + Script script = rbdEncryption.buildFormatScript("pool/vol", CryptSetup.LuksType.LUKS2, + "/tmp/pass", "mon:6789", null, null); + String cmd = script.toString(); + Assert.assertTrue(cmd, cmd.contains("rbd encryption format pool/vol luks2 /tmp/pass --mon-host mon:6789")); + Assert.assertFalse(cmd, cmd.contains("--id")); + Assert.assertFalse(cmd, cmd.contains("--keyfile")); + } + + @Test + public void buildResizeScriptGrowDoesNotAllowShrink() { + Script script = rbdEncryption.buildResizeScript("cloudstack/img", 10240L, "/tmp/pass", false, + "1.2.3.4:6789", "cloudstack", "/tmp/key"); + String cmd = script.toString(); + Assert.assertTrue(cmd, cmd.contains("rbd resize --size 10240 cloudstack/img --encryption-passphrase-file /tmp/pass")); + Assert.assertTrue(cmd, cmd.contains("--id cloudstack")); + Assert.assertFalse(cmd, cmd.contains("--allow-shrink")); + } + + @Test + public void buildResizeScriptShrinkPassesAllowShrink() { + Script script = rbdEncryption.buildResizeScript("cloudstack/img", 5120L, "/tmp/pass", true, + "1.2.3.4:6789", "cloudstack", "/tmp/key"); + Assert.assertTrue(script.toString(), script.toString().contains("--allow-shrink")); + } + + @Test + public void importTemplateFromRbdSourceConvertsThroughQemuImg() throws Exception { + RbdEncryption spy = Mockito.spy(new RbdEncryption()); + QemuImg qemuImg = Mockito.mock(QemuImg.class); + Mockito.doReturn(qemuImg).when(spy).createQemuImg(); + + spy.importTemplate("srcpool", "srcimg", null, null, "1.2.3.4", 6789, "cloudstack", "secret", + "cloudstack", "dst", "passphrase".getBytes(StandardCharsets.UTF_8), CryptSetup.LuksType.LUKS2); + + ArgumentCaptor<QemuImageOptions> srcOpts = ArgumentCaptor.forClass(QemuImageOptions.class); + ArgumentCaptor<QemuImageOptions> destOpts = ArgumentCaptor.forClass(QemuImageOptions.class); + ArgumentCaptor<List<QemuObject>> objects = ArgumentCaptor.forClass(List.class); + Mockito.verify(qemuImg).convertIntoExistingTarget(Mockito.any(QemuImgFile.class), Mockito.isNull(), + objects.capture(), srcOpts.capture(), destOpts.capture(), Mockito.eq(false)); + + String src = String.join(" ", srcOpts.getValue().toCommandFlag()); + Assert.assertTrue(src, src.startsWith("--image-opts ")); + Assert.assertTrue(src, src.contains("driver=rbd")); + Assert.assertTrue(src, src.contains("pool=srcpool")); + Assert.assertTrue(src, src.contains("image=srcimg")); + Assert.assertTrue(src, src.contains("user=cloudstack")); + Assert.assertTrue(src, src.contains("conf=")); + + String dest = String.join(" ", destOpts.getValue().toCommandFlag(QemuImg.TARGET_IMAGE_OPTS_FLAG)); + Assert.assertTrue(dest, dest.startsWith(QemuImg.TARGET_IMAGE_OPTS_FLAG + " ")); + Assert.assertTrue(dest, dest.contains("driver=rbd")); + Assert.assertTrue(dest, dest.contains("pool=cloudstack")); + Assert.assertTrue(dest, dest.contains("image=dst")); + Assert.assertTrue(dest, dest.contains("encrypt.format=luks2")); + Assert.assertTrue(dest, dest.contains("encrypt.key-secret=luks0")); + + String secretObjects = objects.getValue().stream() + .map(o -> String.join(" ", o.toCommandFlag())).collect(Collectors.joining(" ")); + Assert.assertTrue(secretObjects, secretObjects.contains("--object secret,")); + Assert.assertTrue(secretObjects, secretObjects.contains("id=luks0")); + Assert.assertTrue(secretObjects, secretObjects.contains("file=")); + } + + @Test + public void importTemplateFromFileSourceForcesSourceFormat() throws Exception { + RbdEncryption spy = Mockito.spy(new RbdEncryption()); + QemuImg qemuImg = Mockito.mock(QemuImg.class); + Mockito.doReturn(qemuImg).when(spy).createQemuImg(); + + spy.importTemplate(null, null, "/tmp/tmpl.qcow2", "QCOW2", "1.2.3.4", 6789, "cloudstack", "secret", + "cloudstack", "dst", "passphrase".getBytes(StandardCharsets.UTF_8), CryptSetup.LuksType.LUKS2); + + ArgumentCaptor<QemuImgFile> srcFile = ArgumentCaptor.forClass(QemuImgFile.class); + ArgumentCaptor<QemuImageOptions> srcOpts = ArgumentCaptor.forClass(QemuImageOptions.class); + Mockito.verify(qemuImg).convertIntoExistingTarget(srcFile.capture(), Mockito.isNull(), + Mockito.anyList(), srcOpts.capture(), Mockito.any(QemuImageOptions.class), Mockito.eq(true)); + + Assert.assertEquals(QemuImg.PhysicalDiskFormat.QCOW2, srcFile.getValue().getFormat()); + String src = String.join(" ", srcOpts.getValue().toCommandFlag()); + Assert.assertTrue(src, src.contains("file.filename=/tmp/tmpl.qcow2")); + } + + @Test + public void formatRejectsEmptyPassphrase() { + Assert.assertThrows(CloudRuntimeException.class, () -> rbdEncryption.format( + "1.2.3.4", 6789, "cloudstack", "secret", "cloudstack", "img", + new byte[0], CryptSetup.LuksType.LUKS2)); + } + + @Test + public void resizeRejectsNullPassphrase() { + Assert.assertThrows(CloudRuntimeException.class, () -> rbdEncryption.resize( + "1.2.3.4", 6789, "cloudstack", "secret", "cloudstack", "img", + 1L << 30, false, null)); + } + + @Test + public void importTemplateRejectsEmptyPassphrase() { + Assert.assertThrows(CloudRuntimeException.class, () -> rbdEncryption.importTemplate( + "srcpool", "srcimg", null, null, "1.2.3.4", 6789, "cloudstack", "secret", + "cloudstack", "dst", "".getBytes(StandardCharsets.UTF_8), CryptSetup.LuksType.LUKS2)); + } +} diff --git a/server/src/main/java/com/cloud/deploy/DeploymentPlanningManagerImpl.java b/server/src/main/java/com/cloud/deploy/DeploymentPlanningManagerImpl.java index 32e904c8de3..4c062b1eaff 100644 --- a/server/src/main/java/com/cloud/deploy/DeploymentPlanningManagerImpl.java +++ b/server/src/main/java/com/cloud/deploy/DeploymentPlanningManagerImpl.java @@ -20,6 +20,7 @@ import static com.cloud.utils.NumbersUtil.toHumanReadableSize; import java.util.ArrayList; import java.util.Arrays; +import java.util.Collection; import java.util.Comparator; import java.util.HashMap; import java.util.HashSet; @@ -111,6 +112,7 @@ import com.cloud.service.dao.ServiceOfferingDetailsDao; import com.cloud.storage.DiskOfferingVO; import com.cloud.storage.GuestOSVO; import com.cloud.storage.ScopeType; +import com.cloud.storage.Storage; import com.cloud.storage.StorageManager; import com.cloud.storage.StoragePool; import com.cloud.storage.StoragePoolHostVO; @@ -428,7 +430,7 @@ StateListener<State, VirtualMachine.Event, VirtualMachine>, Configurable { long hostId = dest.getHost().getId(); avoids.addHost(dest.getHost().getId()); - if (volumesRequireEncryption && !Boolean.parseBoolean(_hostDetailsDao.findDetail(hostId, Host.HOST_VOLUME_ENCRYPTION).getValue())) { + if (volumesRequireEncryption && !hostMeetsVolumeEncryptionRequirements(hostId, _volsDao.findByInstance(vm.getId()), null)) { logger.warn("VM's volumes require encryption support, and the planner-provided host {} can't handle it", dest.getHost()); continue; } else { @@ -599,8 +601,8 @@ StateListener<State, VirtualMachine.Event, VirtualMachine>, Configurable { return false; } - if (volumesRequireEncryption && !Boolean.parseBoolean(host.getDetail(Host.HOST_VOLUME_ENCRYPTION))) { - logger.warn("The last host of this VM {} does not support volume encryption, which is required by this VM.", host); + if (volumesRequireEncryption && !hostMeetsVolumeEncryptionRequirements(host.getId(), _volsDao.findByInstance(vm.getId()), null)) { + logger.warn("The last host of this VM {} does not support the volume encryption required by this VM.", host); return false; } return true; @@ -689,6 +691,51 @@ StateListener<State, VirtualMachine.Event, VirtualMachine>, Configurable { return false; } + private boolean hostHasEncryptionDetail(long hostId, String detailName) { + DetailVO detail = _hostDetailsDao.findDetail(hostId, detailName); + return detail != null && Boolean.parseBoolean(detail.getValue()); + } + + /** + * Checks that the host advertises the encryption mechanism each encrypted volume needs: + * {@link Host#HOST_RBD_VOLUME_ENCRYPTION} (librbd) for volumes on RBD pools, + * {@link Host#HOST_VOLUME_ENCRYPTION} (qemu-native LUKS) for volumes on any other pool type. + * The pool of a volume is taken from {@code proposedPools} when present (pools being allocated + * along with the host), falling back to the volume's persisted pool. For an encrypted volume + * with no pool yet (initial deployment without a proposed pool), either mechanism is accepted; + * the storage pool allocator selects a pool the host can serve. + */ + protected boolean hostMeetsVolumeEncryptionRequirements(long hostId, Collection<? extends Volume> volumes, Map<Volume, StoragePool> proposedPools) { + for (Volume volume : volumes) { + if (volume.getPassphraseId() == null && volume.getKmsKeyId() == null) { + continue; + } + Storage.StoragePoolType poolType = null; + StoragePool proposedPool = proposedPools != null ? proposedPools.get(volume) : null; + if (proposedPool != null) { + poolType = proposedPool.getPoolType(); + } else if (volume.getPoolId() != null) { + StoragePoolVO pool = _storagePoolDao.findById(volume.getPoolId()); + poolType = pool != null ? pool.getPoolType() : null; + } + boolean hostMeets; + if (poolType == Storage.StoragePoolType.RBD) { + hostMeets = hostHasEncryptionDetail(hostId, Host.HOST_RBD_VOLUME_ENCRYPTION); + } else if (poolType != null) { + hostMeets = hostHasEncryptionDetail(hostId, Host.HOST_VOLUME_ENCRYPTION); + } else { + hostMeets = hostHasEncryptionDetail(hostId, Host.HOST_VOLUME_ENCRYPTION) + || hostHasEncryptionDetail(hostId, Host.HOST_RBD_VOLUME_ENCRYPTION); + } + if (!hostMeets) { + logger.debug("Host [{}] does not support the encryption mechanism required by volume [{}] (pool type [{}])", + hostId, volume, poolType); + return false; + } + } + return true; + } + private boolean isDeployAsIs(VirtualMachine vm) { long templateId = vm.getTemplateId(); VMTemplateVO template = templateDao.findById(templateId); @@ -1648,11 +1695,7 @@ StateListener<State, VirtualMachine.Event, VirtualMachine>, Configurable { } } - HostVO potentialHostVO = _hostDao.findById(potentialHost.getId()); - _hostDao.loadDetails(potentialHostVO); - - boolean hostHasEncryption = Boolean.parseBoolean(potentialHostVO.getDetail(Host.HOST_VOLUME_ENCRYPTION)); - boolean hostMeetsEncryptionRequirements = !anyVolumeRequiresEncryption(new ArrayList<>(volumesOrderBySizeDesc)) || hostHasEncryption; + boolean hostMeetsEncryptionRequirements = hostMeetsVolumeEncryptionRequirements(potentialHost.getId(), volumesOrderBySizeDesc, storage); boolean hostFitsPlannerUsage = checkIfHostFitsPlannerUsage(potentialHost, resourceUsageRequired); if (hostCanAccessPool && haveEnoughSpace && hostAffinityCheck && hostMeetsEncryptionRequirements && hostFitsPlannerUsage) { diff --git a/server/src/test/java/com/cloud/deploy/DeploymentPlanningManagerImplTest.java b/server/src/test/java/com/cloud/deploy/DeploymentPlanningManagerImplTest.java index 5b03260d2d6..1c803e936fd 100644 --- a/server/src/test/java/com/cloud/deploy/DeploymentPlanningManagerImplTest.java +++ b/server/src/test/java/com/cloud/deploy/DeploymentPlanningManagerImplTest.java @@ -40,6 +40,7 @@ import com.cloud.exception.InsufficientServerCapacityException; import com.cloud.gpu.GPU; import com.cloud.gpu.dao.HostGpuGroupsDao; import com.cloud.gpu.dao.VgpuProfileDao; +import com.cloud.host.DetailVO; import com.cloud.host.Host; import com.cloud.host.HostVO; import com.cloud.host.Status; @@ -197,6 +198,9 @@ public class DeploymentPlanningManagerImplTest { @Inject HostDao hostDao; + @Inject + HostDetailsDao hostDetailsDao; + @Inject CapacityManager capacityMgr; @@ -548,6 +552,69 @@ public class DeploymentPlanningManagerImplTest { Assert.assertFalse("Volumes do not require encryption, but reporting they do", _dpm.anyVolumeRequiresEncryption(volumes)); } + private void stubHostEncryptionDetails(long hostId, boolean qemuNative, boolean rbd) { + Mockito.when(hostDetailsDao.findDetail(hostId, Host.HOST_VOLUME_ENCRYPTION)) + .thenReturn(new DetailVO(hostId, Host.HOST_VOLUME_ENCRYPTION, String.valueOf(qemuNative))); + Mockito.when(hostDetailsDao.findDetail(hostId, Host.HOST_RBD_VOLUME_ENCRYPTION)) + .thenReturn(new DetailVO(hostId, Host.HOST_RBD_VOLUME_ENCRYPTION, String.valueOf(rbd))); + } + + private VolumeVO encryptedVolumeOnPool(Long poolId, Storage.StoragePoolType poolType) { + VolumeVO volume = new VolumeVO("vol1", dataCenterId, podId, 1L, 1L, instanceId, "folder", "path", Storage.ProvisioningType.THIN, (long) 10 << 30, Volume.Type.ROOT); + volume.setPassphraseId(1L); + if (poolId != null) { + volume.setPoolId(poolId); + StoragePoolVO pool = new StoragePoolVO(); + pool.setPoolType(poolType); + Mockito.when(primaryDataStoreDao.findById(poolId)).thenReturn(pool); + } + return volume; + } + + @Test + public void hostMeetsVolumeEncryptionRequirementsRbdPoolNeedsRbdSupportTest() { + VolumeVO volume = encryptedVolumeOnPool(77L, Storage.StoragePoolType.RBD); + stubHostEncryptionDetails(5L, true, false); // qemu-native only + stubHostEncryptionDetails(6L, false, true); // librbd only + Assert.assertFalse("qemu-native-only host must not pass for an encrypted volume on an RBD pool", + _dpm.hostMeetsVolumeEncryptionRequirements(5L, List.of(volume), null)); + Assert.assertTrue("librbd-capable host must pass for an encrypted volume on an RBD pool", + _dpm.hostMeetsVolumeEncryptionRequirements(6L, List.of(volume), null)); + } + + @Test + public void hostMeetsVolumeEncryptionRequirementsNonRbdPoolNeedsQemuSupportTest() { + VolumeVO volume = encryptedVolumeOnPool(78L, Storage.StoragePoolType.NetworkFilesystem); + stubHostEncryptionDetails(5L, true, false); // qemu-native only + stubHostEncryptionDetails(6L, false, true); // librbd only + Assert.assertTrue("qemu-native host must pass for an encrypted volume on a non-RBD pool", + _dpm.hostMeetsVolumeEncryptionRequirements(5L, List.of(volume), null)); + Assert.assertFalse("librbd-only host must not pass for an encrypted volume on a non-RBD pool", + _dpm.hostMeetsVolumeEncryptionRequirements(6L, List.of(volume), null)); + } + + @Test + public void hostMeetsVolumeEncryptionRequirementsNoPoolAcceptsEitherMechanismTest() { + VolumeVO volume = encryptedVolumeOnPool(null, null); + stubHostEncryptionDetails(5L, false, true); // librbd only + stubHostEncryptionDetails(6L, false, false); // no encryption at all + Assert.assertTrue("a volume with no pool yet accepts any encryption mechanism", + _dpm.hostMeetsVolumeEncryptionRequirements(5L, List.of(volume), null)); + Assert.assertFalse("a host with no encryption support must not pass for an encrypted volume", + _dpm.hostMeetsVolumeEncryptionRequirements(6L, List.of(volume), null)); + } + + @Test + public void hostMeetsVolumeEncryptionRequirementsUsesProposedPoolTest() { + VolumeVO volume = encryptedVolumeOnPool(null, null); + StoragePoolVO proposedRbdPool = new StoragePoolVO(); + proposedRbdPool.setPoolType(Storage.StoragePoolType.RBD); + Map<Volume, StoragePool> proposedPools = Map.of(volume, proposedRbdPool); + stubHostEncryptionDetails(5L, true, false); // qemu-native only + Assert.assertFalse("the proposed pool's type must drive the required encryption mechanism", + _dpm.hostMeetsVolumeEncryptionRequirements(5L, List.of(volume), proposedPools)); + } + /** * Root requires encryption, chosen host supports it */ @@ -863,6 +930,12 @@ public class DeploymentPlanningManagerImplTest { Mockito.when(vmProfile.getHypervisorType()).thenReturn(HypervisorType.KVM); Mockito.when(hostDao.findById(hostId)).thenReturn(host); Mockito.doNothing().when(hostDao).loadDetails(host); + // encryption capability checks read host_details through the dao; answer them from the test host's detail map + Mockito.when(hostDetailsDao.findDetail(ArgumentMatchers.anyLong(), ArgumentMatchers.anyString())).thenAnswer(invocation -> { + String detailName = invocation.getArgument(1); + String detailValue = host.getDetails() != null ? host.getDetails().get(detailName) : null; + return detailValue != null ? new DetailVO(host.getId(), detailName, detailValue) : null; + }); Mockito.doReturn(volumeVOs).when(volDao).findByInstance(ArgumentMatchers.anyLong()); Mockito.doReturn(suitable).when(_dpm).findSuitablePoolsForVolumes( ArgumentMatchers.any(VirtualMachineProfile.class),
