andrijapanicsb commented on PR #13656:
URL: https://github.com/apache/cloudstack/pull/13656#issuecomment-5147572560
# Functional test report — VMware to KVM cold/warm migration, RBD and
Linstor destinations
Full end-to-end functional test of this PR, run on a two-host KVM cluster
against three destination
storage types: NFS, Ceph/RBD and Linstor.
**Result: 15 of 15 planned test cells pass.**
All cells were run from scratch on the final build. Source VMs were migrated
directly. No clones or
throwaway copies were used, and all source VMs were left intact.
Guests used: Rocky Linux 9.8 (2 disks), Ubuntu 26.04 LTS (1 disk), Windows
Server 2012 R2 (2 disks).
---
## How results were verified
A VM was only recorded as booted when the guest itself provided the
evidence. CloudStack reporting
`state=Running` was not accepted, because that only means the libvirt domain
exists and says nothing
about whether the operating system came up.
| Check | Evidence used |
|---|---|
| Disk order and placement | `listVolumes`: ROOT = deviceid 0, DATADISK =
deviceid 1, on the expected pool |
| Boot | Guest console screenshot showing a login prompt, and/or `virsh
domhostname` answered by the in-guest agent |
| Data integrity | `sha256sum` run inside the migrated guest, compared
against the value recorded at the source |
### Data integrity method
Each source data disk carries a test file (`seed1.bin`) together with a
manifest file holding its
`sha256sum`. For the warm migrations, additional test files were written
**into the running source
guest while replication was in progress**, in between delta cycles, and
their `sha256sum` values were
recorded at the source at the time of writing.
After cutover, each migrated VM was booted and the checksums were computed
again, from inside the
migrated guest, on the migrated volumes. In every cell the checksums matched
the values recorded at
the source exactly, including the files that were created after the initial
full sync had completed.
This confirms that both the initial copy and the subsequent delta cycles
transferred the data
correctly and completely.
---
## Summary
| # | Cell | Destination | Result |
|---|---|---|---|
| C1 | Cold direct-VDDK import, 2-disk source | Ceph/RBD | PASS |
| C2 | Cold direct-VDDK import, 2-disk source | Linstor | PASS |
| C3 | Cold direct-VDDK import, 2-disk source | NFS (qcow2) | PASS |
| C4 | Cold direct-VDDK import, 1-disk source | Ceph/RBD | PASS |
| C5 | Cold import with `nbdcopy` removed (fallback path) | Ceph/RBD | PASS |
| W1 | Warm CBT migration | Ceph/RBD | PASS |
| W2 | Warm CBT migration | NFS | PASS |
| W3 | Warm CBT migration | Linstor | PASS |
| A1 | Adopt a pre-existing volume (`importsource=shared`) | Ceph/RBD | PASS
|
| R1 | Native (non-imported) VM lifecycle | RBD + Linstor | PASS |
| N1 | Cutover while source still powered on, expect rejection | — | PASS |
| N2 | Cold import of a powered-on source, expect rejection | — | PASS |
| N3 | Cancel an in-flight warm migration, expect no orphans | Ceph/RBD |
PASS |
| N4 | Warm start without a disk-offering mapping, expect rejection | — |
PASS |
| N5 | Import with fewer NIC mappings than source NICs, expect rejection | —
| PASS |
| X1 | Windows Server 2012 R2, cold import | Ceph/RBD | PASS (boots) |
| X2 | Windows Server 2012 R2, warm CBT | Ceph/RBD | PASS (boots) |
---
## Disk ordering
The ordering behaviour is correct on every destination type. In all
dual-disk cells the result was
`deviceid 0 = ROOT` and `deviceid 1 = DATADISK`, with the expected sizes and
both volumes on the
requested pool.
The single-disk source produced exactly one volume, with no additional data
disk created. This was
confirmed from inside the guest, where `lsblk` showed a single disk and no
second device.
---
## Warm CBT: delta cycles transfer only changed blocks
This was measured rather than assumed. The full disk set per VM is 20 GB
root plus 5 GB data, i.e.
**26,843,545,600 bytes (25 GiB)**. If a sync cycle were re-copying the whole
image, every cycle would
move about 26.8 GB. Measured `changedbytes` per cycle:
| Destination | Cycle | changedbytes | Block ranges | % of full 25 GiB |
Data written into the source before that cycle |
|---|---|---|---|---|---|
| RBD | 1 | 70,516,736 (~67 MiB) | 41 | 0.26 % | 64 MiB |
| RBD | 2 | 72,744,960 (~69 MiB) | 63 | 0.27 % | (final cutover cycle) |
| NFS | 1 | 3,080,192 (~2.9 MiB) | 35 | 0.011 % | nothing, guest idle |
| NFS | 2 | 53,673,984 (~51 MiB) | 38 | 0.20 % | 48 MiB |
| NFS | 3 | 55,115,776 (~52.6 MiB) | 50 | 0.21 % | (final cutover cycle) |
| Linstor | 1 | 4,456,448 (~4.2 MiB) | 51 | 0.017 % | nothing, guest idle |
| Linstor | 2 | 4,587,520 (~4.4 MiB) | 52 | 0.017 % | nothing, guest idle |
| Linstor | 3 | 106,037,248 (~101.1 MiB) | 61 | 0.40 % | 96 MiB |
The Linstor rows come from a single migration and vary only the amount
written into the running guest:
two consecutive idle cycles moved 4.2 MiB and 4.4 MiB, then 96 MiB was
written into the source and the
next cycle moved 101.1 MiB.
Two results follow from this. An idle cycle moving 2.9 MiB out of 25 GiB is
not a full re-copy, and
the amount transferred tracks the amount actually written, with the small
excess accounted for by
filesystem metadata and normal OS activity.
CloudStack reports this per cycle, for example:
> `VMware CBT cycle 1 ... copied 35 changed block range(s), coalesced into
35 copy range(s) across 2 disk(s), totaling 3080192 bytes.`
together with a `dirtyrate` value.
---
## Data path coverage
<details>
<summary>nbdcopy path and qemu-img fallback, with log output</summary>
With `nbdcopy` available, RBD copies use it over a qemu-nbd bridge:
```
Copying VMware disk <src> to RBD image <pool>/<image> via nbdcopy over
qemu-nbd bridge
```
and Linstor writes directly to the DRBD device:
```
nbdkit -r -U - vddk file=<src> ... compression='fastlz' \
--run 'nbdcopy --destination-is-zero "$uri" /dev/drbd/by-res/<res>/0'
```
For cell C5, `nbdcopy` was removed from the conversion host. The probe
failed and the code fell back
to `qemu-img`, for both disks:
```
Executing command [/bin/bash -c nbdcopy --version ] -> Exit value ... is
[127] (WARN)
nbdkit -r -U - vddk file=<src> ... compression='fastlz' \
--run 'qemu-img convert -f raw -O raw "$uri" rbd:<pool>/<image>...'
```
The resulting VM booted and the checksum taken inside the guest matched the
source value, so the
fallback path produces the same data as the `nbdcopy` path. VDDK
`compression='fastlz'` is active in
both cases.
</details>
---
## Adoption of a pre-existing volume
A standalone bootable RBD image was created outside CloudStack and then
adopted with:
```
importVm importsource=shared hypervisor=KVM storageid=<pool>
diskpath=<image> networkid=<net>
```
The resulting ROOT volume's `path` is the pre-existing image name, so the
volume was adopted in place
with no copy and no data movement. The VM booted, and the root filesystem
was mounted with the
expected size when checked inside the guest.
---
## Negative testing cases
All rejections were returned quickly, and each message identifies both the
problem and the required
action.
> **N1** — `errorcode 431`: *Cannot cut over VMware CBT migration <id>
while source VM <vm> is in power state PowerOn. Gracefully shut down the
source VM, then retry cutover.*
> **N2** — `errorcode 530`: *Direct VDDK import to a block storage pool is
supported only for powered-off VMware VMs. Please power off VM <vm>, or
disable forceconverttopool and use staged import with temporary conversion
storage.*
> **N4** — `errorcode 431`: *Source VM <vm> has 1 data disk(s) but 0
disk offering mapping(s) were provided. Provide one datadiskofferinglist entry
per data disk so the cutover import can place every disk.*
> **N5** — `errorcode 530`: *Different number of nics found on instance
<vm>: 2 vs 1 nics provided.*
N4 and N5 are both validated at start, within seconds, rather than after a
full replication or a large
conversion has already run.
**N3 (cancel)** was checked by inspecting the storage rather than relying on
the API result. During
replication the pool contained the migration's target images. After
`cancelVmwareCbtMigration` there
were zero images matching that migration id, the source hypervisor snapshot
had been removed, no
detached or orphaned volumes remained, and no partial VM record was left
behind.
---
## Windows guests
Scope for the Windows cells was boot verification. The original Windows
Server 2012 R2 source was
migrated directly, without cloning, and left intact afterwards. It has a 40
GB root disk and a 5 GB
data disk.
**Cold import (X1)**
* Disk order correct: ROOT deviceid 0, DATADISK deviceid 1.
* Boots: the console reached the Windows lock screen with a live clock. The
guest also switched the
console to 1024x768, showing the display driver had loaded, and block
statistics showed real guest
I/O.
* The Windows disk-online first-boot script is injected as intended:
```
(<vm>) Windows guest detected; injecting first-boot script to online
migrated disks: ...
(<vm>) virt-v2v: Converting Windows Server 2012 R2 Standard (win2k12r2) to
run on KVM
(<vm>) virt-v2v: Installing firstboot script: .../cloudstack-first...
```
**Warm CBT migration (X2)**
Change tracking was enabled on the source, the initial full sync completed
with the source powered on,
a delta cycle ran, the source was gracefully shut down and cutover completed
with no errors. Disk
order was correct and the migrated guest booted to the Windows lock screen.
The delta cycle moved
16,580,608 bytes (~15.8 MiB) against a 45 GiB disk set, which is 0.035 %.
In-guest networking and data contents were out of scope for the Windows
cells.
---
## Notes
**Disk controller on adopted volumes.** Adopting a volume without further
parameters produced
`<target dev='hdb' bus='ide'/>`. Passing
`details[0].rootDiskController=virtio` produced
`<target dev='vdb' bus='virtio'/>` as expected. When adopting a volume it is
therefore advisable
either to set a guest OS type that implies virtio support, or to pass the
disk controller explicitly
in `details`, so that the VM does not end up on the IDE bus.
**Disks are copied one at a time.** On both the cold and warm paths, disks
are converted sequentially
rather than in parallel. This was confirmed by process inspection, where
only one copy pipeline exists
at any moment and references a single disk, and by log timestamps, where the
second disk starts only
after the first pipeline has exited. Total migration time is therefore the
sum of the disks rather
than the duration of the largest one.
---
## Scope of this testing
- All operations were driven through the API. The migrations were observable
in the UI while they ran,
but the UI wizards were not used to drive them, so this report does not
cover whether the UI collects
and passes the parameters the API requires.
- Migrations were run one at a time. Concurrent migrations were not tested.
- Only the direct-to-pool conversion path (`forceconverttopool=true`) was
exercised. That was
deliberate, since the goal was to confirm the most efficient pool-to-pool
path; it requires a recent
conversion host (EL9 or newer, with a current virt-v2v). The staged import
path using temporary
conversion storage was not exercised.
- Volume adoption (`importsource=shared`) was exercised on Ceph/RBD and
Linstor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
