weizhouapache opened a new pull request, #14426:
URL: https://github.com/apache/cloudstack/pull/14426
### Description
`migrateVolume` with `livemigrate=true` on KVM was refused for every pair of
pools, except for storage with its own handling (StorPool, PowerFlex). The
error told the operator to use `migrateVirtualMachineWithVolumes`, which moves
the whole Instance to another host. This PR lets an operator move a single
volume of a running Instance, without stopping the Instance and without moving
it, between:
* two Ceph RBD pools;
* an RBD pool and an NFS, SharedMountPoint or host-local pool, in both
directions;
* NFS, SharedMountPoint and host-local pools between themselves.
Encrypted volumes can also be migrated between RBD and NFS, online and
offline.
How it works:
* **Management server.** `VolumeApiServiceImpl` accepts these pool pairs for
a KVM live migration. A local pool has to be the one of the host the Instance
is running on, as the copy runs there. `AncientDataMotionStrategy` keeps the
volume's format right after the move (RAW on RBD, QCOW2 on a file pool) and its
encryption format (`luks2` on RBD, `luks` elsewhere).
* **KVM agent.** `LibvirtMigrateVolumeCommandWrapper` builds the disk XML of
the destination for the type of the destination pool (an RBD image or a qcow2
file) and runs a libvirt block copy on the running domain. The whole disk is
copied, so a volume cloned from a template does not depend on a backing file of
the source pool. `SHALLOW` is used only from RBD to RBD. After the pivot the
source volume is removed. An encrypted volume gets its destination created with
the encryption header first; an offline copy goes through `qemu-img` with both
secrets.
* **A format bug found on the way.** `LibvirtStorageVolumeXMLParser` read
the volume format with its two arguments swapped, so it never found one. After
a migration from RBD, a qcow2 file still named like an RBD image was taken for
a raw one, and libvirt then failed to start the Instance again. The parser now
reads `<format type='…'/>`, so `getPhysicalDisk` reports the format libvirt
sees.
The branch has four commits:
1. `utils: add RbdUtils for the Ceph monitors of an RBD pool`. This is the
first commit of #14402, so the encrypted copy can build its `mon_host` with
`RbdUtils.getRbdMonitors`. It will drop out of this PR when #14402 is merged.
2. `KVM: live volume migration on ceph`. RBD to RBD.
3. `smoke tests: exercise online volume migration for KVM on RBD`.
4. `KVM: live migrate volumes between RBD, NFS, SharedMountPoint and local
pools`. Everything else above, with its unit tests and
`test_volume_migration_rbd_nfs.py`.
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Feature/Enhancement Scale
- [x] Major
- [ ] Minor
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [ ] Major
- [ ] Minor
- [ ] Trivial
### How Has This Been Tested?
**Unit tests, on this branch:** `RbdUtilsTest` (17),
`VolumeApiServiceImplTest` (167), `AncientDataMotionStrategyTest` (19),
`LibvirtMigrateVolumeCommandWrapperTest` (12) and `LibvirtStorageAdaptorTest`
(23), all passing. New cases cover the pool pairs that are accepted, the local
pool of another host, and the dispatch of the agent to the block copy.
**A live matrix, on the same changes from my `4.23.0-ceph` branch**, which
carries them together with the commits of my other Ceph PRs. Two KVM hosts;
Ceph with two pools; two NFS pools; two SharedMountPoint pools; one local pool
per host. One Instance with a ROOT and a DATA volume, moved together, 23 legs:
RBD to RBD and back, RBD to and from NFS, SharedMountPoint and local, and every
pair among NFS, SharedMountPoint and local. After each leg:
* both volumes are on the expected pool in the API;
* the checksum of the first 31 KiB of the ROOT disk is unchanged, and a 4
MiB pattern written to the DATA disk before the first move is intact;
* the SSH port of the Instance answers;
* then the Instance is stopped and started, and the same four checks are
repeated.
Four of five full matrices passed. A 5 GiB DATA disk took about 45 to 95
seconds to copy, a small ROOT disk 3 to 9 seconds.
The fifth run failed once, on an RBD to RBD copy of the DATA disk. The copy
finished and the Instance switched to the new image, but the removal of the old
image by libvirt failed after about a minute ("failed to remove volume … No
such file or directory"). The migration was reported as failed, so CloudStack
kept the old pool while the Instance used the new one. I did not find the
cause, and it did not happen again in the three runs that followed.
**Not done:** the Marvin smoke test `test_volume_migration_rbd_nfs.py` has
not been run by Marvin from this branch (Marvin does not work on that setup).
Offline and detached migrations and encrypted volumes were not part of the
final matrices. `migrateVirtualMachineWithVolumes` was not retested.
#### How did you try to break this feature and the system with this change?
* The Instance is stopped and started after every move. That is what exposed
the format bug above, because a wrong format on the volume only shows when the
Instance starts again.
* A local pool of a host other than the one the Instance runs on is refused
with a clear error. Pool pairs outside the list keep the old error.
* A destination on a pool over its usage threshold is refused by the
existing capacity check, as before.
* This branch is based on `apache/main`, which does not yet have #14188 or
#14393. Two edits of this work to `mon_host` lines in methods from those PRs
were left out, so expect a small conflict in `LibvirtStorageAdaptor` when they
merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
