bhouse-nexthop opened a new pull request, #14188:
URL: https://github.com/apache/cloudstack/pull/14188
### Description
Starting a VM whose root disk is on RBD primary storage refreshes the
**entire** libvirt storage pool. The cost grows with the number of volumes in
the Ceph pool, and it is paid on every single VM start.
**Why it happens**
`LibvirtStorageAdaptor.getPhysicalDisk()` asks libvirt for the volume. The
volume was just created by the management server, so this host's libvirt has
never seen it and the lookup misses. `getVolume()` then refreshes the whole
pool:
```java
vol = pool.storageVolLookupByName(volName);
if (vol == null) {
refreshPool(pool); // opens and stats EVERY image
in the pool
vol = pool.storageVolLookupByName(volName);
}
```
The existing comment in `getVolume()` already describes the case: *"This can
happen when a volume has just been created on a different host and since then
the libvirt storage pool has not been refreshed."* For a cluster that creates
VMs continuously, that is every start.
**Measured cost**
Ceph pool holding 950 RBD images, measured on the hypervisor:
| operation | time |
| --- | --- |
| list image names only | 0.005 s |
| open + stat + close every image, serially (what a pool refresh does) |
11.67 s |
Running four of these at once does not slow them down (9.9 - 11.1 s each),
so this is not Ceph contention. It is 950 serial round trips.
Agent-side `StartCommand` on a host in a 12-node cluster running short-lived
VMs:
| | p50 | p90 | max |
| --- | --- | --- | --- |
| whole `StartCommand` | 35.2 s | 68.4 s | 141 s |
| the stall inside `getPhysicalDisk` | 20.3 s | 46.6 s | 304 s |
The stall is 0 when the volume happens to be cached, which is what points at
the refresh rather than at fixed work.
### Fix
Look RBD volumes up directly through librbd and skip libvirt entirely.
This makes `getPhysicalDisk()` consistent with the rest of the class.
Creating, cloning, resizing, copying and deleting RBD volumes in
`LibvirtStorageAdaptor` already use `Rados`/`Rbd` directly. Only the lookup
went through libvirt.
- Applies to `StoragePoolType.RBD` only. Every other pool type is untouched.
- The disk path `<ceph pool>/<volume uuid>` is the same format the class
already builds in `createPhysicalDisk`, `createDiskFromTemplate` and
`createDiskFromTemplateOnRBD`.
- Size and virtual size come from `rbd stat`, matching what the class
already does after converting an image into an RBD volume.
### Types of changes
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] build/CI
### How Has This Been Tested?
- Measured the cost of the pool refresh against the observed agent stall, as
in the tables above. `rbd_list` on its own is 0.005 s; opening and stating all
950 images serially, which is what the refresh does, is 11.67 s.
- Ran the same enumeration four times in parallel to rule out Ceph
contention: 9.9 - 11.1 s each, no slowdown.
- Checked that the new path produces the same disk path format, format and
size as the libvirt path for RBD volumes.
- Full KVM plugin test suite on this branch: 676 tests, 0 failures, 1
skipped.
Worth a reviewer's eye: the librbd lookup reports `rbd stat` size as both
size and virtual size. The libvirt path reported `allocation` and `capacity`
separately, which for an RBD pool libvirt derives as `obj_size * num_objs` and
`size`. These agree for RBD, and the class already takes the `rbd stat` size
for both after converting an image, but I would rather that be confirmed than
assumed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
