bhouse-nexthop opened a new issue, #14622:
URL: https://github.com/apache/cloudstack/issues/14622
### problem
On KVM, resizing a volume on RBD primary storage fails in the agent if this
host's libvirt has not yet cached the image in its view of the pool. A freshly
deployed VM's ROOT volume is the usual case:
```
resizeVolume id=<ROOT> size=12
-> Failed to resize volume operation of volume UUID: [<uuid>] due to -
org.libvirt.LibvirtException: Storage volume not found: no storage vol
with matching path '<ceph pool>/<uuid>'
```
`resizeVolume` with a size or a disk offering, `changeOfferingForVolume` and
`scaleVirtualMachine` all fail the same way, since they all end in the same
`ResizeVolumeCommand`. A size-only grow of a ROOT volume fails too. The resize
works again once something else on that host has made libvirt refresh the pool
(or after a `virsh pool-refresh` by hand).
### Cause
`LibvirtResizeVolumeCommandWrapper` gets the disk with
`pool.getPhysicalDisk()` and then looks the volume up through libvirt with
`conn.storageVolLookupByPath(path)` to call `StorageVol.resize()`.
- Before #14188, `getPhysicalDisk()` went through
`LibvirtStorageAdaptor.getVolume()`, which refreshes the libvirt pool on a
miss. So by the time the wrapper called `storageVolLookupByPath()`, the volume
was in libvirt's cache.
- #14188 made `getPhysicalDisk()` stat RBD images through librbd, on
purpose, so that a miss no longer refreshes the whole pool. The wrapper's
`storageVolLookupByPath()` now runs against libvirt's cache cold.
- CloudStack creates RBD volumes through librbd (the clone from the
template), not through libvirt, so a new volume is not in libvirt's cache on
any host until something refreshes the pool there.
The other libvirt volume lookups in the KVM plugin don't depend on
`getPhysicalDisk()`: `deletePhysicalDisk()` uses `getVolume()`, which still
refreshes on a miss, and `LibvirtMigrateCommandWrapper.deleteLocalVolume()`
only handles local files.
The unit tests mock `Connect.storageVolLookupByPath`, so they did not see
this.
### versions
The 4.22 branch since #14188, and main. Released versions without #14188 are
not affected. Found while testing #14602 in a lab: KVM, Ceph RBD primary
storage, a running Ubuntu VM with its ROOT on RBD.
### The steps to reproduce the bug
1. Deploy a VM on KVM with its ROOT volume on RBD primary storage.
2. Before anything refreshes the libvirt pool on its host (no `virsh
pool-refresh`, no storage stats poll yet), resize its ROOT volume, for example
`resizeVolume id=<ROOT> size=<larger>`.
3. The agent fails with "Storage volume not found: no storage vol with
matching path ...".
### What to do about it?
In the resize wrapper, when the lookup by path fails with "not found"
(VIR_ERR_NO_STORAGE_VOL) on an RBD pool, refresh that pool and look the volume
up once more. That is what `getVolume()` did for this path before #14188, and
it only costs a refresh when the volume is not cached. Other pool types are
unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
