fermosan opened a new pull request, #14499:
URL: https://github.com/apache/cloudstack/pull/14499
### Description
KVM-hosted VM disks have no way to configure libvirt/QEMU's `error_policy`
(`rerror`/`werror`).
Every disk defaults to QEMU's `report` policy, so any I/O error from
backing storage — even a
brief, otherwise-recoverable one — is passed straight through to the guest
as a real disk
error. This PR adds first-class `error_policy` support to disk/compute
offerings, modeled on
the existing `cacheMode` offering attribute, end to end: schema, domain
model, API, KVM agent,
and the Vue UI (Add/Clone offering forms).
**Problem.** Tested against a KVM cluster backed by NFS primary storage
during a *planned,
successful* storage-side HA failover (floating IP moved in ~6s, well
within normal NFS client
timeouts): a VM with a continuous write workload hit a hard
`blk_update_request: I/O error` on
its data disk within 3-4 seconds — long before any NFS-level recovery had
a chance to run. The
guest's ext4 filesystem aborted its journal and remounted read-only. Root
cause: the disk's
libvirt XML carries no `rerror`/`werror` attribute at all, so QEMU's
default `report` policy
applies unconditionally, independent of how well the storage/HA layer
itself is tuned.
Manually patching a live domain's XML with `error_policy='stop'` and
re-running the same
failover eliminated the problem entirely — identical failover, zero
guest-visible impact.
Today the only way to get this is manually patching each VM's persisted
domain XML outside
CloudStack's orchestration, which gets silently overwritten the next time
CloudStack redefines
the domain.
**What this adds:**
- **Schema**: `disk_offering.error_policy` column (default `'report'`),
propagated to
`disk_offering_view`/`service_offering_view`/`volume_view`.
- **Domain model**: `DiskOffering.DiskErrorPolicy` enum (`REPORT`, `STOP`,
`IGNORE`,
`ENOSPACE`), getter/setter on `DiskOffering`/`DiskOfferingVO`.
- **API**: new `errorpolicy` param on
`createDiskOffering`/`updateDiskOffering` and the
compute-offering equivalents, surfaced on
`DiskOfferingResponse`/`ServiceOfferingResponse`.
- **Orchestration**: value carried `DiskOfferingVO` ->
`VolumeObject`/`VolumeObjectTO` ->
hypervisor command, mirroring the existing `cacheMode` path.
- **KVM agent**: `LibvirtVMDef.DiskDef` renders `error_policy='<value>'`
on the `<driver>`
element, set from the volume's offering in `KVMStorageProcessor`.
- **UI**: `errorpolicy` exposed on the Add/Clone Disk Offering and
Add/Clone Compute Offering
forms, defaulting to `report`.
- Other hypervisors (XenServer/VMware) simply ignore the new offering
field — this is a
libvirt/QEMU-specific concept.
Functional behavior change: none for existing offerings/VMs (default
`report` is the current
implicit behavior). Opt-in per offering going forward.
Default is `report` (fully backward compatible); supported on both disk
and compute offerings,
matching `cacheMode`'s precedent; alerting on a long-paused domain is
intentionally left to
operators/monitoring, not part of this feature.
Fixes: #14366
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Feature/Enhancement Scale
- [x] Major
- [ ] Minor
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [ ] Major
- [ ] Minor
- [ ] Trivial
### Screenshots (if appropriate):
<img width="650" height="561" alt="image"
src="https://github.com/user-attachments/assets/f5a91548-d257-42d5-b7b5-aaaef9ce4d33";
/>
<img width="781" height="694" alt="image"
src="https://github.com/user-attachments/assets/a6b47b6e-b963-44ab-995d-770d8e175558";
/>
### How Has This Been Tested?
Extensive manual, end-to-end testing on a real KVM cluster with three
storage backends: local
disk, a self-hosted NFS pool, and a real production HA NFS cluster
(Pacemaker-managed,
active/standby).
- **Core behavior**: `error_policy=stop` pauses the VM on a real I/O error
(via both a
disk-full/ENOSPC test and a real production NFS HA failover) and resumes
cleanly with zero
data loss end-to-end (API -> DB -> KVM agent -> libvirt -> QEMU).
Without the fix (default
`report`), the same failover caused the guest to remount read-only
within 3-4 seconds.
- Confirmed `virsh update-device --live` rejects changing `error_policy`
on a running domain —
it only takes effect from the persisted definition at next (re)start; a
full stop/redefine is
required for an already-running VM to pick up a changed offering.
- **Mixed-storage topology**: tested a 3-way rotation of root/data disks
across local storage,
local-NFS, and production-NFS, mirrored for both Linux and Windows
guests, to rule out
cross-contamination between disks on different pools on the same VM.
- **Windows guest validation**: confirmed identical pause/resume behavior
to Linux once the
virtio-blk driver is installed (Windows has no built-in virtio drivers,
so a virtio-bus data
disk is otherwise invisible to the guest).
- **Combined-outage test** (6-VM fleet, both NFS pools down
simultaneously): only the VMs whose
disks spanned *both* outaged pools actually paused; data integrity
verified via a continuous
per-second write-and-verify loop on every VM throughout, confirmed via
the resulting log
sequence rather than just checking VM state.
#### How did you try to break this feature and the system with this change?
- Disk-full (ENOSPC) on local storage, and a real HA failover on
NFS-backed disks — both
correctly paused under `stop` and resumed with zero data loss.
- Mixed root/data topology across three different storage backends
simultaneously, on both
Linux and Windows, specifically to look for cross-contamination between
a VM's own disks.
- A simultaneous outage of two separate NFS pools across a 6-VM fleet,
checking not just which
VMs paused but verifying actual data integrity afterward, not just final
VM state.
- Deliberately went looking for the feature's limits, not just its success
cases: found and
root-caused a related gap via live, controlled fault injection — a
storage backend that goes
silently unreachable under a `hard` NFS mount (as opposed to returning a
real I/O error)
bypasses `error_policy=stop` entirely, since the guest's I/O just hangs
rather than erroring,
with no visible signal to libvirt/CloudStack. Confirmed this is
protocol-independent (same
behavior under NFSv3 and NFSv4) via a dedicated isolated test. Worth
documenting as a known
limitation alongside this feature rather than treating it as a blocker —
it's outside what
`error_policy` can address by itself.
- Confirmed the config-change-doesn't-apply-live constraint so
UX/documentation expectations
are set correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
