bhouse-nexthop opened a new issue, #14206:
URL: https://github.com/apache/cloudstack/issues/14206
##### ISSUE TYPE
* Bug Report
##### COMPONENT NAME
~~~
engine-orchestration, VM power state sync, KVM
~~~
##### CLOUDSTACK VERSION
~~~
4.22
~~~
##### CONFIGURATION
KVM hosts. Any host where instances stop or crash while another instance is
being deployed.
##### OS / ENVIRONMENT
KVM / libvirt.
##### SUMMARY
An instance can end up running on a KVM host with no record in CloudStack.
Its IP address is released and later handed to another instance, so two
instances answer for the same address. The root volume is marked for deletion
but cannot be deleted while the domain holds it, so the storage is stranded too.
The trigger is the out-of-band ping the KVM agent sends when **another**
instance on the same host shuts down or crashes.
##### STEPS TO REPRODUCE
Needs a host with instance churn during a deploy.
1. Deploy an instance on a KVM host.
2. While the `StartCommand` is still in flight, have another instance on the
same host shut itself down or crash.
3. That fires `LibvirtDomainListener.onLifecycleChange`, which calls
`Agent.triggerUpdate()` and sends a `PingCommand` with `outOfBand=true`.
4. Repeat as needed. On a busy host these pings can arrive every few seconds.
The window that does the damage is small: the report has to be collected
before the new domain exists but processed after the start job finishes. It
does not fire on every deploy, but it recurs.
##### EXPECTED RESULTS
The deploy succeeds, or it fails and the host is told to stop the instance.
An instance absent from a report collected before it started is not treated
as missing.
##### ACTUAL RESULTS
Sequence, from a real occurrence (times shortened):
| time | what happens
|
|----------|--------------------------------------------------------------------------------|
| 20:23:5x | DeployVM starts, instance -> Starting, `StartCommand` sent
|
| 20:25:14 | another instance crashes; agent collects a report and sends it
with `force` |
| 20:25:16 | the new domain is created on the host
|
| 20:25:17 | `StartAnswer` arrives, instance -> Running
|
| 20:25:18 | the 20:25:14 report is processed; instance is not in it ->
`PowerReportMissing` |
| | NICs and IP released, instance -> Stopped. **No StopCommand is
sent** |
| 20:25:19 | the deploy job finds it Stopped, logs an error, instance ->
Error, job fails |
| 20:25:28 | user expunges it; `advanceStop()` returns at once because state
is Error |
| 20:25:39 | expunged and removed. **No StopCommand was ever sent**
|
| 20:25:40 | every later report: `Unable to find matched VM in CloudStack
DB. name: i-2-3-VM`|
The domain is still running the whole time.
##### CAUSE
Five separate things line up.
**1. `force` skips the graceful period for a *missing* instance.**
`VirtualMachinePowerStateSyncImpl.processMissingVmReport()`:
~~~java
if (!force) {
vmsThatAreMissingReport =
filterOutdatedFromMissingVmReport(vmsThatAreMissingReport);
}
...
if (force || (milliSecondsSinceLastStateUpdate >
milliSecondsGracefulPeriod)) {
~~~
`force` comes from `PingCommand.outOfBand`, set only by
`Agent.triggerUpdate()`, called only from `LibvirtDomainListener` on a
self-shutdown or crash. It was added so a *reported* stop takes effect
immediately. Applying it to the *absence* of a report is the defect: a report
says nothing about an instance that did not exist yet when it was collected.
The KVM report lists only powered-on domains, so an instance that is still
starting is simply absent.
The log line makes this hard to see. It prints "has passed graceful period"
even when `force` short-circuited the check, so it reports an elapsed time far
below the graceful period as having passed it.
**2. The missing-report branch releases resources without stopping
anything.**
`VirtualMachineManagerImpl.handlePowerOffReportWithNoPendingJobsOnVM()`:
~~~java
if (PowerState.PowerOff.equals(vm.getPowerState())) {
...
Pair<Boolean, String> result = sendStop(vmGuru, profile, true, true);
if (!result.first()) { return; } else { releaseVmResources(profile,
true); }
} else if (PowerState.PowerReportMissing.equals(vm.getPowerState())) {
releaseVmResources(profile, true); // no StopCommand
}
~~~
The IP is freed while the instance is still using it.
**3. Expunge never contacts the host for an instance in Error or Stopped.**
`advanceStop()` returns before it looks at the host id:
~~~java
if (state == State.Stopped) return;
if (state == State.Destroyed || state == State.Expunging || state ==
State.Error) return;
~~~
`vm.destroy.forcestop` does not help, the early return happens first.
**4. A power-on report for a destroyed instance is only logged.**
`handlePowerOnReportWithNoPendingJobsOnVM()` has `case Destroyed: case
Expunging:` log and `break`. CloudStack knows the host and the instance name at
that moment and does nothing with them.
**5. Unknown instances are only logged at debug.**
`convertVmStateReport()` writes one debug line per unknown instance per
report and nothing acts on it, so the condition can persist unnoticed
indefinitely.
##### IMPACT
- an instance runs unmanaged and invisible
- its IP is reassigned, giving an address conflict between two live instances
- its root volume stays in `Destroy` and cannot be deleted while the domain
holds it, so primary storage is stranded
- for RBD the delete fails on the image watcher, which is the only reason
the disk is not deleted underneath a running guest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
