bhouse-nexthop opened a new pull request, #14207:
URL: https://github.com/apache/cloudstack/pull/14207
### Description
An instance can end up running on a KVM host with no record in CloudStack.
Its IP address is released and later handed to another instance, so two
instances answer for the same address. Its root volume stays in `Destroy` and
cannot be deleted while the domain holds it.
Fixes: #14206
The trigger is the out-of-band ping the KVM agent sends when **another**
instance on the same host shuts down or crashes. That ping is processed with
`force`, which skips the graceful period. The KVM report lists only powered-on
domains, so an instance that is still starting is absent from it and is marked
`PowerReportMissing`.
Sequence:
| time | what happens
|
|----------|--------------------------------------------------------------------------------|
| 20:23:5x | DeployVM starts, instance -> Starting, `StartCommand` sent
|
| 20:25:14 | another instance crashes; agent collects a report and sends it
with `force` |
| 20:25:16 | the new domain is created on the host
|
| 20:25:17 | `StartAnswer` arrives, instance -> Running
|
| 20:25:18 | the 20:25:14 report is processed; instance is not in it ->
`PowerReportMissing` |
| | NICs and IP released, instance -> Stopped. No StopCommand is
sent |
| 20:25:19 | the deploy job finds it Stopped, logs an error, instance ->
Error, job fails |
| 20:25:28 | user expunges it; `advanceStop()` returns at once because state
is Error |
| 20:25:39 | expunged and removed. No StopCommand was ever sent
|
One commit per finding, so they can be reviewed independently.
| commit | change |
|--------|--------|
| 1 | do not report an instance missing when its state changed within the
graceful period, forced or not |
| 2 | send a StopCommand before releasing resources on a missing report, as
the PowerOff branch already does |
| 3 | on a power-on report for a Destroyed or Expunging instance, stop it on
the reporting host; alert for Error |
| 4 | send a StopCommand to the last known host before expunging, whatever
the database state says |
| 5 | log instances a host reports that CloudStack has no record of at warn
level, once per change |
| 6 | unit tests |
Commit 1 is the root cause. Commits 2 to 4 are independent backstops, each
of which would have prevented the end result on its own.
Behaviour that does **not** change:
- an instance that has been Running for a while and then shuts itself down
is still detected immediately, so out-of-band stop detection keeps working
- nothing is stopped automatically for an unknown instance, it is only made
visible, since an operator may have put a domain on the host deliberately
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Bug Severity
- [ ] BLOCKER
- [x] Critical
- [ ] Major
- [ ] Minor
- [ ] Trivial
### How Has This Been Tested?
- unit tests added for the sync changes,
`VirtualMachinePowerStateSyncImplTest` passes (11 tests)
- the condition was traced on a running 4.22 deployment: agent logs show the
out-of-band pings lining up one for one with the management server processing
each report, including one collected two seconds before the domain existed and
processed one second after the start completed
- on the affected hosts the forced pings arrived every few seconds, so the
graceful period almost never applied
#### How did you try to break this feature and the system with this change?
- checked that `checkBeforeCleanup` is set correctly for each new
StopCommand. It makes the host refuse to stop a running domain, so it is false
where a running domain is what must be removed (commits 3 and 4) and true where
the point is to back off if the instance turns out to be alive (commit 2)
- checked the missing-report change does not weaken out-of-band stop
detection. The guard only applies within the graceful period of a state change,
and an instance that has been Running is well outside it
- checked that a failed StopCommand does not release resources, so an
unreachable host cannot cause an address to be freed while still in use
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
