bhouse-nexthop commented on PR #14207:
URL: https://github.com/apache/cloudstack/pull/14207#issuecomment-6737450086
### Test report: power reports, unknown instances and expunge on KVM
**Environment**
- A CloudStack 24.0 (main) build with this PR applied, alongside some of my
other open PRs, including #14233.
- Why the result carries over to 4.22: `VirtualMachinePowerStateSyncImpl`
and the power-report handling in `VirtualMachineManagerImpl` are the same on
4.22 and main in the paths this PR changes. I have not run it on a 4.22 build.
- Three KVM hosts on Ubuntu 24.04 (libvirt 10.0.0, QEMU 8.2.2), nested
virtualisation; Ceph RBD primary storage; one management server.
- `ping.interval=60`, so the graceful period is 120 s. DEBUG logging for
`com.cloud`.
- Test VMs are CirrOS, deployed to one host with `hostid=`.
**A lab setting worth knowing:** the agent only sends a forced (out-of-band)
power report on a guest shutdown or crash when `libvirt.events.enabled=true` is
set in `agent.properties`, and that defaults to `false`. My first pass ran with
the default, so it had no forced reports at all. I set it to `true` on the test
host for S4 and S5 below, and back afterwards. Without it, the race this PR
closes can still happen, through a regular report collected just before a state
change, but much less often.
**S1: unknown instance on a host**
- I started a minimal diskless domain named `i-2-99999-VM` with `virsh
create`.
- 42 s later, one WARN:
```
Host reports 1 instance(s) that do not exist in CloudStack DB, they are
running unmanaged. host: Host {"id":1,"name":"<host>",...}, instances:
[i-2-99999-VM]
```
There was still only that one line after several more pings.
- An alert appeared in `listAlerts`: "Instances running on a host that
CloudStack has no record of", with "Host <host> reports 1 instance(s) that do
not exist in CloudStack: i-2-99999-VM. They are running unmanaged and may still
be holding addresses and storage."
- After `virsh destroy`, the next report logged `No unknown instances are
reported anymore`.
- A domain named `foo-test` produced no WARN and no alert, only the existing
DEBUG `Unable to find matched VM in CloudStack DB`.
- **Caveat: the alert can be swallowed.** The PR raises its alerts as
`ALERT_TYPE_SYNC` (27). Stock `AlertManagerImpl` only stores a new alert of a
non-repeating type when there is no unarchived alert of that type for the zone
and pod. On my first pass, an older, unrelated "VM state sync alert" was still
unarchived, and both of this PR's alerts were dropped: only the WARN in the
log, plus `Have already sent: 1 emails for alert type '27' -- skipping send
email`. With that alert archived, both appeared.
- So in a zone that has any open sync alert, the unknown-instance alert
only reaches the log.
- Possible fixes: add 27 to `alert.allowed.repetitive.types` in the docs
or the default, or give the unknown-instance alert a type of its own. This is
the stock dedup and not a bug in the PR, but it limits how visible the new
alert is.
**S2: power-on report for a Destroyed instance**
- I saved VM W's XML, ran `destroyVirtualMachine` (no expunge), and started
the domain again from the XML on the same host.
- The next report logged `Receive power on report when Instance is in
destroyed or expunging state`. The StopCommand followed, and then `Stopped
unmanaged instance ... on host 1`. The domain was gone from `virsh list`.
- The stop took about 2 minutes: the guest ignored the ACPI shutdown, so
the stop waited before destroying the domain.
- With the instance's DB state set to `Error` instead: `Receive power on
report when Instance is in error or unexpected state`. One alert ("Instance
i-2-N-VM is reported running by host 1 but is in Error state. It is not managed
by CloudStack and may need to be stopped on the host."), and no stop: the
domain kept running.
**S3: expunge of an instance in Error stops it on its last host**
- VM X running, DB state set to `Error`, then `destroyVirtualMachine
expunge=true`.
- A StopCommand went to the host, and the domain was gone afterwards. On
stock, `advanceStop` returns at once for an Error instance, and the domain is
left running.
**S4: the qemu process killed 3 s after Running**
- The agent logged `Triggering out of band status update due to completed
self-shutdown or crash of VM`.
- The forced report arrived 3 s after the Running transition, and was
skipped:
```
vm id: 166 - state changed at 08:22:54, which is within the graceful
period (120000 ms); the report may have been collected before that change,
skipping missing report
```
The next two regular reports were skipped the same way.
- At 158 s: `Detected missing VM ... reporting missing (time since last
state update: 157943 ms, graceful period: 120000 ms, forced: false)`. A
StopCommand with `forceStop=false, checkBeforeCleanup=true` followed, and the
VM ended `Stopped`.
- So detection of a real crash is delayed by up to about one graceful
period, but not lost. The run with events off behaved the same through the
regular reports (Stopped at 132 s).
**S5: race stress**
- On the test host, a throwaway non-CloudStack domain was started and its
qemu killed every ~3.5 s, for a forced power report each time. That made 754
forced reports during the run.
- Meanwhile 40 VMs were deployed to that host, 4 at a time. Each batch was
checked 150 s after its deploy, longer than the graceful period, and then
expunged.
| Check | Result |
|---|---|
| VMs that ended in Error | 0 / 40 |
| VMs not Running after 150 s | 0 / 40 |
| Running in the DB but no domain on the host | 0 |
| `within the graceful period ... skipping` for these VMs | 94 |
| `Detected missing VM` for a Starting or just-Running VM | 0 |
| Unknown-instance alerts | 0 |
- The 94 skipped reports are cases where the race happened: a report missing
an instance that had just changed state. Each one was ignored rather than
turned into a stop.
- There was one `Detected missing VM` for a test VM, but that was during its
own destroy. It was in `Stopping` with a pending job after a 2 minute graceful
shutdown, so the report was ignored (`There is pending job or HA tasks working
on the VM`).
**S6: the cost on expunge**
- Every expunge now sends one StopCommand to the last host first.
- For an instance that is already stopped, the agent answers at once, so the
destroy+expunge of a Stopped VM took 10 s end to end.
- For a running one, it is the ordinary stop.
**Result:** pass.
- No orphaned or Error instances under the stress run.
- A forced report racing a start is ignored within the graceful period, and
a real crash is still detected afterwards.
- A Destroyed instance reported running is stopped, and an Error one is
alerted.
- Expunge stops the instance on its host first.
- Unknown CloudStack-named instances are logged once and alerted, with the
dedup caveat above.
**Not covered**
- VMware and other hypervisors, more than one management server, and a
stock-build baseline for S5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
