bhouse-nexthop commented on PR #14233:
URL: https://github.com/apache/cloudstack/pull/14233#issuecomment-6727079616
### Test report: system VM destroy (the `test_11_destroy_ssvm` error in
Trillian tid-17281)
This covers one part of the PR: whether it affects destroying the SSVM and
CPVM. The Trillian run (tid-17281) ended `test_11_destroy_ssvm` in `Error`
after 960 s. The rest of the PR's live behaviour is not covered here (see "Not
covered" below).
**Environment**
- A CloudStack 24.0 (main) build with this PR applied, alongside some of my
other open PRs. The PR's commits apply cleanly to main.
`VirtualMachineManagerImpl` differs between 4.22 and main, but not in the stop
and expunge paths this PR changes, so the result carries over to 4.22.
- The zone was upgraded in place from 4.20 through 4.22 to this build.
- Three KVM hosts on Ubuntu 24.04 (libvirt 10.0.0, QEMU 8.2.2), nested
virtualisation, one cluster.
- Ceph RBD primary storage (squid 19.2.3), NFS secondary storage.
- System VM template 4.22.0 (Debian 12).
- One management server.
**Steps**
1. With all hosts Up and the default `vm.destroy.forcestop=false`, run
`destroySystemVm` on the SSVM three times and on the CPVM three times, one
after another.
2. Each time, record when the API job completes. Check that the old domain
is gone from `virsh list --all` on its host. Then poll `listSystemVms` until
the replacement is `Running` and its agent is `Up`.
3. Set `vm.destroy.forcestop=true` and repeat twice for each, then set it
back to `false`.
4. For each destroy job, read the management server log: the StopCommand
flags, the state transitions, the expunge, and any WARN or ERROR.
**Results**
All ten destroys succeeded, and every replacement came up with its agent
`Up`.
| Run | forcestop | Destroy job start to stop done (MS log) | Replacement
Running | Replacement agent Up |
|---|---|---|---|---|
| SSVM 1 | false | 5.8 s | 64 s | 77 s |
| SSVM 2 | false | 6.6 s | 45 s | 59 s |
| SSVM 3 | false | 5.6 s | 54 s | 68 s |
| CPVM 1 | false | 3.2 s | 59 s | 68 s |
| CPVM 2 | false | 5.0 s | 48 s | 57 s |
| CPVM 3 | false | 5.2 s | 41 s | 50 s |
| SSVM 1 | true | 2.2 s | 46 s | 60 s |
| CPVM 1 | true | 1.7 s | 46 s | 55 s |
| SSVM 2 | true | 2.4 s | 50 s | 64 s |
| CPVM 2 | true | 2.0 s | 46 s | 55 s |
The expunge and the destroy job's completion followed the stop within about
0.6 s (for example, 19:47:55.43 to 19:47:56.05). The "Running" and "agent Up"
times are measured from the destroy call. In every run:
- the old domain was gone from its host;
- the old system VM's host row was removed (`Removing host entry for
secondary storage VM`);
- the destroy job logged no WARN or ERROR, and no `Not releasing the
resources of ...` line.
The management server log for each destroy shows the same sequence:
```
Running -> Stopping [StopRequested]
Sending StopCommand ... "forceStop":"false" (forcestop=false; "true"
with forcestop=true)
Received StopAnswer (~4.8 s graceful shutdown;
~0.7 s forced)
Stopping -> Stopped [OperationSucceeded]
Stopped -> Expunging [ExpungeOperation]
Expunged VM instance ...
Complete async job ..., jobStatus: SUCCEEDED
```
**Why this explains the Trillian failure**
- **The code.** With the default `vm.destroy.forcestop=false`,
`advanceStopForDestroy()` calls `advanceStop(uuid, false, false)`. Every new
branch the PR adds is gated on `releaseOnlyIfHostIsGone` or
`cleanUpEvenIfUnableToStop` being true, so the destroy stop is the same as
before. Trillian runs with the default.
- **The Trillian logs.**
- The destroy itself was quick: job `dca1a978` stopped `s-350-VM` at
21:06:45.9, and it was expunged and SUCCEEDED at 21:06:47.7.
- The replacement `s-352-VM` was Running at 21:07:38, and its "Secondary
Storage Vm is up" alert fired at 21:08:11.
- Neither management server's log has a `StartupSecondaryStorageCommand`
from `s-352-VM` before the test gave up. So the test timed out waiting for the
*replacement's* agent to connect ("Failed to wait for SSVM agent to be Up"). It
did not time out in the destroy.
- **Other PRs.** The same test fails on unrelated PRs. For example, #12426
(tid-16698, kvm-ol8) shows the same `test_11_destroy_ssvm` `Error` at 954 s,
and several other PRs show it as `Failure`.
So I don't think the Trillian error comes from this change. A re-run should
confirm it.
**Not covered here**
- The PR's own scenarios: destroy with the host Disconnected, Alert or Down;
the hung agent; routers; account cleanup; stalled Migrating or Starting
instances. They are planned as a separate run.
- The test build also contains #14207, which adds a StopCommand before
expunge (`ensureInstanceIsStoppedOnLastKnownHost`). It shows as the second,
about 0.1 s StopCommand in each destroy above. It did not affect the timings,
but it was present.
- A second management server: Trillian uses two, and this lab has one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
