bhouse-nexthop commented on PR #14233:
URL: https://github.com/apache/cloudstack/pull/14233#issuecomment-6740870727
### Test report: destroy and router replacement with the host unreachable,
hung, Down, or the instance stalled (S1-S7, retest on the current head)
This follows the earlier report on system VM destroy (S8, the
`test_11_destroy_ssvm` error). It covers the PR's own scenarios.
The first run of these scenarios was on the PR before 095ad969e6. S5 failed
there:
- `restartNetwork cleanup=true` succeeded with the router's host
Disconnected.
- The rolling restart stopped the old router with an explicit forced stop,
so it was released without the host confirming. Its domain kept running next to
the new router.
095ad969e6 changes that. I upgraded the lab to a build with it and reran S5,
S5 for a VPC, an all-hosts-Up control, S7b, and S1, S4, S7a and S8 as
regression checks. S2, S3 and S6 come from the first run. 095ad969e6 does not
touch their paths: the destroy stop, an explicit forced stop of an instance,
and account cleanup.
**Environment**
- A CloudStack 24.0 (main) build with this PR, including 095ad969e6, applied
alongside some of my other open PRs, including #14207. All three hosts and the
management server were upgraded in place to the same build.
- 4.22 (the PR's target): not run on a 4.22 build. The code paths match in
both branches:
- `VirtualMachineManagerImpl` and `AccountManagerImpl` don't differ
between 4.22 and main in the stop, expunge and account-cleanup paths.
- `VpcManagerImpl`'s rolling restart is identical.
- In `NetworkOrchestrator.rollingRestartRouters()`, only the opening "is
rolling restart enabled" check differs. The part this PR changes, the stop and
destroy of the old routers, is the same in both.
- Three KVM hosts on Ubuntu 24.04 (libvirt 10.0.0, QEMU 8.2.2), nested
virtualisation, one cluster; Ceph RBD primary storage; one management server.
- `vm.destroy.forcestop=true` for these runs (except where noted), set back
to `false` afterwards. `ping.interval=60`, `ping.timeout=2.5`,
`network.rolling.restart=true` (default).
- All test VMs (CirrOS) and the routers under test were on one host, host 1.
- **About host states in this lab:** with only RBD primary storage,
`KVMInvestigator` has no HA storage to check, so a host with its agent stopped
stays `Disconnected`. In the first run it stayed that way for 10+ minutes and
never went Alert or Down. A truly Down host could not be produced without
killing other workloads on the hypervisor, so S4 and S5-Down set `Down` in the
DB.
**Results**
| Scenario | Run on | Result |
|---|---|---|
| S1: destroy with the host Disconnected | first run, and again on the
current head | pass |
| S2: hung agent (`kill -STOP`) on an Up host | first run | pass |
| S3: explicit `stopVirtualMachine forced=true` still releases | first run
(and the forced router stops in S5 below) | pass |
| S4: destroy with the host Down (DB-forced) | first run, and again on the
current head | pass |
| S5: `restartNetwork cleanup=true`, router on a Disconnected host |
**failed on the first run**; current head | pass |
| S5-VPC: `restartVPC cleanup=true`, VPC router on a Disconnected host |
current head | pass |
| S5-Up: rolling restart with every host Up (network and VPC) | current head
| pass |
| S5-Down: rolling restart with the router's host Down (DB-forced),
`vm.destroy.forcestop=false` | current head | pass |
| S6: account cleanup with an instance on a Disconnected host | first run |
pass |
| S7a: stalled Migrating (DB-forced) | first run, and again on the current
head | pass |
| S7b: stalled Starting (DB-forced), destroy with
`vm.destroy.forcestop=true` | **stock error on the first run**; current head |
pass |
| S8: SSVM destroy (quick recheck) | current head | pass |
**S5: `restartNetwork cleanup=true` with the router on the Disconnected
host**
- **Setup:** isolated network with source NAT, its router on host 1. I
stopped host 1's agent; the host was `Disconnected` within seconds. After 30 s,
I ran the restart.
- **The restart failed** in about 2 s:
```
Resource [Host:1] is unreachable: Unable to replace router r-373-VM: its
host <host 1> is Disconnected and may still be running it. Retry once the host
is Up, or stop the router with forced=true first.
```
- The check runs before the new router is deployed:
- No new `vm_instance` row was created, and the network still had one
router.
- `r-373-VM` was still `Running` on host 1 in CloudStack, and its domain
was still running there.
- `stopRouter forced=true` (the explicit decision) returned `Stopped`.
- The same restart then succeeded. A new router, `r-375-VM`, came up on host
2, and `r-373-VM` was destroyed and expunged.
- On the first run, without 095ad969e6, the same restart succeeded on the
first try. The old router was released with `Unable to actually stop ... but
continue with release because it's a force stop`, and its domain kept running.
**S5-VPC: `restartVPC cleanup=true` with the VPC router on the Disconnected
host**
- **Setup:** a VPC (default VPC offering) with one tier and a CirrOS VM in
it. The VPC router was on host 1, and host 1's agent was stopped.
- The restart failed the same way, with no new router deployed. The old
router stayed `Running` in CloudStack and on host 1:
```
Resource [Host:1] is unreachable: Unable to replace router r-374-VM: its
host <host 1> is Disconnected and may still be running it. Retry once the host
is Up, or stop the router with forced=true first.
```
- After `stopRouter forced=true`, the restart succeeded. The new router
`r-376-VM` came up on host 2, and the old one was expunged.
**S5-Up: the rolling restart with every host Up still replaces the router**
- `restartNetwork cleanup=true`: a new router was deployed and the old one
destroyed. The old router's domain was gone from its host, so the host
confirmed the stop.
- `restartVPC cleanup=true`: the same, and the tier VM stayed Running.
**S5-Down: rolling restart with the router's host Down (DB-forced while the
agent was stopped)**
- I set `vm.destroy.forcestop=false` for this step, to check that the stop
of the old router stays forced whatever that setting says.
- The restart succeeded. A new router came up on host 3, and the old one was
released:
```
WARN Unable to actually stop VM instance
{"id":370,"instanceName":"r-370-VM","state":"Stopping","type":"DomainRouter",...}
DEBUG Expunged VM instance {"id":370,"instanceName":"r-370-VM",...}
```
This is the "host is gone" case.
- The host status was then put back to `Disconnected`.
- As expected, the domain kept running on the real host, which never got the
stop.
**After host 1 reconnected (old router domains)**
- The agent was started and host 1 was `Up` again 21 s later.
- Three old router domains were still running on host 1: `r-373-VM` and
`r-374-VM` (from the explicit forced stops) and `r-370-VM` (released as Down).
Their DB rows were expunged.
- #14207 is in this build. The first power report flagged the three domains
and raised the "Instances running on a host that CloudStack has no record of"
alert:
```
WARN Host reports 3 instance(s) that do not exist in CloudStack DB, they
are running unmanaged. host: ..., instances: [r-370-VM, r-373-VM, r-374-VM]
```
- They were not stopped automatically, and were still running after 200 s. I
destroyed them by hand.
- This is the expected cost of `forced=true`, or of a host wrongly marked
Down. Without #14207 they would not be reported at all.
- The S4 VM (destroyed while the host was DB-forced Down) still had a row in
`Destroyed` state. On reconnect, #14207 stopped it automatically: `Stopped
unmanaged instance ... on host 1`.
**S1: agent stopped (host Disconnected)**
- `destroyVirtualMachine` failed with `Unable to stop VM instance ... Host
with specified id is not in the right state: Disconnected`, after:
```
WARN Not releasing the resources of VM instance
{...,"state":"Stopping",...}: its host Host {"id":1,"name":"<host 1>",...} is
Disconnected and did not confirm the instance is stopped. Retry the destroy, or
stop the instance with forced=true if it is known to be gone.
```
- The VM stayed `Running`. Its domain was still in `virsh list` on host 1,
and its NIC row (IP, MAC, state `Reserved`) was unchanged.
- Once the agent was started and the host was Up, the same destroy
succeeded, and the domain was gone.
- Same result in the first run and on the current head.
**S2: hung agent on an Up host (`kill -STOP` of the agent JVM), destroy at
once (first run)**
- The destroy **failed after 234 s**, by which time the host had gone Alert:
```
WARN Unable to stop VM instance {...,"state":"Stopping",...}
WARN Not releasing the resources of VM instance {...}: its host <host 1>
is Alert and did not confirm the instance is stopped. Retry the destroy, or
stop the instance with forced=true if it is known to be gone.
```
- After `kill -CONT`, the host was Up again 31 s later. The VM was still
Running, and its domain was still on host 1.
- The retried destroy succeeded, and the domain was gone.
**S3: an explicit `stopVirtualMachine forced=true` still releases (first
run)**
- With host 1 Disconnected, the forced stop returned `Stopped`, and the NIC
went to `Allocated`, as before.
- The domain was still running on host 1, which is expected: the host never
got the stop. I destroyed it by hand.
- The forced `stopRouter` calls in S5 and S5-VPC on the current head behaved
the same way.
**S4: host Down (DB-forced)**
- With the agent stopped, I set the host's status to `Down` in the DB,
destroyed a VM, then set the status back.
- The destroy succeeded and released the NIC (`Allocated`). This is the
"host is gone" case, where the forced release is still taken.
- The domain kept running on the real host, as it would on a host that was
only marked Down.
- First run: I destroyed it by hand.
- Current head: #14207 stopped it when the host reconnected (see above).
**S6: account cleanup with an instance on a Disconnected host (first run)**
- `deleteAccount` returned success and logged:
```
WARN Not releasing the resources of VM instance {"id":218,...}: ...
WARN Not expunging instance VM instance {"id":218,...}, it may still be
running on its host.
WARN Deferring the rest of the cleanup of account Account
[{"accountName":"acct-x",...}]: it has instances that may still be running.
```
- The VM stayed Running with its NIC, and the account's network stayed
`Implemented`.
- The next cleanup run (`account.cleanup.interval=300` for the test), with
the host still away, deferred again.
- After the agent was started, the cleanup finished within 223 s: the VM was
expunged, the domain gone from host 1, the network destroyed, and the account
removed.
**S7a: stalled Migrating**
- **Setup (DB-forced):** `state=Migrating`, `host_id` set to host 2, and
`last_host_id` set to host 1, where the VM really runs.
- The destroy was refused:
```
Unable to stop VM instance {...,"state":"Migrating",...} in state
Migrating: the host it is migrating from may still be running it
```
- The domain stayed on host 1, and the NIC stayed `Reserved`.
- After the DB state was put back to Running, the destroy succeeded.
- Same result in the first run and on the current head.
**S7b: stalled Starting (DB-forced on a running VM), one destroy with
`vm.destroy.forcestop=true`**
- **First run:** the release happened, then the API call failed with `We
cannot proceed with stop VM ... since it is not in 'Stopping' state, current
state: Stopped`, and a second destroy was needed.
- That error is stock. The forced-cleanup branch of `advanceStop()` fell
through to the Stopping check after a successful cleanup.
- 095ad969e6 adds the missing `return`.
- **Current head: the destroy succeeded in one call**, and the VM is
`Destroyed`.
- The ordinary stop went first: `StopCommand` with `forceStop=false`.
- The forced cleanup followed (`Unable to transition the state but we're
moving on because it's forced stop ... from Starting via StopRequested`). Host
1 confirmed, and the release followed (`Starting -> Stopped
[AgentReportStopped]`).
- The domain was gone and the NIC `Allocated`.
- The work job ended `SUCCEEDED`, and the destroy went on to `Destroying
vm ... expunge flag off`.
- **Timing:** the call took about 2 minutes this time (4 s in the first
run). The CirrOS guest did not answer the graceful shutdown, so the agent
waited for its timeout (`Timed out waiting for domain ... to shutdown
gracefully`) and then destroyed the domain. That is how any ordinary stop
behaves; it is not a hang.
**S8: system VM destroy (quick recheck on the current head)**
- With all hosts Up and `vm.destroy.forcestop=true`, `destroySystemVm` on
the SSVM completed in 5 s, and the old domain was gone.
- The replacement was Running with its agent Up 57 s after the destroy call.
**Other observations (not from this PR)**
- The failed `restartNetwork` is logged at ERROR, as `Unexpected exception
while executing ... RestartNetworkCmd`, with error code 530. The failed
`restartVPC` is logged at WARN with 534.
- `RestartNetworkCmd.execute()` lets `ResourceUnavailableException` reach
the dispatcher, while `RestartVPCCmd` catches it. That is existing behaviour
for any `ResourceUnavailableException` from a network restart.
**Result:** **pass.**
- A destroy no longer releases an instance in these cases:
- its host is Disconnected, Alert or hung (S1, S2);
- its migration source may still be running it (S7a).
- A destroy still releases when the host is Down (S4), or when the caller
forces the stop (S3).
- A rolling `restartNetwork`/`restartVPC` with `cleanup=true` refuses while
an old router's host is Disconnected, before deploying anything (S5, S5-VPC).
It succeeds after an explicit `stopRouter forced=true`, and it succeeds when
every host is Up (S5-Up) or the router's host is Down (S5-Down).
- Account cleanup defers and then completes once the host is back (S6).
- A destroy of an instance stalled in Starting now completes in one call
(S7b).
**Not covered**
- A real host Down: S4 and S5-Down used the DB.
- The window in the "Known limitations": a host that becomes unreachable
while the new router is being deployed.
- Network GC with the router on an unreachable host.
- Redundant routers.
- More than one management server.
- A 4.22 build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
