bhouse-nexthop commented on PR #14611:
URL: https://github.com/apache/cloudstack/pull/14611#issuecomment-6756505918
### Test report: scale-down when the removed node's VM can't be expunged or
destroyed (commit 1370152b9d)
**Summary:** a normal scale-down still drains, removes and expunges the
node. When the removed node's VM can't be expunged, the scale-down now
completes: it records a WARN event naming the VM, drops the VM's map entry, and
fixes the node count, cores and memory. The cluster goes back to Running, not
Alert. Once the blocker is gone, the expunge task removes the VM. When the VM
can't even be destroyed, the scale-down fails with an error that names the VM.
Its map entry is still dropped, so the cluster stays Running and a retry works.
**Environment**
- A CloudStack 24.0 (main) build with this PR, alongside some of my other
open PRs. The management server and all three agents were upgraded in place to
this build.
- The PR targets 4.22; it wasn't run on a 4.22 build.
`KubernetesClusterScaleWorker` and the expunge code it depends on are the same
on both branches.
- Three KVM hosts, Ceph RBD primary storage, one management server.
- A fresh CKS cluster, Kubernetes v1.32.13, on an isolated network. It had 1
control node and 1-2 workers, and autoscaling was off.
- `expunge.delay=60`, `expunge.interval=60`.
**Results**
| Scenario | Result |
|---|---|
| R: normal scale-down, 2 -> 1 worker | pass, 26 s. The node is removed and
the VM expunged. The map entry is gone, counts are right, and the cluster is
Running |
| E: scale-down with the removed VM's expunge blocked | pass, 311 s.
Completes, WARN event, map entry gone, counts right, Running. The VM is left in
Expunging |
| E2: expunge blocker removed | pass. The expunge task removed the VM 52 s
later, and its root volume is Expunged |
| D: scale-down with the removed VM's destroy blocked | pass. The scale
fails with an error that names the VM. The map entry is dropped, counts are
right, and the cluster is Running |
| D2: retry after D | pass. `size=1` returns OK and the cluster stays
Running |
**How the failures were caused**
- **Expunge (E).** I held the VM's lock-table lock: a row in `op_lock` with
key `user_vm<vm id>`, owned by a management server id that doesn't exist. This
is what a concurrent operation on the VM does. `UserVmManagerImpl.expunge()`
takes that lock first, and gives up after its 300 s wait, so it returns false.
The VM itself is untouched, and deleting the row undoes it.
- **Why not the RBD route.** I didn't make the root volume's RBD image
undeletable (a protected snapshot or a watcher).
`VolumeOrchestrator.cleanupVolumes()` only logs a volume that fails to expunge,
so the VM's expunge would still succeed, and this code path wouldn't run.
- **Destroy (D).** I assigned the VM to a backup offering of the dummy
backup provider and took one backup. `destroyVm(id, expunge=true)` refuses a VM
that has backups ("This Instance ... has a Backup Offering ... with 1
backups"). I had planned this as an expunge blocker, but the check runs in the
destroy as well, so it covers the destroy failure instead. Deleting the backup
undoes it. The backup framework was enabled for the test and turned off again
afterwards.
**R: normal scale-down**
- The last worker was removed. The cluster went `Running` -> `Scaling` ->
`Running`, with size 1:
```
before: node_count=2 cores=6 memory=12288 map rows=3 k8s nodes=3
after: node_count=1 cores=4 memory=8192 map rows=2 k8s nodes=2
removed VM: Expunging, removed set 11 s after the "Removing vm" line; map
rows for it: 0
```
- Events: Scheduled, Started, Completed ("Successfully completed scaling").
The log had no ERROR lines.
**E: expunge fails**
- The worker was scaled back up to 2, and the new worker was the one to be
removed.
- The cluster went `Running` -> `Scaling` -> `Running` (size 1), and the
call returned OK after 311 s. Most of that is the expunge's 300 s lock wait.
- Log:
```
00:13:06 INFO Removing vm ... i-2-487-VM ... from cluster pin-as
00:18:06 WARN Unable to expunge VM ... i-2-487-VM ... of the node removed
from Kubernetes cluster pin-as ...
```
- Events:
```
KUBERNETES.CLUSTER.SCALE WARN Completed Unable to expunge VM ...
i-2-487-VM ... of the node removed from Kubernetes cluster ...
The VM is no longer part of the
cluster and is left in state Expunging, for the expunge
task to retry. Until it is
expunged it keeps the cluster's network in use, and the cluster
can't be deleted with its
network
KUBERNETES.CLUSTER.SCALE INFO Completed Successfully completed scaling
Kubernetes cluster
```
- Afterwards:
```
cluster: Running, size 1 node_count=1 cores=4 memory=8192 map rows=2
k8s nodes=2
removed VM: Expunging, not removed; map rows for it: 0
its SSH port-forwarding rule (2224 -> 22): gone
```
- **Seven minutes later:** the cluster was still Running with the same
counts, so the state scanner didn't move it to Alert. The expunge task had
tried the VM twice and logged `Unable to expunge` both times, since the lock
was still held.
**E2: blocker removed**
- I deleted the `op_lock` row. 52 s later the expunge task had expunged the
VM (`removed` set), and its root volume was `Expunged`. The cluster was
unchanged.
**D: destroy fails, then a retry**
- The scale-down took 10 s. The call failed with this error:
```
Scaling Kubernetes cluster ... pin-as ... failed, unable to destroy VM ...
i-2-486-VM ... (left in state Stopped).
Its node was removed from the Kubernetes cluster and the VM is no longer
part of the cluster; it may still be
running and has to be destroyed manually
```
The log has the cause: `This Instance [...] has a Backup Offering [...]
with 1 backups`.
- The cluster went `Scaling` -> `Running`; a failed scale returns to Running
in the state machine. The map entry was dropped, and the counts already matched
the map: `node_count=1 cores=4 memory=8192`, 2 map rows, 2 Kubernetes nodes. It
was still Running 3 minutes later.
- The VM left behind was Stopped. `listVirtualMachines` showed it with no
Kubernetes cluster, and its SSH port-forwarding rule had been revoked.
- **Retry:** `scaleKubernetesCluster size=1` returned OK, and the cluster
stayed Running. I then deleted the backup and destroyed the VM with expunge,
which worked.
**Result:** pass. A removed node's VM that can't be expunged no longer
leaves a stale map entry or holds the cluster in Alert. The scale-down
completes with a WARN event, and the expunge task cleans the VM up once it can.
A VM that can't be destroyed fails the scale with a clear error, and the
cluster can still be scaled.
**Not covered**
- Deleting the cluster with its network while a removed VM is still waiting
for expunge. The commit says this is refused with an error that names the VM. I
didn't try it here.
- An NSX zone, and VPC tiers.
- The 7-minute wait in E covered two passes of the expunge task. It didn't
include a management server restart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
