bhouse-nexthop commented on PR #14609:
URL: https://github.com/apache/cloudstack/pull/14609#issuecomment-6746838959
### Test report: concurrent CKS operations and stuck-state recovery, on a
live cluster
**Summary:** concurrent operations are refused cleanly, and a cluster left
in Upgrading or Scaling by a killed management server is recovered on the
scanner's first pass. Starting a cluster in Alert holds Starting until the
start ends. The recovery fixes the state only, as the description says. What
the interrupted operation left behind is still there, and two of those
leftovers needed a manual step. Details are below.
**Environment**
- A CloudStack 24.0 (main) build with this PR, alongside some of my other
open PRs. Among them are #14604 (scaling in Alert) and #14611 (drain and
resumable upgrade), which touch the same workers.
- The PR targets 4.22; it wasn't run on a 4.22 build. The state machine, DAO
and scanner code it changes is the same on both branches. One finding below
only applies to main.
- Three KVM hosts, Ceph RBD primary storage, one management server.
- A CKS cluster with 1 control node and 1 worker, isolated network, upgraded
from v1.32.5 to v1.33.1 during the test.
- The management server was killed with `systemctl kill -s KILL
cloudstack-management` and started again. Its java process restarted each time.
The scanner's first pass is 300 s after start.
**Results**
| Scenario | Result |
|---|---|
| S1a: two `size=2` scales at once | pass: one runs, the other is refused
with "please retry" |
| S1b: a scale and a stop at once | pass: the stop is refused, and the scale
completes |
| S2a: `addNodesToKubernetesCluster` on a Stopped cluster | pass: refused,
and no ISO is attached to the candidate VM |
| S2b: stop while Scaling | pass: refused |
| S3a: management server killed during an upgrade, after the first node |
pass: Upgrading -> Alert on the first scanner pass, then Running. On main the
retry needs a manual ISO detach (see the main-only finding) |
| S3b: management server killed during a scale-up | pass for the state:
Scaling -> Running on the first scanner pass. It leaves an unjoined VM and
missing SSH rules (see the leftovers) |
| S4: start a cluster in Alert | pass: Starting is held through the whole
start, then Running once |
**S1: concurrent operations**
- Two `scaleKubernetesCluster size=2` requests sent together:
```
first: "Kubernetes cluster cluster-b changed from state Running to
Scaling before ScaleUpRequested could start; another operation may be in
progress, please retry once it has finished"
second: OK -> size 2, one new worker
```
- A scale and, 0.3 s later, a stop:
```
scale: OK
stop: "Kubernetes cluster cluster-b is in state Scaling, which does not
allow StopRequested"
```
**S2: refusals from states with no transition**
- `addNodesToKubernetesCluster` with a stopped candidate VM, on a Stopped
cluster:
```
"Failed to add nodes to cluster ID 1 due to: Kubernetes cluster cluster-b
is in state Stopped, which does not allow AddNodeRequested"
```
The cluster stayed Stopped, and the candidate VM had no ISO attached.
Before this PR, the ISO was attached before the state change.
- A stop while a scale-up was running was refused with the same "does not
allow StopRequested" message. The scale completed.
**S3a: management server killed during an upgrade**
- I upgraded v1.32.5 to v1.33.1, and killed the management server once the
control node reported v1.33.1. While Upgrading, a scale was refused ("is in
Upgrading state and can not be scaled").
- After the restart:
```
Alert after 296s (the scanner's first pass)
Running after 32s (the existing Alert reconciliation)
WARN Kubernetes cluster cluster-b was in state Upgrading with no operation
in progress, as its operation ended without moving it on, for example because
the management server running it stopped; marked the operation as failed
```
The upgrade job was marked failed.
- The cluster was left as the killed job had left it: the control node on
v1.33.1 and cordoned, the worker on v1.32.5, and the cluster's version still
v1.32.5.
- **Main-only finding.** Rerunning the upgrade failed at once:
```
Failed to attach binaries ISO for VM : cluster-b-control-... :
InvalidParameterValueException: The specified ISO is already attached to this
Instance.
```
- The killed job had left the v1.33.1 ISO on both VMs. The description
says "The next operation's attach replaces it", which is true on 4.22.
- On main, `TemplateManagerImpl.enforceCdromAttachLimits` (added by
#13101, multiple CD-ROMs per VM) refuses an ISO that is already attached.
- I've written this up separately as a main-only stock bug, with a
proposed fix in `attachIsoKubernetesVMs`. It will matter when this PR is
forward-merged.
- After I detached the ISO from both VMs, the same upgrade completed in 116
s. Both nodes were upgraded, the control node again, because its version hadn't
been recorded before the kill. Both were uncordoned, and the cluster was
Running on v1.33.1.
**S3b: management server killed during a scale-up**
- I started `size=2` and killed the management server 20 s after the new VM
appeared in the cluster.
- After the restart:
```
WARN Kubernetes cluster cluster-b was in state Scaling with no operation
in progress, ... marked the operation as failed
Scaling -> Running after 296 s (Scaling -OperationFailed-> Running is the
existing transition)
```
- **Leftovers** (not something the PR claims to repair):
- **An unjoined VM.** The new worker VM was left in the cluster, Running
but never a Kubernetes node, with the size still 1. A stock scale-up that fails
also leaves its VMs.
- **No SSH rules.** The kill came while the scale was rebuilding the
cluster's network rules, which it does by revoking all port forwards and adding
them again. The cluster was left with **no SSH port-forward rules**, and the
SSH firewall rule was stuck in `Add`.
- **The next start hung.** S4's start retried `kubeconfig` over SSH and
would have run until the 3600 s start timeout. I recreated the firewall rule
and the three port forwards by hand, in VM-map order, as the scale would have,
and the start then completed.
- **Removing the unjoined VM.** I stopped it and tried to remove it with
`nodeids`. #14604 refused: "has 3 VMs for 2 nodes; recover the cluster before
scaling it". I set `node_count` to 2 in the database, to match the 2 worker
VMs. A `size=1` scale in Alert then removed the stopped VM with no drain, and
the cluster was Running with 1 control node and 1 worker. I also corrected the
cluster's cores and memory totals, which my `node_count` edit had thrown off.
- I suggest the stuck-state recovery could at least log what an interrupted
Scaling may have left (VMs not in the node count, rules being rebuilt).
Repairing it is a separate change.
**S4: starting a cluster in Alert**
- I stopped a worker VM, and the cluster was in Alert after 21 s. I then
sent `startKubernetesCluster`.
- States sampled every second, through the S3b rule repair above:
```
Alert -> Starting (about 200 samples) -> Running (once)
```
The state didn't change during the start, and no other operation or
scanner pass took the cluster midway.
**Result:** pass, with the main-only ISO issue and the S3b leftovers noted
above.
- Concurrent operations are refused with clear messages.
- Operations from states with no transition are refused before they change
anything.
- A cluster stuck in Upgrading or Scaling after a management server stop is
recovered on the scanner's next pass.
**Not covered**
- More than one management server (a peer seeing the server leave), and the
mixed-version job-parameter fallback.
- Recovery from Starting, Stopping, Importing, RemovingNodes or Recovering.
Only Upgrading and Scaling were interrupted.
- The scale of a Created cluster racing a start.
- The Alert-to-Stopped reconciliation racing a user start.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
