bhouse-nexthop commented on PR #14616:
URL: https://github.com/apache/cloudstack/pull/14616#issuecomment-6746840215
### Test report: CKS node settings and rolling node restarts, on a live
cluster
**Environment**
- A CloudStack 24.0 (main) build with this PR, alongside some of my other
open PRs, including #14609 and #14611, which this PR depends on, and #14604.
- Three KVM hosts, Ceph RBD primary storage, one management server.
- Two CKS clusters, Kubernetes v1.33.1, isolated networks:
- A: 3 stacked control nodes, 2-3 workers;
- C: 1 control node, 2 workers.
Both were created on 4.20 with one cluster offering, so the per-type
settings start empty.
- Offerings: `cks-node` (2 vCPU, 4 GB) and a test offering `k-4c4g` (4 vCPU,
4 GB). Worker root disks were 20 GB.
- `cloud.kubernetes.cluster.node.drain.timeout=60` for the test.
`cloud.kubernetes.cluster.update.node.timeout` was left at its default.
- A watcher sampled the cordoned, NotReady and annotated nodes and the VM
states every 3 s.
**Results**
| Scenario | Result |
|---|---|
| S1: store a worker offering only (`rollout=false`) | pass: no node
touched, out-of-date count right, a later scale-up gets the new offering |
| S7: worker rollout blocked by a PDB | pass: stops with `lasterror`,
nothing restarted, nothing left cordoned, Running |
| S2+S3: worker offering + root disk 20 -> 30 GB, `maxunavailable=2` | pass:
at most 2 cordoned, new boot IDs, disks grown in the guest, capacity recounted |
| S3b: root disk shrink with `rollout=true` | pass: refused, naming the node
|
| S11: other operations while Updating (scale, upgrade, stop, remove nodes,
store) | pass: all refused |
| S4: strict host anti-affinity for workers, applied on restart | pass:
three workers on three hosts, all in the group |
| S5: control offering rollout on a 3-node stacked control plane | pass: one
at a time, never more than one NotReady, etcd ready throughout |
| S5b: control rollout with one control VM already stopped | pass: the
stopped node goes first, then the others one at a time |
| S9: `cancelrollout` | pass: stops after the batch in progress;
`rollout=true` resumes |
| S8: management server killed mid worker rollout, with a worker stopped |
pass: Updating -> Alert and capacity recounted; resume takes the stopped node
first |
| S10: a single control node | pass: `rollout=true` refused for control; a
worker rollout skips it and says so |
| S6: external etcd | not run (capacity) |
**S1: store only, then a scale-up**
- I set the worker offering to `k-4c4g` with no rollout:
```
response:
{"state":"Running","controlnodesoutofdate":0,"workernodesoutofdate":2,...}
db: service_offering_id=cks-node, control=cks-node, worker=k-4c4g
(control copied from the cluster offering)
```
The nodes' boot IDs didn't change.
- A scale to `size=3` created the new worker with `k-4c4g`.
- Right after that scale, the cluster reported 12 cores, but the VMs add up
to 14. The scale's capacity update uses the cluster offering for the new worker
(stock `calculateNewClusterCountAndCapacity`). The next rollout's recount
corrected it to 14.
**S7: a PDB that allows no eviction**
- Setup: a 4-replica deployment, `minAvailable=4`, and 3 workers.
- `maxunavailable=5` was refused:
```
"maxunavailable must be 1, or more and less than the 3 worker nodes of
Kubernetes cluster cluster-a, for some to keep running the workloads"
```
- With `maxunavailable=2`, the first batch was the two out-of-date workers,
cordoned together. The drain of the first one failed 3 times, 90 s apart:
```
FAILED: "The rollout of Kubernetes cluster cluster-a stopped: unable to
drain node <worker-1>: ... Cannot evict pod as it would violate the pod's
disruption budget ..."
```
- The cluster stayed Running with `workernodesoutofdate=2` and `lasterror`
set.
- Both workers were uncordoned, and no VM changed state or boot ID.
- The recount set cores to 14.
**S2 + S3: offering and root disk, two at a time**
- With the PDB relaxed to `minAvailable=1`, `noderootdisksizes worker=10`
was refused:
```
"The root disk of node <worker-1> is larger than 10 GB, and a rollout only
grows root disks. Use rollout=false to only store the size for new nodes"
```
- Then `rollout=true maxunavailable=2` with the worker root disk at 30 GB:
```
Restarting worker node(s) [worker-1, worker-2]
Restarting worker node(s) [worker-3] (already k-4c4g; its disk was
out of date)
Rollout of Kubernetes cluster cluster-a completed (167 s)
```
- At most 2 nodes were cordoned at once. Each carried the
`cloudstack.apache.org/rollout-boot-id` annotation with its old boot ID while
it was out.
- All three workers came back with new boot IDs, and the control nodes'
boot IDs didn't change.
- Each worker had `/dev/vda` at 30G and `/` at 29G, grown by `growpart`
and `resize2fs` at boot.
- The cluster was Running with `workernodesoutofdate=0`. Cores were
recounted to 18 (3 x 2 + 3 x 4).
- **S11.** While that rollout was Updating:
```
scale size=4: "Kubernetes cluster cluster-a is in Updating state and
can not be scaled"
upgrade: "Kubernetes cluster : cluster-a is not in running state"
stop: "Kubernetes cluster cluster-a is in state Updating,
which does not allow StopRequested"
remove nodes: "... is in state Updating, which does not allow
RemoveNodeRequested"
store (no rollout): "The node settings of Kubernetes cluster cluster-a can
not be changed while it is Updating"
```
**S4: affinity groups applied on restart**
- I migrated a worker so that two of the three shared a host. I then set the
worker type's group to a new strict host anti-affinity group, with
`rollout=true`.
- Each worker was restarted in turn. Afterwards the three workers were on
three different hosts, each in the group. The type's group is recorded (`WORKER
-> k-gw`).
**S5: control nodes one at a time (3 stacked)**
- Before: `/health` on port 2381 was `true` on all three, and `/readyz/etcd`
was `ok`.
- The control offering rollout to `k-4c4g` took 506 s:
```
Restarting control node(s) [control-1] -> [control-2] -> [control-3]
```
- All three got new boot IDs, and no worker restarted.
- At most one node was NotReady and at most one cordoned at any sample.
`/readyz/etcd` was `ok` afterwards.
- While control-1 was down (16:25:03-16:27:50), the management server's
SSH sessions went to control-2 (3 logins) and none to control-1. It was working
through another control node.
- Cores were recounted to 24.
- **S5b.** I stopped control-3 by hand, then rolled the control nodes back
to `cks-node`. I expected a refusal, but the description says that stopped
nodes go first, and that's what happened:
```
Restarting control node(s) [control-3] (the stopped one, started with
the new offering)
Restarting control node(s) [control-1]
Restarting control node(s) [control-2]
```
Never more than one control node was down. The cluster ended Running.
**S9: cancel**
- I started a control rollout to `k-4c4g`, and called `cancelrollout=true`
once the first node was cordoned:
```
cancelrollout: OK
cancelrollout + rollout=true: "cancelrollout takes no other parameter"
rollout: FAILED: "The rollout of Kubernetes cluster cluster-a stopped: it
was cancelled. Call updateKubernetesCluster with rollout=true to resume it"
```
- Exactly one control node was restarted.
- The cluster was Running with `controlnodesoutofdate=2`, `lasterror="it
was cancelled"`, and cores 20, which matches the mixed offerings.
- `rollout=true` with no other parameter finished the other two, leaving
`controlnodesoutofdate=0`.
**S8: management server killed during a worker rollout**
- I rolled the workers back to `cks-node`. 18 s in, the first worker was
Stopped with its new offering, and I killed the management server (`kill -9`,
then a start).
- On the scanner's first pass:
```
WARN Kubernetes cluster cluster-a was in state Updating with no operation
in progress, ... marked the operation as failed
```
- The cluster went to Alert, with capacity recounted from the VMs to 22.
- The stopped worker stayed stopped, cordoned, and still annotated.
Nothing reached into the nodes, as described.
- `rollout=true` again:
```
Restarting worker node(s) [worker-1] (the stopped, annotated one, first)
Restarting worker node(s) [worker-2]
Restarting worker node(s) [worker-3]
```
The cluster ended Running, all workers on `cks-node`, all nodes Ready and
uncordoned, and cores 18.
**S10: a single control node (cluster C)**
- A control offering with `rollout=true` was refused:
```
"The 1 control node(s) of Kubernetes cluster cluster-c can not be
restarted one at a time, as the cluster can not keep running without one, so a
rollout can not change their settings. Use rollout=false to store them without
restarting the nodes"
```
- The same with `rollout=false` was stored (`controlnodesoutofdate=1`).
- A worker offering rollout then restarted both workers and skipped the
control node:
```
Not restarting the control nodes of Kubernetes cluster cluster-c: the 1
control node(s) were not restarted, as a cluster with 1 of them can not keep
running without one
Rollout of Kubernetes cluster cluster-c completed, but the 1 control
node(s) were not restarted, ...
```
The control node's boot ID didn't change, and the cluster was Running with
that `lasterror`.
**Nits on the description**
- "This is clamped to `min(maxunavailable, workers - 1)`": the API refuses a
value at or above the number of workers (S7), so the clamp only matters inside
a rollout. The description could say that the API refuses it.
- "`lasterror`: ... which control or etcd nodes it could not restart": the
message gives the count ("the 1 control node(s)"), not the names. One or the
other could be changed.
**Result:** pass.
- Settings can be stored without touching nodes.
- Rollouts restart nodes in batches within the limits described: one control
node at a time, `maxunavailable` workers, stopped nodes first. They refuse what
would lose the control plane, and stop cleanly on a blocked drain, a cancel, or
a management server stop.
- Capacity is recounted whenever a rollout ends.
**Not covered**
- External etcd (S6): no capacity for etcd VMs in this lab.
- `nodeaffinitygroups` with an explicit dedication group (refusal), and a
disk offering with a strictly fixed size.
- `cloud.kubernetes.cluster.update.node.timeout` expiring.
- A 2-node stacked control plane.
- The UI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
