bhouse-nexthop commented on PR #14611:
URL: https://github.com/apache/cloudstack/pull/14611#issuecomment-6746839456
### Test report: CKS node drain on scale-down and upgrade, and resuming a
failed upgrade
**Environment**
- A CloudStack 24.0 (main) build with this PR, alongside some of my other
open PRs, among them #14604 and #14609.
- The PR targets 4.22; it wasn't run on a 4.22 build. The scale and upgrade
workers and the node scripts it changes are the same on both branches.
- Three KVM hosts, Ceph RBD primary storage, one management server.
- Two CKS clusters on isolated networks:
- A: 3 control nodes and 1-3 workers;
- C: 1 control node and 2 workers.
Both went from Kubernetes v1.32.5 to v1.33.1 in the upgrade tests.
- `cloud.kubernetes.cluster.node.drain.timeout=60`, set for the test
(default 300). `cloud.kubernetes.cluster.upgrade.retries=3`.
- The order of commands on the cluster comes from the first control node's
sudo journal (`COMMAND=/opt/bin/kubectl ...`). The management server doesn't
log a cordon or drain that succeeds.
**Results**
| Scenario | Result |
|---|---|
| S1: scale-down blocked by a PodDisruptionBudget | pass: rolled back,
nothing removed, an operator's cordon kept, back to Running |
| S2: scale-down with the PDB relaxed | pass: every node cordoned first,
then drained one by one, then deleted |
| S3: upgrade with VM order and SSH port order different | pass: each node's
script ran on that node; all 5 nodes on v1.33.1 |
| S3b: "upgrade" to the version the cluster already has | pass (unplanned):
every node skipped, nothing run |
| S4a: upgrade fails part way (a worker stopped) | pass: Alert, ISO detached
first, target version recorded, node uncordoned |
| S4b: refusals in Alert | pass: VMs not all running; a different version |
| S4c: retry from Alert | pass: finished nodes skipped, the rest upgraded,
Running on v1.33.1, detail removed |
**S1: drain blocked by a PodDisruptionBudget**
- Setup:
- Cluster A was scaled to 3 workers, with a 4-replica `web` deployment
spread over them and a PDB with `minAvailable=4`.
- I cordoned one worker (C) by hand, as an operator would.
- I then scaled to `size=1`, which removes workers B and C.
- Order of commands on the cluster:
```
14:52:37 kubectl cordon C
14:52:37 kubectl cordon B
14:52:38 kubectl drain C --ignore-daemonsets --delete-emptydir-data
--timeout=60s
14:54:08 kubectl drain C ... --timeout=60s
14:55:39 kubectl drain C ... --timeout=60s
14:56:39 kubectl uncordon B
```
That is three attempts, each 60 s plus a 30 s wait. Only B was uncordoned.
C, which was already cordoned, stayed cordoned.
- The API error names the node and ends with kubectl's output:
```
Scaling failed for Kubernetes cluster ..., unable to drain Kubernetes
node: <C> on VM: ...: ... Cannot evict pod as it would violate the pod's
disruption budget.
...
error: unable to drain node "<C>" due to error: ... global timeout
reached: 1m0s, continuing command...
. No node was removed
```
- The cluster was Running with size 3, all three workers in Kubernetes, B
schedulable and C still cordoned. The call took 244 s.
- Nit: the 2000-character tail of kubectl's output starts mid-line
(`4f6ccb-fjf66" -n "default" (will retry after 5s): ...`). Starting it at the
next newline would read better.
**S2: the same scale-down, with the PDB relaxed**
- I set `minAvailable=2`, uncordoned C, and scaled to `size=1` again:
```
14:56:43 kubectl cordon C
14:56:44 kubectl cordon B
14:56:44 kubectl drain C ... --timeout=60s
14:56:46 kubectl drain B ... --timeout=60s
14:56:48 kubectl delete node C --ignore-not-found
14:56:59 kubectl delete node B --ignore-not-found
```
- The cluster was Running with size 1. The deployment rolled out again on
the remaining nodes.
**S3: per-node SSH address during the upgrade**
- Isolated networks rebuild the SSH port forwards in VM order on every
scale. To make the orders differ, I swapped the rules of the third control node
and the worker by hand:
```
VM order: control-1 control-2 control-3 worker-1 worker-2
port: 2222 2223 2225 2224 2226
```
The old code would have run the script for VM *i* on port 2222 + *i*. That
is control-3's script on worker-1 and the reverse.
- Upgrade v1.32.5 to v1.33.1: OK in 657 s.
```
Upgrading node control-1 ... Successfully upgraded (15:01:18 - 15:05:21)
Upgrading node control-2 ... Successfully upgraded (- 15:07:53)
Upgrading node control-3 ... Successfully upgraded (- 15:10:08)
Upgrading node worker-1 ... Successfully upgraded (- 15:11:11)
Upgrading node worker-2 ... Successfully upgraded (- 15:12:03)
```
- Every node had management server SSH logins during its own step. The first
control node had 33 logins, because kubectl runs there. Every kubelet reports
v1.33.1, which needs each node's own script to have run on it. All nodes were
Ready and uncordoned. The cluster was Running on v1.33.1, and no
`upgradeKubernetesVersionId` detail was left.
- **S3b (unplanned).** My first attempt looked up the target version with a
keyword filter, which picked v1.32.5, the cluster's own version. Stock only
refuses a downgrade, so the request was accepted. Every node was found already
at the target (recorded and in `kubectl`, and Ready), and skipped:
```
Skipping node control-1 / control-2 / control-3 / worker-1
```
It returned in 9 s and nothing was run.
**S4: a failed upgrade, then a retry from Alert (cluster C)**
- **S4a.** I started the upgrade to v1.33.1 and force-stopped the last
worker once the first two nodes were on v1.33.1.
```
Upgrading node control ... Successfully upgraded
Upgrading node worker-1 ... Successfully upgraded
Upgrading node worker-2
Draining node: worker-2 16:01:43 / 16:03:14 / 16:04:44 / 16:06:15
(retries + 1 attempts)
Detached Kubernetes binaries from VM ... (all three) 16:06:16
-> "Failed to upgrade Kubernetes cluster : cluster-c, unable to drain
Kubernetes node : worker-2 ..."
```
- The cluster went to Alert, still on v1.32.5. The ISO was detached before
that.
- `upgradeKubernetesVersionId` was set to the v1.33.1 version's id.
- worker-2 was uncordoned, because the upgrade had cordoned it.
- **S4b: refusals in Alert.**
```
upgrade to v1.33.1 with worker-2 stopped:
"Kubernetes cluster : cluster-c is in Alert state and not all of its VMs
are running. Start them before retrying the upgrade"
(start worker-2)
upgrade to v1.33.13 (a different version):
"Kubernetes cluster : cluster-c is in Alert state. It can be upgraded in
that state only to retry an upgrade that failed, to the same Kubernetes version
as that upgrade"
```
- **S4c: the retry to v1.33.1, OK in 112 s.**
```
Skipping node control
Skipping node worker-1
Upgrading node worker-2 ... Successfully upgraded
```
- The skipped nodes ran no upgrade script. The skipped worker had no
management server login, and the control node's logins were the `kubectl`
checks.
- The cluster was Running on v1.33.1, with all nodes Ready on v1.33.1 and
uncordoned. The detail was removed.
**Result:** pass.
- Scale-down cordons all nodes first and rolls back cleanly, naming the
node, when a drain can't finish. An operator's cordon is kept.
- The upgrade reaches each node through its own port forward.
- A failed upgrade can be retried from Alert, only to the same version and
only with all VMs running, and it skips the nodes that are already done.
**Not covered**
- The default drain timeout (300 s), and the timeout being cut short near
the operation's deadline.
- A scale-down that times out after the drain, and a failure deleting a node
or destroying its VM (the Alert path).
- An upgrade that failed before this change, with no detail (the
node-version fallback).
- Shared networks, where nodes are reached on their own address.
- The UI's Upgrade action for a cluster in Alert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
