bhouse-nexthop commented on PR #14604:
URL: https://github.com/apache/cloudstack/pull/14604#issuecomment-6748473958
### Test report: scaling a Kubernetes cluster that is in Alert, on a live
cluster
**Summary:** with the second commit ("Wait for each new node by name before
a scale-up detaches the ISO"), every scenario passes. That includes the case
that failed in my first run: the ready-node count is met by a stopped node that
Kubernetes still reports as Ready. The scale-up now holds the binaries ISO
until the new node is Ready by name, and then succeeds. Scale-down, `nodeids`
removal and the refusals behave as in the first run. A normal scale-up
afterwards completes without waiting on anything.
**Environment**
- A CloudStack 24.0 (main) build with this PR, both commits, alongside some
of my other open PRs. Among them are #14609 and #14611, which touch the same
workers.
- The PR targets 4.22; it wasn't run on a 4.22 build. The code it changes is
the same on both branches: the scale worker, `KubernetesClusterUtil` and the
scale validation in `KubernetesClusterManagerImpl`. That includes
`isKubernetesClusterNodeReady`, which the fix reuses from the upgrade worker.
- Three KVM hosts, Ceph RBD primary storage, one management server.
- CKS cluster A: 3 control nodes and 2 workers, Kubernetes v1.33.1, isolated
network.
- Scale timeout at its default (3600 s).
`cloud.kubernetes.cluster.node.drain.timeout=60` during the run, put back
afterwards.
- Kubernetes marked a stopped worker's node `Unknown` about 50 s after its
VM stopped.
**Results**
| Scenario | Result |
|---|---|
| S1: stop a worker VM; the scanner moves the cluster to Alert | pass (Alert
after 0-24 s, over 6 runs) |
| S2: `size` 2 -> 3 in Alert, one worker stopped, scale started as soon as
the cluster is in Alert | pass, 4 runs. The ISO was detached 7-15 s after the
new node was Ready. These runs didn't reach the stale-Ready window (see below) |
| S2, inside the stale-Ready window: the count is met by a stopped node that
is still Ready | **pass**. The count matched 45 s after the stop. The worker
then waited 1 min 50 s for the new node by name, and detached the ISO 15 s
after it was Ready. The scale-up succeeded |
| Follow-up: a normal scale-up in Running after S2 | pass, 133 s (twice) |
| S3: `size` 3 -> 2 in Alert: the stopped worker goes first, and isn't
drained | pass (0 drain lines, node object removed, back to Running). Also
`size` 3 -> 1 with two stopped workers: both removed, no drain |
| S4: stop the other worker, then remove it with `nodeids` in Alert | pass
(0 drain lines, removed, back to Running) |
| S5a-e: refusals in Alert (offering, autoscaling, same size, a control node
in `nodeids`, a control VM down) | pass, same messages as the first run |
| S5: start the stopped worker; the scanner moves the cluster back to
Running | pass (Running after 21 s, 5 nodes Ready) |
**S2: the first run's failure, and what changed**
- **First run (first commit only).** I stopped a worker, and the cluster was
in Alert 6 s later. I then scaled to `size=3`.
- The scale worker's first ready-node check came 43 s after the stop.
Kubernetes still reported the stopped node as Ready at that point.
- So the 4 running nodes plus the stopped one made the expected 5 before
the new node had booted.
- The scale-up reported success and detached the ISO 0.6 s after attaching
it. The new node never joined, and the next scale-up waited on it until I
attached the ISO by hand.
- **The change.** Once the count matches, the worker also waits for each new
node to be Ready by name, within the same scale timeout, before it detaches the
ISO. The name is the lower-cased hostname, checked with
`isKubernetesClusterNodeReady`.
- **Reaching the window again.** I first repeated S2 exactly as before,
starting the scale the moment the cluster went to Alert (polled every second).
- In these runs the cluster reached Alert 17-24 s after the stop, and
provisioning the new VM took about 37 s.
- So the first count ran 58-63 s after the stop, after Kubernetes had
already marked the node `Unknown`. The count never matched early, and these
runs didn't reach the window.
- They do show the normal path on the new build. The ISO stayed attached
until the new node was Ready, and was detached 7-15 s after.
- **Deterministic variant.**
- With the cluster already in Alert (worker 1 stopped, its node
`Unknown`), I stopped worker 2 and immediately scaled to `size=3`. Worker 2
stays Ready in Kubernetes for about 50 s, while its VM is down.
- The expected count is 6 provisioned nodes minus 2 down = 4. The 3
control nodes plus the stale worker 2 make 4 before the new node exists. That
is the same early match as in the first run.
- Timeline (worker 2 stopped at 18:39:31; node samples every 2-3 s):
```
18:39:39 scaleKubernetesCluster size=3 sent; cluster in Alert
18:40:15,874 INFO KubernetesClusterScaleWorker Attached binaries ISO
for VM: <new worker> in cluster: cluster A
18:40:15,877 DEBUG KubernetesClusterUtil Checking ready nodes for the
Kubernetes cluster cluster A with total 6 provisioned nodes, 4 expected to be
ready
18:40:16,334 INFO KubernetesClusterUtil Kubernetes cluster cluster A
has 4 ready nodes now
(k8s at 18:40:14: worker 1 Unknown, worker 2 True, new
worker not registered) <- the early match
18:40:16,686 DEBUG KubernetesClusterUtil Failed to retrieve status for
node: <new worker> in Kubernetes cluster: cluster A. Output:
... the same line every 15 s, 8 times in all ...
18:42:04,902 DEBUG KubernetesClusterUtil Failed to retrieve status for
node: <new worker> ...
18:42:06 new worker Ready (lastTransitionTime)
18:42:21,365 INFO KubernetesClusterScaleWorker Detached Kubernetes
binaries from VM: <new worker> in the Kubernetes cluster: cluster A
18:42:22 API returns success (163 s)
```
- Before the fix, the worker would have detached the ISO at 18:40:16, as
it did in the first run.
- Node and ISO samples around the join:
```
18:40:19 worker 1=Unknown worker 2=True | ISO on:
new worker
18:40:27 worker 1=Unknown worker 2=Unknown | ISO on:
new worker
18:42:04 worker 1=Unknown worker 2=Unknown new worker=False | ISO on:
new worker
18:42:16 worker 1=Unknown worker 2=Unknown new worker=True | ISO on:
new worker
18:42:28 worker 1=Unknown worker 2=Unknown new worker=True | ISO on:
(none)
```
- `kubectl get nodes` when the API returned:
```
control-1 Ready control-plane 24h v1.33.1
control-2 Ready control-plane 24h v1.33.1
control-3 Ready control-plane 24h v1.33.1
worker-1 NotReady <none> 21m v1.33.1
worker-2 NotReady <none> 5m37s v1.33.1
worker-3 Ready <none> 23s v1.33.1 <- new
```
- The cluster stayed in Alert, as both stopped workers were still in it.
- `size=1` then removed both stopped workers in 28 s, with no drain, and
the cluster went back to Running.
- A normal `size=2` scale-up in Running then took 133 s. The ISO was
detached 2.6 s after the count matched, with the new node already Ready.
**S3, S4, S5 on the new build**
- **S3:** `size=2` with one worker stopped removed that worker first, in 19
s. There were 0 drain lines for it, its node object was deleted, and the
cluster was Running.
- **S4:** `nodeids=<stopped worker>` in Alert removed it in 19 s, with 0
drain lines. The cluster was Running, and a `size=2` scale-up afterwards took
133 s. In the first run, that scale-up waited on the stranded node.
- **S5a-e** were all refused, with the messages from the first run:
```
a, b) "... is in Alert state; only its size can be changed or nodes
removed. Recover the cluster before changing its service offering or
autoscaling"
c) "... is in Alert state; only its size can be changed or nodes
removed, and the request does neither"
d) "... is in Alert state; only worker nodes can be removed from it"
e) "... is in Alert state and a control or etcd node VM is down;
recover the control plane before scaling the cluster"
```
**Not caused by this PR**
- My first attempt on the new build failed at once with an NPE, `Cannot
invoke "VirtualMachineTemplate.getUniqueName()" because "template" is null`, in
`createKubernetesNode`.
- The cluster was created before 4.21. Its per-node-type template columns,
added by #9102, are still NULL after the upgrade.
- I've written that up separately as a stock bug. For this run I set the
cluster's control and worker template ids to its template, as a cluster created
on 4.21+ would have them, and put them back afterwards.
**Result:** pass. The fix closes the failure from my first run. A scale-up
whose ready count is met early by a stale node now waits for the new node by
name, keeps the ISO attached until the node has joined, and succeeds. No node
is left stranded for the next scale-up. Everything else in the PR behaves as in
the first run.
**Not covered**
- The timeout path of the by-name wait: a new node that never becomes Ready,
so the scale-up fails and the cluster goes to Alert. This is covered by the
unit test only.
- The UI's Scale form for a cluster in Alert.
- A VM in `Unknown` (host disconnected), which the PR treats as not down.
- The cluster-autoscaler driving the API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
