bhouse-nexthop opened a new issue, #14603: URL: https://github.com/apache/cloudstack/issues/14603
### problem When any node VM of a running Kubernetes cluster isn't Running (for example one worker VM stopped, or a VM a host failure left stopped), the CKS status scanner moves the cluster to `Alert`. `scaleKubernetesCluster` only accepts `Created`, `Running` or `Stopped`, so it refuses every request with: ``` Kubernetes cluster <name> is in Alert state and can not be scaled ``` The CloudStack cluster-autoscaler provider scales through the same API (`size` to add nodes, `nodeids` to remove them), so autoscaling stops too. A single stopped node therefore blocks adding capacity, and also blocks removing that broken node with `scaleKubernetesCluster nodeids=<vm>`. This is when scaling is most needed. The state machine matches: `Alert` already allows `AddNodeRequested`, `RemoveNodeRequested`, `RecoveryRequested`, `StopRequested` and `DestroyRequested`, but has no `ScaleUpRequested` or `ScaleDownRequested` transition. Even with the state check removed, two more things get in the way: - After a scale-up, the worker waits until the number of Ready nodes equals the cluster's total node count. A stopped node is never Ready, so the scale-up always times out. - Removing a node runs `kubectl drain` on it first. On a stopped VM's NotReady node the drain waits for pod deletions that the node can never confirm, so removing the stopped node fails. ### versions Seen on 4.22 with KVM. The code is the same on `main`. ### The steps to reproduce the bug 1. Create a CKS cluster with 2 or more workers and wait for it to be Running. 2. Stop one worker VM. The cluster goes to Alert. 3. `scaleKubernetesCluster id=<cluster> size=<current+1>`: refused, "is in Alert state and can not be scaled". 4. `scaleKubernetesCluster id=<cluster> nodeids=<stopped worker VM>`: refused the same way. ### What to do about it? Allow changing the size and removing worker nodes by `nodeids` while the cluster is in Alert: - add `Alert -> Scaling` for `ScaleUpRequested` and `ScaleDownRequested`. `Scaling -> Running` on success and `Scaling -> Alert` on failure stay as they are, and the scanner moves the cluster back to Alert if a node is still down; - after a scale-up, expect only the nodes whose VMs aren't down to be Ready; - on a scale-down by size, remove nodes whose VMs are down first; - don't drain a node whose VM is down; delete the Kubernetes node object directly. Service offering and autoscaling changes act on every node, so they stay refused in Alert, with a message saying to recover the cluster first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
