bhouse-nexthop opened a new pull request, #14604:
URL: https://github.com/apache/cloudstack/pull/14604
### Description
A Kubernetes cluster in `Alert` can now change its size, and have worker
nodes removed with `nodeids`. Previously `scaleKubernetesCluster` refused any
request in Alert. So one stopped node VM blocked scaling, and blocked the
CloudStack cluster-autoscaler, which scales through the same API. It also
blocked removing that stopped node.
Fixes #14603
#### What changes
- **State check**
(`KubernetesClusterManagerImpl.validateKubernetesClusterStateForScale`, moved
out of `validateKubernetesClusterScaleParameters`). In Alert it accepts a size
change, or removing worker nodes by `nodeids`. It refuses:
- a service offering or autoscaling change, since both act on every node,
including the one that is down. Sending the offerings and autoscaling setting
the cluster already has is accepted;
- a request that changes neither the size nor the nodes. Such a request
would do nothing and then move the cluster from Alert to Running;
- removing control, etcd or external nodes by `nodeids`. Scaling never
acts on external nodes; they are removed with
`removeVirtualMachinesFromKubernetesCluster`;
- any request while the first control node VM isn't running. Scaling
reaches the cluster over SSH to that node;
- any request while the cluster has more or fewer VMs than nodes, for
example VMs left by a failed scale-up, which a scale-up would only add to.
- any request while a control or etcd node VM is down. etcd may then have
lost quorum, and new nodes can't join, so the control plane has to be recovered
first;
- any request while none of the cluster's worker VMs is down (see "Down"
below), leaving out external nodes, which scaling doesn't act on. The cluster
is then in Alert for another reason that scaling won't fix, such as a failed
start or upgrade, or a VM on a disconnected host.
- **What the worker gets in Alert:**
- No offerings, so it only changes the size, and can't apply the cluster
offering to a node type whose VMs use another one.
- The cluster record that was validated is passed on as is, rather than
read again.
- The worker fails the request if the cluster's state has changed since it
was validated, for example to Alert, or because another operation has started.
- **State machine** (`KubernetesCluster`): `Alert -> Scaling` on
`ScaleUpRequested` and `ScaleDownRequested`. `Scaling -> Running` on success
and `Scaling -> Alert` on failure are unchanged, so the status scanner puts the
cluster back in Alert if a node is still down.
- **Scale-up** (`KubernetesClusterScaleWorker`):
- After provisioning, the worker waits for the nodes whose VMs aren't down
to be Ready. Before, it waited for every node, and a stopped node, which is
never Ready, made every scale-up time out.
- The new nodes must all be Ready. The expected count is read again on
every check, in case a VM is started or stopped meanwhile. This goes through a
new `KubernetesClusterUtil.validateKubernetesClusterReadyNodesCount` overload;
the existing signature behaves as before.
- The capacity check plans only the new VMs in Alert, as it does in
Running. Before, it planned capacity for the whole cluster again.
- **Scale-down by size:** workers whose VMs are down are removed first.
Otherwise the order is unchanged.
- **Removing a node whose VM is down:** the worker no longer runs `kubectl
drain`, and deletes the Kubernetes node object directly.
- The drain waits for pod deletions that a NotReady node can never
confirm, so it failed, and the scale-down with it.
- A down VM runs no pods. Once its node object is deleted, Kubernetes
garbage-collects the pods and their controllers recreate them elsewhere.
- **"Down"** means Stopped, Shutdown, Error, Destroyed or Expunging.
Migrating, Unknown (a disconnected host) and the other states may still be
running the VM, so those nodes are treated as before.
- **UI:** the Scale action is shown for a cluster in Alert that doesn't
autoscale. The dialog then shows only the size field, and sends only the size.
An autoscaling cluster in Alert is scaled by the cluster-autoscaler itself. The
per-node delete button in the cluster's instances tab is unchanged, so removing
a specific node in Alert is done through the API (`nodeids`) for now.
Running, Created and Stopped clusters behave as before. The exception is a
cluster that is still Running while one of its VMs is already down, before the
status scanner has moved it to Alert. There, the down node is now removed
first, isn't drained, and isn't expected to become Ready.
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [x] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [x] Major
- [ ] Minor
- [ ] Trivial
### How Has This Been Tested?
- Unit tests, all in `plugins/integrations/kubernetes-service` (`mvn -pl
plugins/integrations/kubernetes-service -am test
-Dtest='KubernetesCluster*Test'`), pass. Checkstyle on `api` and the plugin is
clean.
- New tests in `KubernetesClusterManagerImplTest`:
- accepted in Alert: a size change; `nodeids` for a worker; a size change
sent with the cluster's current offerings and autoscaling setting;
- refused in Alert: a cluster or node-type offering change; an autoscaling
change, including a different minsize/maxsize; a request with no size or node
change; a request with the current size; removing a control node; the first
control node VM stopped; more VMs than nodes; no worker VM down (all running,
or only Unknown); an etcd VM down; only an external node down;
- an offering change is still accepted in Running, and Upgrading is still
refused.
- New tests in `KubernetesClusterScaleWorkerTest`: a stopped worker is
picked first for a scale-down by size; the down-node count leaves out etcd
nodes and newly provisioned VMs; which VM states count as down.
- **Not tested on a live cluster.** I have not run a scale-up, a scale-down
or a `nodeids` removal against a real CKS cluster in Alert. The drain skip, the
ready-node wait and the UI change are covered only by reading the code and the
unit tests above. The UI change was not built or linted locally.
#### How did you try to break this feature and the system with this change?
- **Other states:** for Created, Running and Stopped clusters, the state
check, the offering map and the existing ready-node check are unchanged. The
only difference is the down-VM handling for a Running cluster with a down VM,
as described above.
- **Restarts during a scale-up:** the expected Ready count is read again on
every check. A stopped VM that HA or a user starts again is then expected to
rejoin, and the scale-up can't report success before the new nodes are Ready.
- **Hosts disconnected or VMs migrating:** Unknown and Migrating don't count
as down, so those nodes are still drained, and still expected to be Ready.
- **Concurrent requests:** a second scale request is refused at validation
once the first has moved the cluster to Scaling. If both are validated before
that, the second now fails when its worker starts, because the state has
changed. Before, it ran alongside the first. The check and the transition are
still separate steps, so a very narrow window remains.
- **Not covered here:**
- Removing control nodes by `nodeids` in Running records the wrong worker
count. That is unchanged by this PR, and refused in Alert.
- `removeKubernetesClusterNode` increments its retry counter twice per
pass, so it makes fewer attempts than intended.
- A request that sends only the cluster offering in Running still
re-applies it to node types whose VMs use another offering.
- A worker count change without a new offering records the cluster's cores
and memory using the cluster offering, not the worker offering, which new
worker VMs get. A size change in Alert is always such a change, as is a
size-only request in any state (the cluster-autoscaler's). Where the two
offerings differ, the recorded capacity is wrong. This is a known follow-up.
- A Running VM whose node never joined, or is NotReady, is still drained,
and the drain fails. Scaling up a cluster with such a node still times out, as
it does in Running.
These are existing issues and are left for separate changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
