bhouse-nexthop opened a new pull request, #14616:
URL: https://github.com/apache/cloudstack/pull/14616
### Description
Lets `updateKubernetesCluster` change the service offering, root disk size
and affinity groups of each node type of a CKS cluster. With `rollout=true`, it
then restarts the existing nodes that differ with the new settings, a few at a
time, cordoning and draining each one first.
Fixes #14615
**Depends on #14609 and #14611.** Their commits are the first two on this
branch, and they drop out once 4.22 is merged forward into main. Please review
only the last commit, "CKS: restart the nodes of a cluster with new node
settings, a few at a time". It uses #14609's compare-and-set state changes
(`transitToOperationState`) and #14611's drain helper
(`drainKubernetesClusterNode`).
This targets main only, because it adds three columns.
#### Nodes are restarted, not changed live
Each node is restarted. This PR doesn't scale VMs live, migrate them live,
or resize disks online:
- Live scaling of a KVM VM depends on the offering being dynamic and on the
guest hot-plugging the CPUs and memory. kubelet doesn't pick up the new
capacity until it restarts anyway.
- An affinity group only takes effect when the deployment planner places the
VM, which happens on start.
- Growing the root partition and file system is done by cloud-init at boot.
A stop and start applies all three changes the same way on every hypervisor,
and the node is drained first, so the pods move before it goes down.
#### API
`updateKubernetesCluster` gets these parameters:
- `nodeofferings`, `noderootdisksizes` (GB) and `nodeaffinitygroups`, set
per node type (`control`, `worker`, `etcd`).
- `rollout` (default false).
- `maxunavailable`: the number of workers restarted at a time, default 1,
only with `rollout=true`.
- `cancelrollout`.
With `rollout=false`, the settings are only stored, and nodes created later
get them. This works in any state except while another operation is running on
the cluster or it is being destroyed.
- A cluster created before its node types had settings of their own first
gets the cluster's service offering copied to each type, so changing one type
leaves the others as they are.
- If all nodes of a type share the same affinity groups, those are recorded
for the type.
- Templates are not copied, and this API doesn't change them.
A root disk size is refused with `rollout=true` if it would shrink an
existing node's disk, or if the disk's offering has a strictly fixed size. With
`rollout=false` it is accepted, as `createKubernetesCluster` accepts it.
The response adds:
- the root disk size of each type;
- `controlnodesoutofdate`, `workernodesoutofdate` and `etcdnodesoutofdate`,
the number of nodes of each type that differ from its settings;
- `lasterror`: why the last rollout stopped, or which control or etcd nodes
it could not restart.
#### Rollout
`rollout=true` moves the cluster from Running or Alert to a new transient
state, `Updating`, then stores the settings. It restarts the out-of-date nodes
in this order:
1. the external etcd nodes;
2. the control nodes, one at a time;
3. the workers, `maxunavailable` at a time. This is clamped to
`min(maxunavailable, workers - 1)`, and is always at least 1.
A node is out of date when its service offering differs, its root disk is
smaller than the type's size, or its affinity groups differ from those recorded
for the type. Explicit dedication groups, which CKS adds by itself, are not
compared and are kept. A type with no recorded groups leaves the nodes' groups
as they are. `nodeaffinitygroups` refuses explicit dedication groups. Stopped
nodes, and nodes still carrying the annotation from a rollout that stopped, go
first.
Each batch goes through these steps:
1. **Preflight:** every other VM must be Running and every other Kubernetes
node Ready.
- For a control or etcd node, a quorum of the other etcd members must
also be healthy, and etcd must report ready.
- With external etcd, another control node must be serving the API.
- Some types can't lose a node at all, for example a single control node
or a 2-node stacked control plane. A `rollout=true` that would change such a
type's settings is refused at the API. If such a type is out of date for
another reason, its nodes are skipped and reported in `lasterror`, and the
other types are still rolled.
2. **Cordon and drain:** each node is annotated with
`cloudstack.apache.org/rollout-boot-id=<bootID>`, then cordoned. A node someone
else has already cordoned is left without the annotation, and stays cordoned.
Each node is then drained with #14611's helper.
3. **Change:** the VM is stopped, then `upgradeVirtualMachine` sets the
offering, `resizeVolume` grows the root disk (offline, grow only), and the
affinity groups are updated. The VM is then started, and the deployment planner
places it by its new groups.
4. **Verify:** the node must come back Ready with a new boot ID. A Ready
condition left over from before the stop is not accepted. The VM must also no
longer differ from the settings. Once both hold, the node is uncordoned and the
annotation removed.
- Each batch has `cloud.kubernetes.cluster.update.node.timeout` seconds
(default 1800).
`cancelrollout=true` stops the rollout between two batches.
However a rollout ends, a single idempotent `release()` runs. It:
- uncordons the annotated nodes that are Ready with a new boot ID, and
leaves the rest cordoned for someone to look at;
- recounts the cluster's CPU cores and memory from the VMs' actual offerings;
- moves the cluster to Running if it is healthy, or to Alert if not;
- records or clears `lasterror`.
Nothing else is stored. The nodes that still differ, are stopped or still
carry the annotation are the work left to do. A rollout that stopped is resumed
by calling `rollout=true` again, which also uncordons any node that came back
after the last run.
While a cluster is `Updating`, the other mutating APIs are refused. If the
management server running a rollout stops, the state scanner moves the cluster
from `Updating` to Alert and recounts its capacity. It doesn't reach into the
nodes. The next rollout picks them up.
#### Data model
The schema change adds `control_node_root_disk_size`,
`worker_node_root_disk_size` and `etcd_node_root_disk_size` to
`kubernetes_cluster`, idempotently, in `schema-42300to2400.sql`. Nodes created
later use these sizes. Cancel requests and last errors are cluster details
(`rollout.cancel`, `rollout.lasterror`). There are no new tables.
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] build/CI
- [ ] test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Feature/Enhancement Scale
- [ ] Major
- [x] Minor
### How Has This Been Tested?
**Unit tests only.** Nothing here has been run against a live cluster yet.
`mvn -o test -pl plugins/integrations/kubernetes-service` passes (233
tests). `engine/schema` compiles. The new tests are:
- `KubernetesClusterRollingUpdateWorkerTest` runs the worker against a
simulated cluster (VMs, kubectl output, boot IDs). It covers:
- node selection;
- worker batching and the `maxunavailable` clamp;
- control nodes one at a time on a 3-node stacked control plane;
- quorum refusals: a 2-node stacked control plane, an unhealthy etcd
member, a single control node whose workers still roll, and etcd and control
both skipped;
- replacing the affinity groups while keeping the dedication group;
- the annotate → cordon → drain → stop → change → start → verify order;
- rejecting a stale Ready with the same boot ID;
- a node cordoned by someone else;
- `release()` after success, a drain failure, a start failure, a cancel
and an early exception, each run twice;
- a cancel asked for while the settings are being stored;
- a failure reported together with the skipped nodes;
- rerunning after a failure, which takes the stopped node first and
uncordons a node that came back without restarting it.
- `KubernetesClusterManagerImplUpdateTest` covers:
- `rollout=false`;
- gen 1 to gen 2;
- the API validation (`maxunavailable`, disk sizes, shrink and strict
offerings, node types, dedication groups, control settings that can't be
rolled);
- refusing an update while another operation runs;
- Running or Alert for a rollout;
- `Updating` before the settings are stored, and going back to the
previous state, with the last error kept, if storing fails or the settings are
refused;
- refusing settings that would make a node of a 1-control-node or
2-etcd-node cluster out of date, while accepting no-op settings, nodes that
already differed and stopped nodes;
- cancel;
- the out-of-date reasons, including explicit dedication groups;
- the capacity recount;
- the `Updating` guards on the other APIs.
- `UpdateKubernetesClusterCmdTest` covers parsing of the per-type maps.
Not tested yet, and the main risks in a live test:
- **Root partition growth:** this relies on the CKS node templates'
cloud-init running growpart and resizefs at boot. If a template doesn't, the
disk grows but the file system doesn't. Nothing here grows it over SSH.
- **Affinity on start:** the deployment planner's affinity processors should
honour the new groups when the stopped VM starts, including the last-host
handling. I've read the code but haven't tried it.
- **etcd health checks:** each check runs over SSH on the node itself:
`etcdctl endpoint health` on an external etcd node, and `curl
http://127.0.0.1:2381/health`, the metrics address kubeadm sets up, on a
stacked control node. The cluster-wide check is `kubectl get
--raw=/readyz/etcd`. All of these were checked against the CKS scripts, not
against a running cluster.
- **Quorum on a 3-node stacked control plane:** in particular, kubectl has
to go through another control node while the first is down. On a shared
network, `admin.conf` points at the first control node.
- **Drain timeouts with PodDisruptionBudgets:** a PDB that never allows an
eviction fails the batch at the drain. The node is uncordoned, as it never went
down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
