bhouse-nexthop opened a new pull request, #14651:
URL: https://github.com/apache/cloudstack/pull/14651
### Description
kubeadm issues the control plane certificates of a CKS cluster for one year.
CKS never renewed them and never checked their expiry, so a cluster that isn't
upgraded within a year stops working when they expire. And because CKS only
read the kubeconfig that `getKubernetesClusterConfig` returns when the cluster
started, even an upgrade, which renews the certificates, left it handing out
the old admin certificate until that expired.
This PR checks the expiry, alerts ahead of it, adds an API to renew the
certificates one control node at a time, renews them automatically before they
expire, and refreshes the stored kubeconfig after every upgrade and renewal.
Fixes #14650
#### What changed
1. **Stored kubeconfig refresh.** `kubeConfigData` is read again from the
control node:
- after `upgradeKubernetesCluster` (a failure there is logged and doesn't
fail the upgrade);
- after every certificate rotation;
- whenever the expiry check finds the stored copy's admin certificate
expiring before the one on the cluster, e.g. after a renewal by hand.
The read-and-rewrite code that `StartWorker` had is now
`KubernetesClusterActionWorker.refreshKubeConfigData()`, shared by all three.
`KubernetesClusterUtil.getKubernetesClusterConfig` now waits 5 seconds between
attempts instead of retrying a refused SSH connection in a tight loop until its
timeout.
2. **Expiry check.** A new scanner, on its own thread next to
`KubernetesClusterStatusScanner`, runs `kubeadm certs check-expiration` over
SSH on every control node of each Running or Alert cluster, at most every
`cloud.kubernetes.cluster.certificate.check.interval` hours. It also reads the
client certificate of `/root/.kube/config`, which CKS uses for its own kubectl
calls. The scanner wakes every 15 minutes, but the per-cluster interval is
tracked in a cluster detail, so a cluster is checked once per interval
whichever management server runs it, and an unreachable cluster isn't retried
until the next interval.
- When the first leaf certificate reaches one of the
`cloud.kubernetes.cluster.certificate.expiry.alert.days` thresholds, CKS raises
an alert (new type `ALERT.KUBERNETES.CLUSTER.CERTIFICATE`) and records a
`KUBERNETES.CLUSTER.CERTIFICATES.EXPIRY` event. Each threshold fires once per
expiry date, and expiry itself alerts once more. A renewal moves the date, so
the thresholds start over.
- The first certificate authority to expire is tracked the same way, with
its own alert. CAs are reported, never rotated.
- When none of a cluster's control nodes can be checked, or the cluster
has no address to reach it on, an alert says so, at most once a day, so an
unreachable cluster can't expire unnoticed. Each cluster's state is read again
just before it is checked, so a cluster stopped during a pass is skipped.
3. **`rotateKubernetesClusterCertificates` API** (async, same roles as
`upgradeKubernetesCluster`). For each control node, one at a time, it copies
`rotate-kubernetes-certificates.sh` to the node and runs it. The script:
- puts back any static pod manifest left aside by an interrupted earlier
run;
- fails before touching anything if `crictl` can't list the containers,
since it needs it to confirm the static pods restarted;
- waits up to 3 minutes for the local API server to be ready, because
`kubeadm certs renew` silently falls back to its default configuration when it
can't read the cluster's. If it still isn't ready, the script warns and renews
anyway: once the certificates have expired the API server never becomes ready,
and renewing is the way out;
- runs `kubeadm certs renew all`, and fails if `apiserver.crt` or
`admin.conf` did not change or kubeadm says a certificate "can't be renewed"
(external CA mode, where kubeadm exits 0 without renewing anything);
- copies `admin.conf` to `/root/.kube/config`. If that file was something
else before the renewal, it is kept as `config.cks-backup`, and an existing
backup is never overwritten;
- moves the `kube-apiserver`, `kube-controller-manager`, `kube-scheduler`
and `etcd` manifests out, waits until the kubelet has stopped those containers,
then moves the manifests back. An exit trap moves them back on any failure too;
- waits for `/readyz` on the API server, `/healthz` on the controller
manager and scheduler, and for the API server to serve the renewed certificate.
Once the script is done on a node, the worker waits for the cluster API
endpoint to answer and the node to be Ready before it moves to the next node.
The shared readiness check now also accepts `Ready,SchedulingDisabled`, so a
cordoned control node doesn't fail the rotation. It stops at the first node
that fails, moves the cluster to Alert and raises an alert. After the last node
it refreshes the stored kubeconfig.
A failed rotation leaves the cluster in Alert, and the API accepts Alert
as well as Running, so it can be retried right away. A rotation started from
Alert goes back to Alert when it succeeds. Whatever else put the cluster there
hasn't been checked, so the status scanner moves it to Running once its nodes
are ready.
The API also rescues a cluster whose certificates have already expired.
Its API server no longer answers, so it is in Alert. With stacked etcd and
several control nodes, one renewed node can't get etcd quorum until the others
are renewed too, so waiting for each node in turn would never get past the
first. When `kubeadm certs check-expiration` on any control node shows an
expired certificate that the control plane runs on, the worker switches to
rescue mode. An expired `admin.conf` or `super-admin.conf` breaks no component
and doesn't count. Checking every node matters: after an interrupted rescue the
first node is renewed while the others still block quorum. The decision goes by
expiry, not by whether the API answers. An expired API server that is still
running keeps answering `/version`, while a healthy cluster that is briefly
unreachable must keep the node-by-node path, since restarting every control
node at once would cost it etcd quorum. In rescue mode:
- it renews every control node first, and on each node the script only
waits for etcd (when the node runs one) to run again and for new sandboxes of
the other control plane pods. The API server keeps restarting until etcd has
quorum, so it can't wait for that;
- then it waits for the cluster API and for every control node to be
Ready.
Rerunning is safe. `kubeadm certs renew` issues fresh certificates every
time, and the interrupted-run recovery above makes the script pick up where a
killed run left off.
4. **Automatic renewal.** When the first renewable leaf certificate is
within `cloud.kubernetes.cluster.certificate.auto.renew.days` and
`cloud.kubernetes.cluster.certificate.auto.renew.enabled` is true for the
cluster's account, the scanner hands the same rotation to a small thread pool,
so a renewal that takes minutes doesn't hold up the checks of other clusters.
When it is done, an event tells users to download the kubeconfig again. Values
above 180 days are treated as 180, so a mistyped setting can't renew every
cluster on every check.
- Certificates kubeadm reports as externally managed are left out of this
decision, since renewing can't fix them and every check would renew again.
- After a failed rotation, automatic renewal waits 24 hours before trying
again; the failure has already raised an alert.
- When a control node couldn't be checked, automatic renewal waits too:
it would renew the reachable nodes and then fail on that one. The expiry alert
then says renewal is on hold and names the nodes.
5. **Guards.**
- Rotation and the check need Kubernetes 1.20 or later, where `kubeadm
certs` replaced `kubeadm alpha certs`. The API refuses older clusters and the
scanner skips them.
- The API refuses clusters that aren't Running or Alert, and externally
managed clusters. Clusters in Alert are checked and alerted on, since one can
sit there for long (a stopped worker is enough), but automatic renewal only
runs on Running clusters.
- A queued automatic renewal checks again when it gets a thread, and
skips the cluster if it is no longer Running or was rotated meanwhile,
successfully or not.
- Clusters created before 4.16 have no control node flag in
`kubernetes_cluster_vm_map`. For those, the control nodes are taken by
position, after any etcd nodes, as `fetchControlVmIfMissing` already does.
- `startKubernetesCluster` and `stopKubernetesCluster` refuse a cluster
in `RotatingCertificates`, whose control plane manifests may be moved aside at
that moment. Upgrade already requires Running, and scale Created, Running or
Stopped.
- A new transient state, `RotatingCertificates`, is entered from Running
or Alert through the new `CertificateRotationRequested` event. It goes back to
Running on success and to Alert on failure. As with upgrade and scale, the
state transition is what keeps other operations out: none of them can start
from `RotatingCertificates`, and a rotation can't start while another operation
runs.
- The status scanner's Alert reconciliation could move a cluster to
Running, or to Stopped, while a rotation from Alert had just started.
`reconcileAlertCluster` and the all-VMs-stopped branch now stop when their own
first transition fails, instead of applying `OperationSucceeded` regardless.
- On management server shutdown, automatic renewals still waiting for a
thread are dropped rather than started; the next check after the restart
schedules them again.
- If the management server running a rotation goes away, the cluster
would stay in `RotatingCertificates`. The rotation records which management
server runs it. When that server comes back, its scanner moves the rotation to
Alert straight away; any other server does so once it is well past
`cloud.kubernetes.cluster.certificate.rotation.timeout` (twice the timeout plus
10 minutes). Either way an alert is raised.
6. `upgrade-kubernetes.sh` now also copies the renewed `admin.conf` to
`/root/.kube/config` on control nodes, so CKS's own kubectl keeps working after
the old certificate expires. As in the rotation script, a `/root/.kube/config`
that differed from `admin.conf` before the upgrade is kept as a backup first.
No schema changes. The state column is `char(32)`, and the bookkeeping (last
check, alerted thresholds, rotation start and last failure) is kept in
`kubernetes_cluster_details`.
#### Settings
| Setting | Default | Scope | Meaning |
|---|---|---|---|
| `cloud.kubernetes.cluster.certificate.check.interval` | `6` | Global |
Hours between expiry checks of a cluster. `0` disables the check and automatic
renewal. |
| `cloud.kubernetes.cluster.certificate.expiry.alert.days` | `60,30` |
Global | Days before expiry at which to alert. |
| `cloud.kubernetes.cluster.certificate.auto.renew.enabled` | `true` |
Account | Renew automatically before expiry. |
| `cloud.kubernetes.cluster.certificate.auto.renew.days` | `30` | Account |
How many days before expiry to renew automatically. `0` disables it; values
above 180 are treated as 180. |
| `cloud.kubernetes.cluster.certificate.rotation.timeout` | `3600` | Global
| Timeout of a rotation, in seconds. |
All of them are dynamic.
Automatic renewal is on by default. A plain kubeadm cluster with no upgrades
and no renewal timer is the one setup where the certificates simply run out.
RKE2 and k3s renew on restart, and managed services rotate them unseen.
Renewing 30 days ahead leaves a month for people and CI that downloaded the old
kubeconfig to fetch it again, and the event says so. Anyone who wants to
schedule the restarts themselves can turn it off globally or per account and
use the API.
#### Restart impact
Static pods don't reload certificates, so each control node's API server,
controller manager, scheduler and stacked etcd restart. One node takes about 40
seconds.
- **Several control nodes:** the restart rolls one node at a time and etcd
keeps quorum. The API stays up behind the load balancer. A few requests that
the load balancer sends to the restarting node can fail before its health check
takes the node out; clients retry them. In the three-node test below, 6 of 187
polls through the load balancer failed, two per node.
- **One control node:** the API is unavailable for roughly 30–40 seconds.
- **Workloads keep running in both cases.** Only the control plane restarts;
kubelets and pods don't.
- **Kubeconfigs downloaded before the renewal** keep working until their old
certificate expires. Renewal doesn't revoke the old certificate.
#### External etcd
`kubeadm certs renew` doesn't touch separate etcd nodes, but the etcd nodes
CKS creates (`etcd-node.yml`) serve etcd over plain HTTP: `--listen-client-urls
http://...`, peer URLs `http://...`, no `--cert-file`. They hold no
certificates, so there is nothing to renew or check on them, and the rotation
only touches control nodes. kubeadm sees those clusters as having external
etcd, so `check-expiration` and `renew` leave out the etcd certificates there
and handle the rest normally.
If etcd nodes ever get TLS, they'll need their own renewal step. Unencrypted
etcd traffic is a separate concern and out of scope here.
#### Not in this PR
- CA rotation; CAs are only reported.
- A UI button for the new API.
- `/etc/kubernetes/user.conf`. CKS returns it instead of `admin.conf` when
an operator has put one on the control node. It is the operator's file,
possibly signed by something CKS doesn't manage, so CKS neither renews nor
rewrites it. The stale-kubeconfig check compares the stored copy with it rather
than with `admin.conf`.
- The cluster state update is not a compare-and-set
(`KubernetesClusterDaoImpl.updateState` writes unconditionally). That's how
every CKS operation works today, and this PR uses the same guard. Likewise, the
existing workers carry on when their own first transition fails. An upgrade
validated as Running just before an automatic renewal takes the cluster could
therefore still run alongside it; the window is the time between the API's
state check and the worker's transition. Making `updateState` conditional, and
having every worker stop when its first transition fails, would close this race
for all operations. That's a separate change.
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] New feature (non-breaking change which adds functionality)
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Feature/Enhancement Scale
- [ ] Major
- [x] Minor
#### Bug Severity
- [ ] BLOCKER
- [x] Critical
- [ ] Major
- [ ] Minor
- [ ] Trivial
### How Has This Been Tested?
#### Unit tests
All 189 tests of the CKS plugin pass, including these new ones.
- `KubernetesClusterCertificateUtilTest`:
- parsing real `kubeadm certs check-expiration` output (v1.31): both
tables, missing, expired and externally managed certificates, and error output;
- reading a kubeconfig's client certificate expiry;
- the threshold parsing and alert decision (one alert per threshold per
expiry date; a cluster first seen inside both windows alerts once; expiry
alerts once more);
- the auto-renew decision.
- `KubernetesClusterCertificateWorkerTest`:
- the rolling sequence: node by node, renew then wait for ready, then the
kubeconfig refresh and Running;
- stopping at the first node that fails to renew or doesn't become ready,
with the remaining nodes untouched, Alert and an alert;
- rescue mode when a certificate on any control node has expired (every
node renewed first, then each waited for), and the node-by-node path otherwise;
- a rotation from Alert going back to Alert;
- a second rotation of the same cluster refused without disturbing the
first, and a cluster another server moved out of this rotation meanwhile left
in its state;
- `/root/.kube/config` treated as externally managed when `admin.conf` is
(external CA mode), so it doesn't trigger renewal;
- failure events recorded only for automatic renewals, since the API's
async job records its own;
- a failed kubeconfig refresh;
- refusal when the state transition fails and before 1.20;
- the per-node SSH endpoint (direct network, the cluster IP's port
forwarding rule rather than a user's rule on another IP, and by position);
- the check across control nodes with one unreachable;
- refreshing a stale stored kubeconfig, and comparing it with `user.conf`
when there is one.
- `KubernetesClusterCertificateManagementTest`:
- the scanner: at most one check per interval, the 60 and 30 day alerts
firing once each as expiry approaches, thresholds starting over after a
renewal, CA alerts without rotation, Alert clusters checked but not renewed, no
automatic renewal while a control node is unreachable (and the alert saying
so), the daily alert when no node can be checked or the cluster can't be
reached at all, the alert advice using the effective renewal days, auto-renew
on and off and handed to the rotation thread pool (queued once, requeued after
a rejection, skipped when renewed meanwhile), the 24 hour back-off after a
failure, externally managed certificates not triggering renewal, and the
version gate;
- moving an interrupted rotation to Alert: at once when it belongs to this
management server, otherwise only once it is stale, and never a rotation that
just finished;
- the API validation (Running and Alert accepted) and the new state
machine transitions.
- `KubernetesClusterUpgradeWorkerTest`: the stored kubeconfig is refreshed
after a successful upgrade, before the cluster goes back to Running; a failed
refresh doesn't fail the upgrade; a failed upgrade doesn't refresh.
#### The renewal script on real kubeadm control planes
I ran `rotate-kubernetes-certificates.sh` on throwaway kind clusters
(Kubernetes v1.31, kubeadm with stacked etcd):
- **Single control node:**
- certificates renewed, all four static pods restarted with new container
IDs, the API server serving the renewed certificate, `/root/.kube/config`
identical to the new `admin.conf`, the node Ready. About 40 seconds.
- `ca.key` moved away (external CA mode): kubeadm exits 0 and renews
nothing it needs the CA key for. The script fails before restarting anything,
and the API server container is unchanged.
- Killed half-way, with the `kube-apiserver` and `etcd` manifests left
aside and the API server down: the rerun puts them back, waits for the API
server, renews with the cluster's own kubeadm configuration rather than the
fallback defaults, and completes.
- `ca.crt` hidden so `kubeadm certs renew all` itself fails: the script
prints kubeadm's error, exits 1, and leaves all four manifests in place.
- API server unreachable (port 6443 rejected inside the node, standing in
for expired certificates): the script waits 3 minutes, warns, renews, restarts
the static pods and completes once the API server is reachable again.
- **Rescue of a three-node cluster with no etcd quorum.** I rejected the
etcd peer port on all three nodes, standing in for expired peer certificates.
That leaves etcd without quorum and the API servers restarting. This confirmed
the review's point: a node-by-node wait can't get past the first node.
- In rescue mode, each node took about 40 seconds, and the script exited 0.
- Once the peer port was open again, `/readyz` returned ok and all three
nodes were Ready. Each API server served its renewed certificate, and etcd
reported all three members healthy.
- Without the cluster's API, kubeadm renews with its default
configuration. It takes the certificate attributes (CN, SANs) from the existing
certificates, so they come out the same.
- A custom `/root/.kube/config` survived two renewals in a row as
`config.cks-backup`, and the readiness check matched a cordoned control node.
- **Three control nodes behind kind's load balancer:** renewed node by node,
waiting for each node to be Ready with the same `kubectl get nodes` check the
worker uses, while polling `/readyz` through the load balancer every 0.5 s.
- Every node served its renewed certificate afterwards, and etcd reported
all three members healthy.
- 6 of 187 load balancer polls failed, two per node while its API server
restarted.
I also fed the `check-expiration` output and the `/root/.kube/config`
certificate line from those nodes through the Java parser, and the expiry dates
matched.
I haven't run a full CKS cluster through the new API and the scanner end to
end. The Java orchestration around the script is covered by the unit tests
above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
