bhouse-nexthop commented on PR #14651:
URL: https://github.com/apache/cloudstack/pull/14651#issuecomment-6757094630
### Test report: CKS control plane certificate check, rotation and automatic
renewal, on live clusters
**Summary:**
- **Expiry check.** After the upgrade, the scanner checked every cluster and
raised no alerts at the default thresholds. It found two clusters whose stored
kubeconfig was older than `admin.conf` and refreshed both. With the threshold
raised above the remaining validity, each cluster alerted once, with one event,
and not again on the next pass.
- **Rotation through the API.** `rotateKubernetesClusterCertificates`
renewed a 3-control-node cluster one node at a time, twice in a row. The API
stayed up behind the load balancer, workloads kept running, and the stored
kubeconfig was refreshed and worked. On a single control node the API was down
for about 38 s.
- **Guards.** Start and stop were refused during a rotation, and a rotation
of a stopped cluster was refused.
- **Upgrade.** An upgrade refreshed the stored kubeconfig.
- **Automatic renewal.** It ran on its own for a cluster whose certificates
were within the renewal window, with no change to the clock.
**Environment**
- A CloudStack 24.0 (main) build with this PR, alongside some of my other
open PRs. The management server and all three agents were upgraded in place to
this build.
- The PR targets main; this is a main-based build.
- Three KVM hosts, Ceph RBD primary storage, one management server.
- CKS clusters on isolated networks:
- **A:** 3 control nodes and 2 workers, Kubernetes v1.33.1, with the API
behind the network's load balancer;
- **B:** 1 control node and 1 worker, v1.33.13;
- **C:** 1 control node and 2 workers, v1.33.1;
- **D:** a fresh cluster with 1 control node and 1-3 workers, v1.32.13
upgraded to v1.33.13, then deleted.
A, B and C were created earlier. A and C have since had their certificates
renewed by an upgrade, and B by an earlier upgrade made before this build.
- Settings changed for the test and reset afterwards:
- `cloud.kubernetes.cluster.certificate.check.interval=1`;
- `cloud.kubernetes.cluster.certificate.expiry.alert.days=400`;
- `cloud.kubernetes.cluster.certificate.auto.renew.days=180`, for the
owner account only.
- **API polling.** I polled the API through each cluster's public endpoint
every 0.5 s with `GET /readyz`, using a kubeconfig downloaded before the
rotation, with a 2 s timeout.
**Results**
| Scenario | Result |
|---|---|
| First check after the upgrade (defaults) | pass. All 4 clusters checked,
with no alerts or events. The stale stored kubeconfig of B and C was refreshed |
| Threshold above the remaining validity (`400`) | pass. One alert and one
event per cluster; none on the next pass an hour later |
| Rotation of A (3 control nodes) | pass, 132 s. Node by node, every
certificate renewed, back to Running |
| API during A's rotation | 20 of 205 polls failed, in a ~20 s window per
node; it was never fully down |
| Immediate second rotation of A | pass, 134 s. Same pattern, fresh
certificates again |
| Workloads on A | pass. No pod restarted or was rescheduled apart from the
CCM (see below); scheduling still worked |
| Stored kubeconfig after each rotation | pass. Matches the first control
node's new `admin.conf`, and `kubectl get nodes` works with it |
| Rotation of B (1 control node) | pass, 45 s; API outage 38.1 s |
| Start during a rotation (A, twice; B) | pass: refused, "Kubernetes cluster
... is renewing its certificates" |
| Stop during a rotation (B) | pass: refused, "... is renewing its
certificates, stop it once that is done" |
| Rotation of a Stopped cluster (D) | pass: refused, "Kubernetes cluster ...
is not in Running or Alert state" |
| Upgrade of D, v1.32.13 -> v1.33.13 | pass. The stored kubeconfig after the
upgrade is the new `admin.conf` |
| Automatic renewal (B with real 99-day certificates) | pass. Renewed on the
scanner's pass, back to 1-year certificates, event recorded; API outage 37.2 s |
| Rescue mode (expired certificates) | not covered |
**First check after the upgrade**
- The scanner's first pass ran 10 minutes after the management server
started. It recorded `certificateExpiryLastCheck` for all four clusters. It
raised no alert and no `KUBERNETES.CLUSTER.CERTIFICATES.EXPIRY` event, since
everything had more than 360 days left.
- Log, trimmed:
```
The stored kubeconfig of Kubernetes cluster B expires on
2027-11-09T15:39:46, before the kubeconfig on the cluster
(2027-11-09T20:32:00), refreshing it
Kubernetes cluster B: first control plane certificate to expire is
/root/.kube/config on B-control: Mon Nov 08 18:31:52 UTC 2027, first
certificate authority is ca ...: Wed Nov 05 18:31:00 UTC 2036
Kubernetes cluster A: first control plane certificate to expire is
/root/.kube/config on A-control-1: Mon Nov 08 18:36:47 UTC 2027, ...
The stored kubeconfig of Kubernetes cluster C expires on
2027-11-08T18:34:16, before the kubeconfig on the cluster
(2027-11-09T15:59:00), refreshing it
Kubernetes cluster C: first control plane certificate to expire is
/root/.kube/config on C-control: Mon Nov 08 18:34:16 UTC 2027, ...
```
- **B and C were upgraded before this build.** Their stored kubeconfig still
had the admin certificate from before the upgrade. After this pass,
`getKubernetesClusterConfig` returned each cluster's current `admin.conf`
(serials checked against the control node). A's copy was already current and
was left alone.
- The first entry to expire on each node was `/root/.kube/config`, which
those older upgrades hadn't refreshed. It's checked and reported as the PR
describes.
**Alert thresholds**
- I set `expiry.alert.days=400`, above the ~364 days left, and
`check.interval=1`. On the next check of each cluster:
```
alert 38 A control plane certificate of Kubernetes cluster B expires in
364 days
alert 38 A control plane certificate of Kubernetes cluster A expires in
364 days
alert 38 A control plane certificate of Kubernetes cluster C expires in
363 days
event KUBERNETES.CLUSTER.CERTIFICATES.EXPIRY WARN "... admin.conf on
A-control-1 expires on Tue Nov 09 23:53:00 UTC 2027.
They are renewed automatically 30 days before they expire while the
cluster is Running, or run rotateKubernetesClusterCertificates."
```
There was one alert and one event per cluster. D, created meanwhile,
alerted once on its own first check. The CAs, 9-10 years out, didn't alert.
- **An hour later**, the next pass checked A and C again
(`certificateExpiryLastCheck` moved) and raised nothing. B alerted again
because its first expiry had moved to a new date (see Automatic renewal), which
is the documented "a renewal starts the thresholds over".
- D was upgraded between its two checks, which moved its first expiry by 53
minutes. It didn't alert again, as dates within 7 days count as the same expiry.
- Afterwards I reset both settings to their defaults (`6`, `60,30`).
**Rotation of A, with the API polled**
- Each run went `Running` -> `RotatingCertificates` -> `Running`, with one
control node at a time:
```
23:51:11 Rotating the control plane certificates of Kubernetes cluster A
23:51:55 Renewed the certificates on control node A-control-1
23:52:39 Renewed the certificates on control node A-control-2
23:53:22 Renewed the certificates on control node A-control-3
23:53:23 Rotated the control plane certificates of Kubernetes cluster A
```
- The static pod containers were recreated node by node: etcd,
kube-apiserver, controller-manager and scheduler, at 23:51:48, 23:52:31 and
23:53:15.
- Every certificate moved on every node: `apiserver.crt`, `etcd/peer.crt`
and `admin.conf`, each with a new serial and a notAfter one year out.
`/root/.kube/config` on each node now matches that node's `admin.conf`.
- **Second run, started right away.** It took 134 s and showed the same
sequence. All three nodes got fresh certificates again, with new serials and
notAfter moved by about 2.5 minutes. It ended Running, with all 5 nodes Ready.
- **Polls through the load balancer:**
```
run 1: 205 polls, 20 failed run 2: 286 polls, 22 failed (36
timeouts/connection failures, 6 HTTP 500)
failures came every ~3.5 s in a ~20 s window per node (23:51:17-23:51:38,
23:52:00-23:52:20, 23:52:43-23:53:04, ...),
plus one at each node's end
```
- The API never went fully down. About 1 poll in 3 failed while one node
restarted, which is the share the load balancer sends to that node.
- The PR description says "a few requests ... before its health check
takes the node out", with 2 per node in its kind test. Here it was about 6-7
per node, because this load balancer kept sending requests to the restarting
node for the whole restart.
- **Workloads:**
- A 2-replica test Deployment and every pod in `kube-system` and
`kubernetes-dashboard` kept their UID and restart count, apart from the static
control plane pods.
- The exception was `cloud-controller-manager`. Its restart count went
from 2 to 4, i.e. it restarted once per run, most likely on losing its leader
lease during an API restart.
- Scaling the test Deployment to 3 afterwards scheduled the new pod at
once.
- **Stored kubeconfig:** after each run its client certificate was the first
control node's new `admin.conf` (`77D29D...` then `2EDE16...`), and `kubectl
get nodes` with it showed 5 Ready.
**Rotation of B, with the API polled**
- It went `RotatingCertificates` at 23:57:29 and was back to `Running` at
23:58:13. The node was renewed at 23:58:12.
- The API was unavailable from 23:57:31.9 to 23:58:10.1, which is **38.1
s**. There were 29 timeouts, then 2 HTTP 500s and 1 HTTP 403 while the API
server started.
- The certificates moved from the earlier upgrade's (notAfter Nov 9 2027,
20:32) to Nov 9 2027, 23:57. The stored kubeconfig was refreshed and worked.
- Workloads: the same picture as A. The CCM restarted twice during this
rotation; nothing else restarted.
**Guards**
- **During A's rotations**, `startKubernetesCluster` failed with `Kubernetes
cluster A is renewing its certificates`.
- **During B's rotation:**
- `stopKubernetesCluster` failed with `... is renewing its certificates,
stop it once that is done`;
- `startKubernetesCluster` failed with `... is renewing its certificates`;
- the cluster stayed `RotatingCertificates` and finished normally.
- **On cluster D, Stopped**, `rotateKubernetesClusterCertificates` failed
with `Kubernetes cluster D is not in Running or Alert state`.
**Kubeconfig after an upgrade**
- D was upgraded from v1.32.13 to v1.33.13 with `upgradeKubernetesCluster`,
which took 323 s. I read the kubeconfig as soon as the job returned:
```
before: serial=0784099664F87964 notAfter=Nov 9 23:55:49 2027 (=
admin.conf before)
after: serial=6AC793C3F84605D1 notAfter=Nov 10 00:50:59 2027 (=
admin.conf and /root/.kube/config after the upgrade)
```
- `kubectl` with the new kubeconfig showed both nodes Ready on v1.33.13.
**Automatic renewal, without changing the clock**
- `auto.renew.days` is capped at 180, and every cluster had about 364 days
left. So I gave B real certificates with a 100-day validity.
- I ran `kubeadm certs renew all` on B's control node, with the cluster's
own ClusterConfiguration and `certificateValidityPeriod: 2400h0m0s` added. This
went to a temporary file; the cluster's `kubeadm-config` ConfigMap was left
unchanged. `check-expiration` then showed 99 days for every leaf certificate.
- I set `auto.renew.days=180` for the owner account. On B's next check:
```
01:47:43 Renewing the certificates of Kubernetes cluster B automatically,
admin.conf on B-control expires on 2027-02-18T00:55:00
01:47:43 Rotating the control plane certificates of Kubernetes cluster B
[Kubernetes-Cluster-Certificate-Rotation thread]
01:48:24 Renewed the certificates on control node B-control
01:48:25 Rotated the control plane certificates of Kubernetes cluster B
event KUBERNETES.CLUSTER.CERTIFICATES.ROTATE INFO "Automatically renewed
the control plane certificates of Kubernetes cluster B.
Kubeconfigs downloaded before keep working only until their old
certificate expires; download it again ..."
```
- The certificates were valid for a year again (`Nov 10, 2027`). The stored
kubeconfig was the new `admin.conf`, and the cluster went
`RotatingCertificates` -> `Running`.
- The API outage was 37.2 s. The same pass raised the 99-day expiry alert,
as the 400-day threshold was still set.
- **One more API outage during this wait isn't from the PR.** It lasted 66
s, at 01:01, about 45 minutes before the renewal. Nothing was rotating then;
the control node's kubelet logged `etcdserver: request timed out`, and Ceph
reported BlueStore slow operations at the time.
- Afterwards I reset the account setting.
**Result:** pass.
- The check finds and reports expiry on every control node, alerts once per
threshold, and refreshes stale stored kubeconfigs.
- The rotation renews the control nodes one at a time and keeps the cluster
usable.
- The guards hold, the upgrade refreshes the kubeconfig, and the automatic
renewal runs on its own.
**Not covered**
- **Rescue mode** (certificates already expired, etcd without quorum).
Testing it live would have meant letting a cluster's certificates expire and
leaving it down for that time, so I didn't run it here.
- A failed rotation, interrupted rotations, and a management server restart
during one.
- External etcd clusters, and clusters older than 1.20.
- Several management servers.
**Notes**
- With a 3-node control plane behind the CloudStack network load balancer,
expect about a third of API requests to fail for ~20 s per node. That's longer
than the description's "a few requests". It might be worth saying so in the
restart impact section, or checking whether the load balancer can health-check
port 6443.
- The CCM restarts once or twice per rotation (on losing its leader lease).
It recovers on its own.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
