bhouse-nexthop opened a new issue, #14650: URL: https://github.com/apache/cloudstack/issues/14650
### problem CKS clusters are set up with kubeadm, which issues the control plane certificates (API server, front proxy, stacked etcd, and the client certificates in `admin.conf`, `controller-manager.conf` and `scheduler.conf`) for one year. The certificate authorities last ten years. Nothing in CKS deals with that: 1. **No renewal.** CKS never runs `kubeadm certs renew`, and there is no API to do it. kubeadm's own answer is "upgrade at least once a year", since `kubeadm upgrade` renews the certificates as a side effect. A cluster that isn't upgraded within a year stops working when the certificates expire: the API server rejects the controller manager, the scheduler and every admin client, and kubectl from CloudStack itself (`/root/.kube/config` on the control nodes) fails too. 2. **No expiry check.** Nothing warns the operator before that happens. There's no alert and no event. 3. **The stored kubeconfig is not refreshed after an upgrade.** `getKubernetesClusterConfig` returns the `kubeConfigData` cluster detail, which CKS only reads from the control node when the cluster starts. `upgradeKubernetesCluster` renews the certificates, `admin.conf` included, but the stored copy keeps the old admin certificate. Anyone who downloads the kubeconfig from CloudStack after an upgrade still gets a certificate that dies at the old expiry date. The same happens after renewing the certificates by hand. Related: after an upgrade, `/root/.kube/config` on the control nodes, a copy of `admin.conf` that CKS uses for its own kubectl calls, also keeps the old certificate. ### versions main (and 4.22). The code involved hasn't changed in a long time, so earlier releases are affected too. ### The steps to reproduce the bug 1. Create a CKS cluster and leave it running. 2. On a control node, run `sudo /opt/bin/kubeadm certs check-expiration`. Every leaf certificate expires one year after creation, and nothing in CloudStack reports it. 3. Upgrade the cluster with `upgradeKubernetesCluster`. `check-expiration` now shows renewed certificates. 4. Call `getKubernetesClusterConfig` and decode the `client-certificate-data` of the returned kubeconfig: its expiry is still the original one, a year after creation. ### What to do about it? - Refresh the stored kubeconfig after an upgrade and after a renewal. - Check the certificate expiry periodically, and alert ahead of it (for example 60 and 30 days). - Add an API that renews the control plane certificates one control node at a time and restarts the control plane static pods. - Renew automatically, through the same path, some time before expiry (configurable). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
