kiranchavala opened a new issue, #14181:
URL: https://github.com/apache/cloudstack/issues/14181
### problem
CKS writes a kubeadm configuration file into the control node's cloud-init,
pinned to the `v1beta3` API:
```yaml
#
plugins/integrations/kubernetes-service/src/main/resources/conf/k8s-control-node.yml:245
- path: /etc/kubernetes/kubeadm-config.yaml
...
apiVersion: kubeadm.k8s.io/v1beta3 # :249 (ClusterConfiguration)
...
apiVersion: kubeadm.k8s.io/v1beta3 # :260 (InitConfiguration)
```
`kubeadm.k8s.io/v1beta3` was deprecated when `v1beta4` was introduced in
Kubernetes 1.31, with support to be "removed after a minimum of 3 Kubernetes
minor releases"
([Kubernetes v1.31: kubeadm
v1beta4](https://kubernetes.io/blog/2024/08/23/kubernetes-1-31-kubeadm-v1beta4/)).
It has since been reported removed as of Kubernetes 1.37
([kubernetes-sigs/image-builder#2151](https://github.com/kubernetes-sigs/image-builder/issues/2151)).
Once removed, `kubeadm init --config` rejects the file and cluster creation
fails.
**Scope — this affects external-etcd clusters only.** That config file is
passed to kubeadm on exactly one path:
```bash
# k8s-control-node.yml:299-303
if [[ ${EXTERNAL_ETCD_NODES} == true ]]; then
kubeadm init --config /etc/kubernetes/kubeadm-config.yaml --upload-certs
else
kubeadm init --token {{ ... }} --token-ttl 0 {{
k8s_control_node.cluster.initargs }} --cri-socket
/run/containerd/containerd.sock
fi
```
So:
| Path | Uses kubeadm config file? | Affected on 1.37+ |
|---|---|---|
| Control node, external etcd (`:300`) | yes | **yes** |
| Control node, stacked etcd (`:302`) | no — flags only | no |
| Additional control plane join (`k8s-control-node-add.yml:239`) | no —
flags only | no |
| Worker join (`k8s-node.yml:254`) | no — flags only | no |
`grep -rn "kubeadm.k8s.io/v1beta" plugins/ systemvm/` returns only those two
lines in the whole tree, so the fix is narrowly scoped.
Compounding this: CKS enforces no **maximum** supported Kubernetes version.
`addKubernetesSupportedVersion` will happily accept 1.37.x, and the failure
only surfaces part-way through cluster creation on the control node, which
makes it hard to diagnose
### versions
- Affects `main`, `4.20`, `4.21`, `4.22` — the templated `v1beta3` is
identical on all of them.
- Triggered by Kubernetes >= 1.37 (the release reported to have removed
`v1beta3`).
### The steps to reproduce the bug
1. Build a CKS binaries ISO for Kubernetes 1.37.x
(`scripts/util/create-kubernetes-binaries-iso.sh`).
2. Register it: `addKubernetesSupportedVersion semanticversion=1.37.0
url=...`
3. Create a Kubernetes cluster **with external etcd nodes** (`etcdnodes=1`
or more), which selects the
`kubeadm init --config` path.
4. On the control node, `kubeadm init` fails because
`/etc/kubernetes/kubeadm-config.yaml` declares a
config API version that kubeadm no longer recognises. Cluster creation
does not complete.
A cluster on the same version **without** external etcd should succeed,
since that path passes flags only —
which is a useful way to confirm the diagnosis.
logs
```
root@test-cks-etcd-control-1a0a97ce958:/opt/bin# ./deploy-kube-system status
error: your configuration file uses an old API spec:
"kubeadm.k8s.io/v1beta3" (kind: "ClusterConfiguration"). Please use kubeadm
v1.36 instead and run 'kubeadm config migrate --old-config old-config-file
--new-config new-config-file', which will write the new, similar spec using a
newer API version.
To see the stack trace of this error execute with --v=5 or higher
error: your configuration file uses an old API spec:
"kubeadm.k8s.io/v1beta3" (kind: "ClusterConfiguration"). Please use kubeadm
v1.36 instead and run 'kubeadm config migrate --old-config old-config-file
--new-config new-config-file', which will write the new, similar spec using a
newer API version.
To see the stack trace of this error execute with --v=5 or higher
error: your configuration file uses an old API spec:
"kubeadm.k8s.io/v1beta3" (kind: "ClusterConfiguration"). Please use kubeadm
v1.36 instead and run 'kubeadm config migrate --old-config old-config-file
--new-config new-config-file', which will write the new, similar spec using a
newer API version.
To see the stack trace of this error execute with --v=5 or higher
Error: kubeadm init failed!
```
### What to do about it?
Migrate the template to `kubeadm.k8s.io/v1beta4` — but **not
unconditionally**. `v1beta4` requires
kubeadm >= 1.31, and CKS still supports much older Kubernetes versions
(there are comparisons against
`1.15.0` and `1.16.0` in `KubernetesClusterUpgradeWorker.java:75` and
`KubernetesClusterService.MIN_KUBERNETES_VERSION_HA_SUPPORT`), with versions
registered by the operator.
A straight bump would break clusters on anything below 1.31.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
