kiranchavala opened a new pull request, #14183:
URL: https://github.com/apache/cloudstack/pull/14183
### Description
The control node's `/etc/kubernetes/kubeadm-config.yaml` is written with the
`kubeadm.k8s.io/v1beta3` configuration API. kubeadm **removed v1beta3 in
Kubernetes 1.37**, so `kubeadm init --config` now rejects the file outright:
```
error: your configuration file uses an old API spec:
"kubeadm.k8s.io/v1beta3" (kind: "ClusterConfiguration").
Please use kubeadm v1.36 instead and run 'kubeadm config migrate
--old-config old-config-file --new-config new-config-file',
which will write the new, similar spec using a newer API version.
Error: kubeadm init failed!
```
This PR switches both documents in that template to `kubeadm.k8s.io/v1beta4`.
**Scope: external-etcd clusters only.** That configuration file is passed to
kubeadm on exactly one path:
| Path | kubeadm invocation | Affected |
|---|---|---|
| Control node, external etcd (`k8s-control-node.yml:300`) | `init --config
/etc/kubernetes/kubeadm-config.yaml` | **yes** |
| Control node, stacked etcd (`:302`) | `init --token ... {{ initargs }}` |
no |
| Additional control plane (`k8s-control-node-add.yml:239`) | `join
--control-plane` | no |
| Worker (`k8s-node.yml:254`) | `join` | no |
`grep -rn "kubeadm.k8s.io/v1beta" plugins/ systemvm/` returns only the two
lines changed here.
Only the `apiVersion` changes. The breaking difference in v1beta4 is that
`extraArgs` became a list of `{name, value}` rather than a map, and neither
document written here sets `extraArgs`; everything they do set
(`apiServer.certSANs`, `controlPlaneEndpoint`, `etcd.external.endpoints`,
`bootstrapTokens`, `nodeRegistration.criSocket`, `localAPIEndpoint`,
`certificateKey`) is unchanged between the two versions.
**The failure is also hard to diagnose**, which is worth noting
independently of the fix:
- `kubeadm init` runs from the `deploy-kube-system` **systemd unit**
(`k8s-control-node.yml:318-330`), so its output only reaches the journal.
- `runcmd` ends with `systemctl start deploy-kube-system` on a `Type=simple`
unit, so the call returns as soon as the process forks and cloud-init reports
success — `Ran 13 modules with 0 failures`, `Cloud-init ... finished`.
- The unit has `Restart=on-failure` with `StartLimitInterval=0`, so after
`deploy-kube-system` exhausts its own 3 retries and exits 1, systemd restarts
it and the cycle repeats indefinitely.
An operator therefore sees a cluster that never becomes ready, a clean
`cloud-init.log`, and must know to run `journalctl -u deploy-kube-system` to
find the cause.
Fixes: #14181
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Feature/Enhancement Scale
- [ ] Major
- [ ] Minor
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [x] Major
- [ ] Minor
- [ ] Trivial
### Screenshots (if appropriate):
### How Has This Been Tested?
Tested on CloudStack 4.22 / KVM with Kubernetes **v1.37.0** (`kubeadm
version -o short` on the control node).
**Before** — creating a cluster with external etcd nodes, `kubeadm init`
fails with the error above, repeated for each of the three
`MAX_SETUP_CRUCIAL_CMD_ATTEMPTS` retries and then indefinitely by systemd:
```
root@test-cks-etcd-control-...:/opt/bin# ./deploy-kube-system
error: your configuration file uses an old API spec:
"kubeadm.k8s.io/v1beta3" (kind: "ClusterConfiguration").
...
Error: kubeadm init failed!
```
**After** — with the template changed to v1beta4, `kubeadm init --config`
completes and the control plane comes up:
```
kube-system kube-apiserver-<control-node> 1/1 Running
kube-system kube-controller-manager-<control-node> 1/1 Running
kube-system kube-scheduler-<control-node> 1/1 Running
kube-system kube-proxy-<...> 1/1 Running
```
with no `etcd-*` pod present, confirming the external-etcd path was the one
exercised.
**Control case** — on the same environment and the same Kubernetes version,
a cluster created *without* external etcd nodes deploys normally on the
unmodified template, because `kubeadm-config.yaml` is written but never read.
That isolates the fault to the `kubeadm init --config` branch.
#### How did you try to break this feature and the system with this change?
- Checked every other kubeadm invocation in the CKS templates. The
stacked-etcd `init` and both `join` paths pass command-line flags and never
read a configuration file, so they are unaffected by this change in either
direction.
- Verified `grep -rn "kubeadm.k8s.io/v1beta" plugins/ systemvm/` has no
remaining occurrences outside this file.
- Confirmed the changed file is still valid cloud-init YAML and that the
templated placeholders in the surrounding documents are untouched.
### Open question for reviewers
**This change assumes external-etcd clusters run Kubernetes >= 1.31.**
`v1beta4` was introduced in kubeadm 1.31, so an external-etcd cluster on
1.28–1.30 would now fail in the opposite direction. There is currently no
minimum-version guard on external etcd (the feature arrived in "CKS
Enhancements" #9102), so that combination is reachable in principle.
If older Kubernetes versions must keep working with external etcd, the
`apiVersion` needs to be selected per cluster version rather than hardcoded —
for example by templating it from `KubernetesClusterStartWorker` using
`KubernetesVersionManagerImpl.compareSemanticVersions()` against a `1.31.0`
threshold, which is the pattern already used for
`MIN_KUBERNETES_VERSION_HA_SUPPORT`. Happy to take that approach instead if
preferred; it was kept out of this PR to keep the change minimal.
Relatedly, CKS enforces no **maximum** supported Kubernetes version, so
`addKubernetesSupportedVersion` accepts 1.37 and the failure only surfaces
part-way through provisioning. A supported-version guard at registration or
cluster-creation time would turn this and similar issues into a clear up-front
error.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
