andrijapanicsb opened a new issue, #14703: URL: https://github.com/apache/cloudstack/issues/14703
### problem #### In short CloudStack accepts `createKubernetesCluster` with `etcdnodes` > 0 even when the Kubernetes ISO has no etcd in it. The etcd node fails silently, and the cluster sits in **Starting** for an hour, until the start timeout. During that hour it cannot be deleted, and the user gets no error saying why. #### Why it matters Dedicated etcd nodes (added in #9102) unpack `etcd/etcd-linux-amd64.tar.gz` from the Kubernetes ISO. The ISO build script adds that file only when its optional `ETCD_VERSION` argument is given, so not every published ISO has it: | Official ISO | etcd tarball | |---|---| | `setup-1.28.4.iso` | no | | `setup-v1.33.1-calico-x86_64.iso` | no (only `docker/etcd:3.5.21-0.tar`, the image kubeadm runs on control nodes) | | `setup-v1.35.4-calico-x86_64.iso` and `setup-v1.36.0-calico-x86_64.iso` | yes, etcd v3.6.10 | Nothing in CloudStack tells the user which registered versions can run etcd nodes. Someone who picks an older one waits an hour for a cluster that can never start, then has to search the cloud-init log inside the etcd VM to find out why. #### What happens (Kubernetes 1.28.4, `etcdnodes=1`) | Step | What happened | Expected | |---|---|---| | `createKubernetesCluster` | Accepted; all VMs deployed and Running | Rejected: this version's ISO has no etcd | | etcd node setup (`etcd-node.yml`) | `tar: /mnt/etcddisk//etcd/etcd-linux-amd64.tar.gz: Cannot open: No such file or directory`; `etcd.service` fails; the script carries on | Setup stops with a clear error | | Management server | `API endpoint for Kubernetes cluster ... not available` until the start timeout | Cluster moves to Error with the real reason | | `deleteKubernetesCluster` meanwhile | `Cannot perform delete operation on cluster ... in state: Starting` | — | | After the timeout | Cluster moved to Error; delete then worked | — | #### Cause - `validateManagedKubernetesClusterCreateParameters` has a minimum-version check for HA, but nothing for `etcdnodes`, and CloudStack records nothing about whether an ISO contains etcd. - `conf/etcd-node.yml` unpacks the tarball without checking that it exists, and ignores the failure. #### Suggested fix 1. **Reject the request up front.** When a Kubernetes ISO becomes Ready, check once whether it contains `etcd/etcd-linux-amd64.tar.gz`. Reading the ISO's file list is enough. Store the result as an ISO detail in `vm_template_details`, for example `cks.etcd.included=true`. `createKubernetesCluster` with `etcdnodes` > 0 is rejected with a clear message when that detail is missing or false. Requests without etcd nodes are not checked. The version number cannot replace this check: 1.33.1 is newer than the feature and still has no etcd. 2. **Fail fast at runtime.** `etcd-node.yml` checks that the tarball exists and stops with a clear error if it does not. During cluster start, CloudStack checks over SSH that etcd is running on each etcd node, with a bounded wait. If it is not running, the cluster moves to Error with that reason, instead of waiting for the API server until the timeout. This also covers etcd failing for any other reason. ### versions #### Environment Apache CloudStack main (`24.0.0-SNAPSHOT`), KVM, advanced zone, isolated network, Kubernetes 1.28.4 from `setup-1.28.4.iso`. ### The steps to reproduce the bug #### How to reproduce Add Kubernetes version 1.28.4 from `setup-1.28.4.iso` (or 1.33.1), then create a CKS cluster with it and **1 etcd node**. The cluster stays in Starting for an hour and cannot be deleted until it finally goes to Error. ### What to do about it? _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
