andrijapanicsb opened a new issue, #14703:
URL: https://github.com/apache/cloudstack/issues/14703

   ### problem
   
   #### In short
   
   CloudStack accepts `createKubernetesCluster` with `etcdnodes` > 0 even when 
the Kubernetes ISO has no etcd in it. The etcd node fails silently, and the 
cluster sits in **Starting** for an hour, until the start timeout. During that 
hour it cannot be deleted, and the user gets no error saying why.
   
   #### Why it matters
   
   Dedicated etcd nodes (added in #9102) unpack `etcd/etcd-linux-amd64.tar.gz` 
from the Kubernetes ISO. The ISO build script adds that file only when its 
optional `ETCD_VERSION` argument is given, so not every published ISO has it:
   
   | Official ISO | etcd tarball |
   |---|---|
   | `setup-1.28.4.iso` | no |
   | `setup-v1.33.1-calico-x86_64.iso` | no (only `docker/etcd:3.5.21-0.tar`, 
the image kubeadm runs on control nodes) |
   | `setup-v1.35.4-calico-x86_64.iso` and `setup-v1.36.0-calico-x86_64.iso` | 
yes, etcd v3.6.10 |
   
   Nothing in CloudStack tells the user which registered versions can run etcd 
nodes. Someone who picks an older one waits an hour for a cluster that can 
never start, then has to search the cloud-init log inside the etcd VM to find 
out why.
   
   #### What happens (Kubernetes 1.28.4, `etcdnodes=1`)
   
   | Step | What happened | Expected |
   |---|---|---|
   | `createKubernetesCluster` | Accepted; all VMs deployed and Running | 
Rejected: this version's ISO has no etcd |
   | etcd node setup (`etcd-node.yml`) | `tar: 
/mnt/etcddisk//etcd/etcd-linux-amd64.tar.gz: Cannot open: No such file or 
directory`; `etcd.service` fails; the script carries on | Setup stops with a 
clear error |
   | Management server | `API endpoint for Kubernetes cluster ... not 
available` until the start timeout | Cluster moves to Error with the real 
reason |
   | `deleteKubernetesCluster` meanwhile | `Cannot perform delete operation on 
cluster ... in state: Starting` | — |
   | After the timeout | Cluster moved to Error; delete then worked | — |
   
   #### Cause
   
   - `validateManagedKubernetesClusterCreateParameters` has a minimum-version 
check for HA, but nothing for `etcdnodes`, and CloudStack records nothing about 
whether an ISO contains etcd.
   - `conf/etcd-node.yml` unpacks the tarball without checking that it exists, 
and ignores the failure.
   
   #### Suggested fix
   
   1. **Reject the request up front.** When a Kubernetes ISO becomes Ready, 
check once whether it contains `etcd/etcd-linux-amd64.tar.gz`. Reading the 
ISO's file list is enough. Store the result as an ISO detail in 
`vm_template_details`, for example `cks.etcd.included=true`. 
`createKubernetesCluster` with `etcdnodes` > 0 is rejected with a clear message 
when that detail is missing or false. Requests without etcd nodes are not 
checked. The version number cannot replace this check: 1.33.1 is newer than the 
feature and still has no etcd.
   2. **Fail fast at runtime.** `etcd-node.yml` checks that the tarball exists 
and stops with a clear error if it does not. During cluster start, CloudStack 
checks over SSH that etcd is running on each etcd node, with a bounded wait. If 
it is not running, the cluster moves to Error with that reason, instead of 
waiting for the API server until the timeout. This also covers etcd failing for 
any other reason.
   
   
   
   ### versions
   
   #### Environment
   
   Apache CloudStack main (`24.0.0-SNAPSHOT`), KVM, advanced zone, isolated 
network, Kubernetes 1.28.4 from `setup-1.28.4.iso`.
   
   ### The steps to reproduce the bug
   
   #### How to reproduce
   
   Add Kubernetes version 1.28.4 from `setup-1.28.4.iso` (or 1.33.1), then 
create a CKS cluster with it and **1 etcd node**. The cluster stays in Starting 
for an hour and cannot be deleted until it finally goes to Error.
   
   ### What to do about it?
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to