andrijapanicsb opened a new pull request, #14706: URL: https://github.com/apache/cloudstack/pull/14706
### Description **In short:** a Kubernetes cluster with dedicated etcd nodes is now refused straight away when the Kubernetes ISO has no etcd in it. If something still goes wrong with etcd on a node, the cluster fails in minutes with the reason, not after an hour with none. Fixes: #14703 #### Why it matters An operator registers a Kubernetes version from an ISO built without the optional etcd version (the ISO build script only adds etcd when asked). A user then creates a cluster with one or more etcd nodes. Today the request is accepted, the etcd nodes boot, find no etcd on the ISO and quietly do nothing. The cluster sits in Starting for the full one hour start timeout and then fails with no hint about the cause. #### What changes for the user | Situation | Before | With this PR | |---|---|---| | Cluster with etcd nodes, ISO has no etcd | Accepted, Starting for an hour, no reason | Rejected at once; the message names the version and the missing file | | Same, but the ISO could not be checked (storage VM down, direct download ISO) | Starting for an hour | Accepted; cluster goes to Error in about 10 minutes with the reason | | ISO has etcd, but etcd does not start on a node | Starting for an hour, no reason | Error in about 10 minutes: etcd is not running on etcd node X, and why | | Cluster without etcd nodes | Works | Unchanged, no check is done | | ISO registered before the upgrade | n/a | Checked in the background shortly after the management server starts, or at the first request with etcd nodes | #### How it works Two parts, one commit each. **1. Refuse up front.** The secondary storage VM (the system VM that serves templates and ISOs) can now be asked whether an ISO contains a given file. The answer for the etcd archive is saved with the ISO, so each ISO is looked at only once. A background task looks at Kubernetes ISOs that have no answer yet. If there is still no answer when a cluster with etcd nodes is requested, the check runs right then. When the check cannot run, the request is allowed as before; this never blocks a cluster that would have worked. **2. Fail fast.** The etcd node setup script stops when the etcd archive is missing or broken and writes down why. During cluster start, the management server checks over SSH that etcd is running on every etcd node, within a bounded wait. If not, the cluster goes to Error with the reason. <details> <summary>Details</summary> Part 1 - New `CheckFileInIsoCommand` / `CheckFileInIsoAnswer`. `NfsSecondaryStorageResource` lists the ISO with `isoinfo -R -f -i <iso>` (no mount) and looks for the path. - Result saved as template detail `cks.etcd.included` = `true` / `false` for `etcd/etcd-linux-amd64.tar.gz`. No detail means "not checked yet". - `KubernetesVersionManagerImpl` runs a background check every 5 minutes for Ready Kubernetes ISOs without the detail; it stops trying an ISO after 3 failed background checks (the check at request time still runs). - `createKubernetesCluster` with `etcdnodes` > 0: `true` goes ahead, `false` throws `InvalidParameterValueException`, unknown runs the check once and acts on it, and a check that cannot run lets the request through. - `listKubernetesSupportedVersions` returns `supportsetcd` once the ISO has been checked. The create cluster form shows a warning when etcd nodes are set and the selected version has `supportsetcd` = false. Part 2 - `etcd-node.yml`: on a missing or broken archive, or when the ISO never shows up, the script writes the reason to `/home/cloud/etcd-setup-failed` and exits non-zero. `systemctl enable --now etcd` only runs when the setup worked. - `KubernetesClusterStartWorker.verifyEtcdRunningOnEtcdNodes` runs right after the ISO is attached. etcd has to be seen active on two checks in a row, so a service that keeps restarting does not pass. - New dynamic global setting `cloud.kubernetes.etcd.node.setup.timeout`, default 600 seconds. The wait never goes past the cluster start timeout. - On failure the ISO is detached from the VMs and the cluster goes to Error (`CreateFailed`). </details> ### Types of changes - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] New feature (non-breaking change which adds functionality) - [x] Bug fix (non-breaking change which fixes an issue) - [ ] Enhancement (improves an existing feature and functionality) - [ ] Cleanup (Code refactoring and cleanup, that may add test cases) - [ ] Build/CI - [ ] Test (unit or integration test code) ### Feature/Enhancement Scale or Bug Severity #### Bug Severity - [ ] BLOCKER - [ ] Critical - [x] Major - [ ] Minor - [ ] Trivial ### Screenshots (if appropriate): None yet. ### How Has This Been Tested? Draft: unit tests only so far. A live run is planned on an environment built from this PR's packages. | What | Result | |---|---| | Build of the Kubernetes plugin, the secondary storage module and what they depend on, with checkstyle | Passed | | Kubernetes plugin unit tests, 4 classes, 81 tests (29 new) | All passed | | Secondary storage unit tests, 31 tests (8 new) | 30 passed, including the 8 new ones | | The 1 failure, `testExecuteQuerySnapshotZoneCopyCommand` | Not from this change: an existing test that fails on a Windows build machine because of the path separator | Not verified yet: - The ISO listing on a real secondary storage VM. - The SSH check against live etcd nodes on isolated, VPC and shared networks. - The UI warning in a running UI. - The case from the issue end to end (Kubernetes ISO without etcd plus one etcd node, and an ISO with etcd as the control). #### How did you try to break this feature and the system with this change? In unit tests: ISO missing on storage, non-NFS store, `isoinfo` failing, direct download ISO, no storage VM endpoint, an answer from an older agent that does not know the command, etcd that flips between running and stopped, an etcd node that cannot be reached over SSH, and a setup timeout longer than the cluster start timeout. In all cases where the ISO cannot be checked the request is let through, so an upgrade cannot start refusing clusters that worked before. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
