andrijapanicsb opened a new pull request, #14706:
URL: https://github.com/apache/cloudstack/pull/14706

   ### Description
   
   **In short:** a Kubernetes cluster with dedicated etcd nodes is now refused 
straight away when the Kubernetes ISO has no etcd in it. If something still 
goes wrong with etcd on a node, the cluster fails in minutes with the reason, 
not after an hour with none.
   
   Fixes: #14703
   
   #### Why it matters
   
   An operator registers a Kubernetes version from an ISO built without the 
optional etcd version (the ISO build script only adds etcd when asked). A user 
then creates a cluster with one or more etcd nodes. Today the request is 
accepted, the etcd nodes boot, find no etcd on the ISO and quietly do nothing. 
The cluster sits in Starting for the full one hour start timeout and then fails 
with no hint about the cause.
   
   #### What changes for the user
   
   | Situation | Before | With this PR |
   |---|---|---|
   | Cluster with etcd nodes, ISO has no etcd | Accepted, Starting for an hour, 
no reason | Rejected at once; the message names the version and the missing 
file |
   | Same, but the ISO could not be checked (storage VM down, direct download 
ISO) | Starting for an hour | Accepted; cluster goes to Error in about 10 
minutes with the reason |
   | ISO has etcd, but etcd does not start on a node | Starting for an hour, no 
reason | Error in about 10 minutes: etcd is not running on etcd node X, and why 
|
   | Cluster without etcd nodes | Works | Unchanged, no check is done |
   | ISO registered before the upgrade | n/a | Checked in the background 
shortly after the management server starts, or at the first request with etcd 
nodes |
   
   #### How it works
   
   Two parts, one commit each.
   
   **1. Refuse up front.** The secondary storage VM (the system VM that serves 
templates and ISOs) can now be asked whether an ISO contains a given file. The 
answer for the etcd archive is saved with the ISO, so each ISO is looked at 
only once. A background task looks at Kubernetes ISOs that have no answer yet. 
If there is still no answer when a cluster with etcd nodes is requested, the 
check runs right then. When the check cannot run, the request is allowed as 
before; this never blocks a cluster that would have worked.
   
   **2. Fail fast.** The etcd node setup script stops when the etcd archive is 
missing or broken and writes down why. During cluster start, the management 
server checks over SSH that etcd is running on every etcd node, within a 
bounded wait. If not, the cluster goes to Error with the reason.
   
   <details>
   <summary>Details</summary>
   
   Part 1
   - New `CheckFileInIsoCommand` / `CheckFileInIsoAnswer`. 
`NfsSecondaryStorageResource` lists the ISO with `isoinfo -R -f -i <iso>` (no 
mount) and looks for the path.
   - Result saved as template detail `cks.etcd.included` = `true` / `false` for 
`etcd/etcd-linux-amd64.tar.gz`. No detail means "not checked yet".
   - `KubernetesVersionManagerImpl` runs a background check every 5 minutes for 
Ready Kubernetes ISOs without the detail; it stops trying an ISO after 3 failed 
background checks (the check at request time still runs).
   - `createKubernetesCluster` with `etcdnodes` > 0: `true` goes ahead, `false` 
throws `InvalidParameterValueException`, unknown runs the check once and acts 
on it, and a check that cannot run lets the request through.
   - `listKubernetesSupportedVersions` returns `supportsetcd` once the ISO has 
been checked. The create cluster form shows a warning when etcd nodes are set 
and the selected version has `supportsetcd` = false.
   
   Part 2
   - `etcd-node.yml`: on a missing or broken archive, or when the ISO never 
shows up, the script writes the reason to `/home/cloud/etcd-setup-failed` and 
exits non-zero. `systemctl enable --now etcd` only runs when the setup worked.
   - `KubernetesClusterStartWorker.verifyEtcdRunningOnEtcdNodes` runs right 
after the ISO is attached. etcd has to be seen active on two checks in a row, 
so a service that keeps restarting does not pass.
   - New dynamic global setting `cloud.kubernetes.etcd.node.setup.timeout`, 
default 600 seconds. The wait never goes past the cluster start timeout.
   - On failure the ISO is detached from the VMs and the cluster goes to Error 
(`CreateFailed`).
   
   </details>
   
   ### Types of changes
   
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to change)
   - [ ] New feature (non-breaking change which adds functionality)
   - [x] Bug fix (non-breaking change which fixes an issue)
   - [ ] Enhancement (improves an existing feature and functionality)
   - [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
   - [ ] Build/CI
   - [ ] Test (unit or integration test code)
   
   ### Feature/Enhancement Scale or Bug Severity
   
   #### Bug Severity
   
   - [ ] BLOCKER
   - [ ] Critical
   - [x] Major
   - [ ] Minor
   - [ ] Trivial
   
   ### Screenshots (if appropriate):
   
   None yet.
   
   ### How Has This Been Tested?
   
   Draft: unit tests only so far. A live run is planned on an environment built 
from this PR's packages.
   
   | What | Result |
   |---|---|
   | Build of the Kubernetes plugin, the secondary storage module and what they 
depend on, with checkstyle | Passed |
   | Kubernetes plugin unit tests, 4 classes, 81 tests (29 new) | All passed |
   | Secondary storage unit tests, 31 tests (8 new) | 30 passed, including the 
8 new ones |
   | The 1 failure, `testExecuteQuerySnapshotZoneCopyCommand` | Not from this 
change: an existing test that fails on a Windows build machine because of the 
path separator |
   
   Not verified yet:
   - The ISO listing on a real secondary storage VM.
   - The SSH check against live etcd nodes on isolated, VPC and shared networks.
   - The UI warning in a running UI.
   - The case from the issue end to end (Kubernetes ISO without etcd plus one 
etcd node, and an ISO with etcd as the control).
   
   #### How did you try to break this feature and the system with this change?
   
   In unit tests: ISO missing on storage, non-NFS store, `isoinfo` failing, 
direct download ISO, no storage VM endpoint, an answer from an older agent that 
does not know the command, etcd that flips between running and stopped, an etcd 
node that cannot be reached over SSH, and a setup timeout longer than the 
cluster start timeout. In all cases where the ISO cannot be checked the request 
is let through, so an upgrade cannot start refusing clusters that worked before.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to