bhouse-nexthop commented on PR #14598:
URL: https://github.com/apache/cloudstack/pull/14598#issuecomment-6746838455
### Test report: the CKS add-on scripts' download and option handling, on a
live cluster
**Environment**
- A CloudStack 24.0 (main) build with this PR, alongside some of my other
open PRs. The PR targets 4.22; it wasn't run on a 4.22 build. The three scripts
are the same on 4.22 and main, and they run on the cluster nodes, not in the
management server, so the results carry over.
- Three KVM hosts, Ceph RBD primary storage, one management server.
- A CKS cluster with 3 control nodes and 2 workers, Kubernetes v1.32.5,
isolated network. It was created on 4.20, so its nodes held the old scripts.
- The nodes can reach github.com.
**Setup**
- I took the three scripts from the management server's jar on this build
and checked that each has the `curl -fsSL` download.
- I installed them in `/opt/bin` on the first control node, which is where
CloudStack runs them. I kept the node's originals, and put them back at the end.
- **Deviation.** The cluster wasn't recreated, so I installed the scripts by
hand rather than having CloudStack copy them at create time. In S2 and S5
CloudStack still runs the node's copy, and on a failure it copies the build's
scripts and runs them again.
**Results**
| Scenario | Result |
|---|---|
| S1: `deploy-csi-driver` on a node left with only `snapshot-crds.yaml` (the
old script's broken state) | pass: downloads `manifest.yaml` as well and
applies both. Exit 1 comes from the upstream manifest (see the note) |
| S1b: CSI manifest URL returns 404 | pass: exit 1 with the script's error,
and no `manifest.yaml` or temporary file left |
| S1c: the next run after the failed download | pass: downloads both files
again, exit 0 |
| S2: enabling autoscaling through the API with GitHub unreachable and no
local template | pass: the API call fails with the script's error, no file is
left, and the cluster stays Running |
| S3: `autoscale-kube-cluster -e` with a 404 URL | pass: exit 1, no file
left |
| S4: disabling autoscaling through the API with no autoscaler deployed |
pass: succeeds (failed before, with NotFound under `bash -e`) |
| S5: enable 2..4 through the API (download path), then disable | pass: the
deployment is created, then deleted |
| S5b: the script's flags (`-e`, no flag, `--disable`, `--disable` again) |
pass: 0, 1 (deletes nothing), 0, 0 |
| S6: `deploy-provider` with a 404 URL, then the real URL | pass: 1 with no
file left; then 0 with `provider.yaml` in place |
**S1: `deploy-csi-driver` recovers from the old script's state**
- The node starts with only `/opt/csi/snapshot-crds.yaml`, which is what the
old script left behind. The new script sees that `manifest.yaml` is missing,
downloads both files and applies them:
```
-rw-r--r-- 1 root root 18128 manifest.yaml
-rw-r--r-- 1 root root 57074 snapshot-crds.yaml
customresourcedefinition.../volumesnapshots.snapshot.storage.k8s.io created
...
deployment.apps/cloudstack-csi-controller created
daemonset.apps/cloudstack-csi-node created
Error from server (BadRequest): error when creating
"/opt/csi/manifest.yaml": CSIDriver in version "v1" cannot be handled as a
CSIDriver: strict decoding error: unknown field "roleRef", unknown field
"subjects"
rc=1
```
The controller and node pods came up (7/7 and 3/3 Running).
- **The exit 1 isn't from this PR.** The upstream v3.0.0 `manifest.yaml` has
a document separator glued to the end of a line (`apiGroup:
rbac.authorization.k8s.io---`). That merges a ClusterRoleBinding into the
CSIDriver document, and the API server rejects the result. The ISO path applies
the same file, so it fails the same way. I have written this up separately.
- A second run returns 0 at once, because the controller is present.
- **404:** with the manifest URL pointed at a missing file and the files
removed first, the run printed `curl: (22) The requested URL returned error:
404` and the script's message, and exited 1. Only `snapshot-crds.yaml` was in
`/opt/csi`, with no `manifest.yaml` and no `.tmp.*` file.
- **Recovery:** the next run downloaded both again and exited 0. That run
used a copy with the "controller already present" check removed, as the
controller was running.
**S2 and S4: autoscaling through the API**
- To make GitHub unreachable, I pointed `raw.githubusercontent.com` and
`github.com` at 127.0.0.1 in the node's `/etc/hosts` and moved
`autoscaler_tmpl.yaml` aside.
```
scale autoscalingenabled=true minsize=2 maxsize=4: FAILED: "Failed to
autoscale Kubernetes cluster: ... : Unable to connect to the internet to
download the autoscaler deployment and image
curl: (7) Failed to connect to raw.githubusercontent.com ..."
```
`/opt/autoscaler` was empty afterwards (no template, no `.tmp` file), and
the cluster stayed Running with autoscaling off.
- With nothing deployed, `scale autoscalingenabled=false` succeeded. Before
the PR, `kubectl delete` returned NotFound under `bash -e`, and the API call
failed.
**S5: the real enable and disable, and the flags**
- With the hosts entries removed, enable 2..4 created
`deployment.apps/cluster-autoscaler` and wrote `autoscaler_now.yaml` and
`autoscaler_tmpl.yaml`. Disable removed the deployment.
- Running the script directly:
```
-e -> rc=0, deployment created
(no flag) -> ERROR: one of -e/--enable or -d/--disable is required. -h
or --help for help rc=1, deployment still there
--disable -> deployment.apps "cluster-autoscaler" deleted rc=0
--disable -> rc=0 (nothing to delete)
```
**S6: `deploy-provider`**
- The cloud controller manager was already running. I used a copy with the
"already running" check removed and its target directory moved to `/tmp/prov`,
so the node's own `/opt/provider` stayed as it was.
- With a 404 URL: exit 1, and `/tmp/prov` was empty.
- With the real URL: exit 0, `provider.yaml` was in place, and the apply ran
as a server-side dry run.
- The real script on the node exits 0 at once, because the controller
manager is present.
**Cleanup.** I removed the CSI driver, the autoscaler and the `/etc/hosts`
entries, put the autoscaler template back and restored the node's original
scripts (checksums checked). The cluster was Running throughout.
**Result:** pass. Downloads that fail leave nothing behind for a later run
to apply. A node left with only `snapshot-crds.yaml` recovers. `--disable`, a
missing flag, and disabling with nothing deployed behave as described. The only
non-zero exit on a good download comes from the upstream CSI manifest, not from
the script.
**Not covered**
- A fresh cluster, where CloudStack copies the scripts at create time. Here
they were installed by hand.
- `deploy-csi-driver` run from the API with `csienabled=true`. The script
was run directly.
- A node that already holds an error page under `/opt` from the old scripts.
The PR leaves that as a manual cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
