bhouse-nexthop opened a new pull request, #14649:
URL: https://github.com/apache/cloudstack/pull/14649
### Description
This PR pins the cluster autoscaler and the cloud controller manager (CCM)
that `scripts/util/create-kubernetes-binaries-iso.sh` bundles into the CKS
binaries ISO, and writes a list of the ISO's contents into the ISO. Because the
autoscaler now changes with the Kubernetes version, a cluster upgrade also
redeploys the autoscaler from the new ISO.
Fixes #14648
Related: #14598 fixes the same download problems in the node-side scripts
(`deploy-provider`, `autoscale-kube-cluster`, `deploy-csi-driver`). This PR
doesn't touch those scripts, so the two don't conflict.
#### What was wrong
- **Autoscaler:**
- The manifest came from `kubernetes/autoscaler` `master`, and its image
is `apache/cloudstack-kubernetes-autoscaler:latest`.
- That tag was last pushed in March 2022 and is a 1.24.0-alpha development
build. So every ISO shipped the same old autoscaler, whatever its Kubernetes
version.
- The cluster-autoscaler only supports the Kubernetes minor version it was
released with.
- **CCM:**
- The manifest came from `cloudstack-kubernetes-provider` `main`, with
whatever image `main` named on the day of the build.
- For a long time that was v1.1.0, which does not initialise nodes
(apache/cloudstack-kubernetes-provider#58, fixed in v1.2.0).
- **No record:** nothing in the ISO said which versions it held, so two ISOs
for the same Kubernetes version could differ.
- **Silent download failures:** most downloads used `curl` without `-f`, so
a 404 page was saved and bundled as a manifest or binary.
#### What changed
**Autoscaler (ISO builder)**
- By default, the script picks the newest
`cluster-autoscaler-<minor>.<patch>` release tag for the ISO's Kubernetes minor
version.
- It finds the tag with `git ls-remote`, so no API token is needed.
- Today, 1.31.x gets 1.31.5 and 1.33.x gets 1.33.6.
- `AUTOSCALER_VERSION=1.33.4` picks an exact release instead.
- The script warns if the release is for a different minor version.
- The build stops with an error if no release exists for the minor
version, for example a new Kubernetes minor before its autoscaler release.
- The manifest comes from that release tag.
- I checked that the CloudStack example
(`cloudprovider/cloudstack/examples/cluster-autoscaler-standard.yaml`) exists
at the latest tag of every minor from 1.20 to 1.37, so no fallback manifest is
needed.
- Before 1.20 the example doesn't exist at the release tags, so builds for
those (long EOL) versions now fail on the download.
- The image is replaced with the upstream image of the same release,
`registry.k8s.io/autoscaling/cluster-autoscaler:v<version>`.
- The example names the unversioned
`apache/cloudstack-kubernetes-autoscaler:latest` at every tag, so it can't be
used as is.
- The upstream image is built with all cloud providers, CloudStack
included.
- It runs `./cluster-autoscaler` from `/`, as the manifest expects.
- It is published for amd64 and arm64.
- **Extra RBAC.**
- The CloudStack example's ClusterRole hasn't kept up with newer releases.
They also watch VolumeAttachments (from 1.32) and DRA objects (DRA is on by
default from 1.35).
- Without read access to those, the autoscaler never finishes syncing its
caches and never scales (kubernetes/autoscaler#7663,
kubernetes/autoscaler#8678). The AWS examples were fixed the same way.
- The script appends a `cluster-autoscaler-extra` ClusterRole and binding
with that read access. At 1.36.2 that makes the CloudStack example's
permissions match the AWS example's.
- It only adds read permissions, so it is harmless on older versions.
**CCM (ISO builder)**
- The default is v1.2.0, the current release and the first one that
initialises nodes.
- `CCM_VERSION=v1.3.0-rc1` picks another release tag.
- The manifest comes from that tag, and its image is set to
`apache/cloudstack-kubernetes-provider:<tag>`. This is needed because a tag's
manifest doesn't always name its own image: `v1.3.0-rc1`'s names v1.2.0.
**`versions.txt` in the ISO**
- It lists the Kubernetes, CNI plugins, crictl, CCM, CSI driver, autoscaler
and etcd versions.
- It also lists the URL of every file downloaded, and every bundled image
with its digest.
- It is not a `.yaml` file, so the node scripts that copy `*.yaml` from the
ISO ignore it.
**Download and build robustness (ISO builder)**
- Downloads use `curl -fsSL --retry 3`, one file per call, so a failed
download stops the build. A multi-URL `curl` would only report the last file's
status.
- The `curl | sed` pipelines for `kubelet.service` and `10-kubeadm.conf` now
download first and then edit, so a failed download is no longer hidden by the
pipe.
- The CNI fallback to the old archive name is kept.
- The CSI and etcd downloads move from `wget` to the same helper.
- The ISO is assembled in a `mktemp -d` directory, removed on exit, instead
of a fixed `/tmp/iso`. Files left behind by a failed build no longer end up in
the next ISO.
- The image list is deduplicated before pulling. The CSI manifest names two
images twice.
- Pulled images are removed with `ctr image rm --sync`.
- Without `--sync`, containerd's asynchronous garbage collection could
delete a layer that the next image shares, between that image's pull and its
export.
- That failed my test builds with `ctr: failed to get reader: content
digest ...: not found`.
- The positional arguments are unchanged. Both overrides are environment
variables.
**Upgrade (`KubernetesClusterUpgradeWorker`)**
- `upgrade-kubernetes.sh` already copies the new ISO's `autoscaler.yaml` to
the control node, but the running autoscaler was never touched. With `:latest`
everywhere, that didn't matter.
- Now the autoscaler's image changes with the Kubernetes version. After all
nodes are upgraded and the new version is saved, a cluster with autoscaling
enabled gets its autoscaler redeployed from the new ISO.
- It runs `autoscale-kube-cluster -e` with the cluster's id, min and max
from the database, the same command as enabling autoscaling.
- It runs while the cluster is still `Upgrading`, so autoscaling can't be
changed through the API in the meantime.
- A failure is logged as a warning and leaves the previous autoscaler
running. It doesn't fail the upgrade.
- The command string is now built by one helper that both callers share.
Not changed here:
- **Node-side fallbacks:** `autoscale-kube-cluster` and `deploy-provider`
still download from `master`/`main` when the ISO doesn't have the manifests.
#14598 changes those same lines, so pinning them is left until it is merged.
- **Kubelet unit files:** `kubelet.service` and `10-kubeadm.conf` still come
from `shapeblue/cloudstack-nonoss` `main`. Their URLs are now recorded in
`versions.txt`.
- **aarch64 ISOs:** images are still pulled for the build host's platform,
and etcd is always the amd64 archive.
- **CCM on upgrade:** it is not redeployed during an upgrade, as before.
- **Upgrading onto an old ISO:** an upgrade to a version whose ISO was built
by the old script redeploys that ISO's `:latest` autoscaler. That is the same
autoscaler a cluster created from that ISO would get.
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [x] Major
- [ ] Minor
- [ ] Trivial
### How Has This Been Tested?
**ISO builds.** I built ISOs with the modified script on an Ubuntu x86_64
host with containerd 2.x. I then extracted `versions.txt`, `autoscaler.yaml`,
`provider.yaml` and the image tarballs from each ISO with `isoinfo`.
| Build | Autoscaler (manifest tag / image) | CCM (manifest tag / image) |
|---|---|---|
| 1.31.14, defaults | 1.31.5 /
`registry.k8s.io/autoscaling/cluster-autoscaler:v1.31.5` | v1.2.0 /
`apache/cloudstack-kubernetes-provider:v1.2.0` |
| 1.33.13, defaults, with etcd 3.5.21 | 1.33.6 /
`...cluster-autoscaler:v1.33.6` | v1.2.0 / `...:v1.2.0` |
| 1.33.13, `AUTOSCALER_VERSION=v1.33.4 CCM_VERSION=1.3.0-rc1` | 1.33.4 /
`...cluster-autoscaler:v1.33.4` | v1.3.0-rc1 / `...:v1.3.0-rc1` (the tag's
manifest named v1.2.0) |
The 1.31 build was done with Calico 3.29.3 and Dashboard 2.7.0. Later builds
used flannel and Headlamp manifests, because Docker Hub's anonymous pull limit
stopped the Calico/Dashboard builds partway.
For each ISO I checked the following:
- `autoscaler.yaml` and `provider.yaml` name exactly those images, with
`imagePullPolicy: IfNotPresent`.
- The `<min>:<max>:<cluster-id>` placeholders are intact.
- The autoscaler manifest parses into the six original objects plus the
extra ClusterRole and binding.
- `docker/` holds `cluster-autoscaler:v<version>.tar` and
`cloudstack-kubernetes-provider:<tag>.tar`, and the image name inside each
tarball matches the manifest.
- `versions.txt` has the right versions and URLs.
- There is one line per image, each with a 64-hex digest.
- The etcd version and URL appear only when etcd was requested.
- The number of image tarballs equals the number of image lines.
- No build directory is left behind.
**Failure cases.** Each of these stops the build with `curl: (22) ... 404`
and a non-zero exit:
- `CCM_VERSION=v9.9.9`
- `AUTOSCALER_VERSION=1.33.99`
- a dashboard URL that doesn't exist
**Builder pieces tested on their own:**
- A minor version with no autoscaler release (1.99) stops with the error
asking for `AUTOSCALER_VERSION`.
- `AUTOSCALER_VERSION=1.32.7` on 1.33 prints the mismatch warning.
- CNI v0.7.5 falls back to the old `cni-plugins-amd64-<ver>.tgz` name.
**Unit tests.** New tests in `KubernetesClusterUpgradeWorkerTest` cover
`updateAutoscaler`:
- it does nothing when autoscaling is disabled or min/max is unset;
- it runs `autoscale-kube-cluster -i <uuid> -e -M <max> -m <min>`;
- it doesn't throw when SSH fails or returns an error.
All 82 tests in the `kubernetes-service` module pass, and checkstyle is
clean.
**Not tested:**
- Deploying these ISOs on a live CKS cluster.
- In particular, running the upstream autoscaler image with the CloudStack
provider against a real cluster, and a real upgrade that redeploys it.
- The RBAC addition is based on the autoscaler source and the upstream
issues above.
#### How did you try to break this feature and the system with this change?
- **Existing ISOs and clusters:** unaffected at create time. The node side
reads the same file names (`autoscaler.yaml`, `provider.yaml`, `docker/*.tar`).
- **Existing build commands:** the positional arguments are unchanged.
Without the new variables, builds get version-matched defaults instead of
`master`/`main`.
- **A minor version before its autoscaler release:** the build fails with a
clear message instead of bundling a mismatched autoscaler. `AUTOSCALER_VERSION`
overrides it.
- **A change to an upstream manifest's layout:** setting the image checks
that the manifest has exactly one `image:` line and fails otherwise, instead of
silently bundling the wrong image.
- **A manifest without a trailing newline:** the appended RBAC starts with a
blank line, so `---` can't join the last line.
- **Upgrade with autoscaling disabled:** nothing changes.
- **Upgrade where the redeploy fails:** a warning is logged and the upgrade
still succeeds with the old autoscaler running.
- **On an air-gapped cluster:** the redeploy runs only after every node in
the upgrade has imported the new ISO's images. Nodes marked for manual upgrade
are the exception, but the rolling update keeps the old pod until the new one
is ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
