bhouse-nexthop commented on PR #14649:
URL: https://github.com/apache/cloudstack/pull/14649#issuecomment-6756505549
### Test report: pinned autoscaler and CCM in the binaries ISO, on a live
cluster
**Summary:** I built two ISOs with this PR's
`create-kubernetes-binaries-iso.sh` and registered both. A cluster created from
the first runs the autoscaler release that matches its Kubernetes minor
version, and the CCM at v1.2.0. The autoscaler syncs with no RBAC or cache
errors. It scaled the cluster up from 1 to 3 workers for Pending pods, and back
down to 1 once they were gone. Upgrading the cluster to the second ISO
redeployed the autoscaler with the newer image, keeping the cluster's min, max
and id.
**Environment**
- A CloudStack 24.0 (main) build with this PR, alongside some of my other
open PRs. The management server and all three agents were upgraded in place to
this build.
- The PR targets 4.22; it wasn't run on a 4.22 build. The ISO script and
`KubernetesClusterUpgradeWorker` are the same on both branches apart from this
PR.
- Three KVM hosts, Ceph RBD primary storage, one management server.
- ISOs built on an Ubuntu 24.04 x86_64 host with containerd's `ctr`. The
arguments were the same as for the lab's existing ISOs: CNI 1.9.1, Calico
v3.30.7 and Headlamp 0.40.1.
- A fresh cluster with 1 control node and 1-3 workers, the CKS system VM
template and an isolated network. Autoscaling ran with min 1 and max 3.
**Results**
| Check | Result |
|---|---|
| Build `setup-v1.32.13` and `setup-v1.33.13` with defaults | pass, 139 s
and 113 s |
| `versions.txt` in each ISO | pass. It lists the versions, URLs and image
digests |
| Autoscaler and CCM pinned in the ISOs | pass: 1.32.13 ->
`cluster-autoscaler:v1.32.7` and 1.33.13 -> `v1.33.6`; CCM `v1.2.0` in both |
| Cluster created from the 1.32.13 ISO, CCM | pass.
`apache/cloudstack-kubernetes-provider:v1.2.0` runs, and a `LoadBalancer`
Service gets a CloudStack LB rule |
| Enable autoscaling (1..3) | pass. Pinned image, `--nodes=1:3:<cluster
id>`, no RBAC or cache-sync errors |
| Scale-up on Pending pods | pass, 1 -> 3 workers; all pods Running 232 s
after they were created |
| Scale-down when idle | pass, 3 -> 1 workers 654 s after the pods were
deleted (default idle times) |
| Upgrade 1.32.13 -> 1.33.13 (the second ISO) | pass, 323 s. The autoscaler
was redeployed with `v1.33.6`, keeping `--nodes=1:3:<cluster id>` |
| Node `providerID` and the `uninitialized` taint | not applicable to CKS;
see the CCM section |
**ISOs**
- Each ISO has `versions.txt` at the root, next to `autoscaler.yaml` and
`provider.yaml`. The 1.33.13 one:
```
AUTOSCALER_VERSION=1.33.6
AUTOSCALER_MANIFEST_URL=https://raw.githubusercontent.com/kubernetes/autoscaler/cluster-autoscaler-1.33.6/cluster-autoscaler/cloudprovider/cloudstack/examples/cluster-autoscaler-standard.yaml
AUTOSCALER_IMAGE=registry.k8s.io/autoscaling/cluster-autoscaler:v1.33.6
CCM_VERSION=v1.2.0
CCM_MANIFEST_URL=https://raw.githubusercontent.com/apache/cloudstack-kubernetes-provider/v1.2.0/deployment.yaml
CCM_IMAGE=apache/cloudstack-kubernetes-provider:v1.2.0
...
IMAGE=registry.k8s.io/autoscaling/cluster-autoscaler:v1.33.6@sha256:7b44e0f0...
IMAGE=docker.io/apache/cloudstack-kubernetes-provider:v1.2.0@sha256:09d3042f...
```
It also has the Kubernetes, CNI, crictl, Calico, Headlamp, CSI and kubelet
unit URLs, and one `IMAGE=` line with a digest for every bundled image.
- `docker/` has `cluster-autoscaler_v1.32.7.tar` and
`cluster-autoscaler_v1.33.6.tar` respectively, and
`cloudstack-kubernetes-provider_v1.2.0.tar` in both.
- `autoscaler.yaml` names only the pinned image, with `imagePullPolicy:
IfNotPresent`. It keeps `--nodes=<min>:<max>:<cluster-id>`, and has the six
objects of the upstream example plus the `cluster-autoscaler-extra` ClusterRole
and binding: read access to `volumeattachments`, `deviceclasses`,
`resourceslices` and `resourceclaims`.
- `provider.yaml` names `apache/cloudstack-kubernetes-provider:v1.2.0`.
**CCM**
- On the new cluster, the CCM deployment runs
`apache/cloudstack-kubernetes-provider:v1.2.0`. The existing clusters, built
from ISOs made with the old script, run v1.1.0.
- **A `LoadBalancer` Service.** The CCM acquired a public IP and created a
CloudStack LB rule (`<svc-uid>-tcp-8080`, 8080 -> NodePort, Active). The
Service got that IP as its `EXTERNAL-IP` within 12 s. Deleting the Service
removed the rule.
- **`providerID` stays empty, on every node, as on the clusters running
v1.1.0.**
- CKS starts the kubelets without `--cloud-provider=external`
(`KUBELET_EXTRA_ARGS=--cgroup-driver=systemd`). So no node gets the
`node.cloudprovider.kubernetes.io/uninitialized` taint, and nothing is left
behind.
- The CCM's node controller only initialises nodes that have that taint,
whatever its version. So the providerID fix in v1.2.0 doesn't come into play on
a CKS cluster as it is set up today.
- This PR doesn't change how the kubelet is started, so I don't count it
against the PR. The PR description says v1.1.0 "does not initialise nodes",
which suggests v1.2.0 would initialise them on CKS. It doesn't on its own.
- I didn't see any effect on the autoscaler. It found the cluster's VMs,
scaled up and down, and only logged `Failed to get nodegroup for <vm id>:
Unable to find node ...` warnings for the nodes, as it does without a
providerID.
**Autoscaler**
- **Enabling** autoscaling with `scaleKubernetesCluster
autoscalingenabled=true minsize=1 maxsize=3` deployed it from the ISO's
manifest:
```
registry.k8s.io/autoscaling/cluster-autoscaler:v1.32.7
["./cluster-autoscaler","--v=4","--stderrthreshold=info","--cloud-provider=cloudstack","--cloud-config=/config/cloud-config",
"--skip-nodes-with-local-storage=false","--nodes=1:3:<cluster id>"]
```
- **Log checks over the whole run** (2,628 lines by the end of the
scale-down):
- no `Failed to watch` or `Failed to list` lines, no `forbidden` errors,
and no cache-sync timeouts. The only `forbidden` matches were the
`lastScaleDownForbidden...` field in the status lines;
- one `E` line, `Failed to get node infos for groups: Unable to find node
...`. It came 45 s after a scaled-down node's VM was removed, and the
autoscaler carried on normally.
- **Scale-up:**
- I created a 3-replica Deployment requesting 1200m CPU each. One pod fit
on the single worker and two were Pending.
- The autoscaler logged `Final scale-up plan: [{<cluster id> 1->3 (max:
3)}]` and `Scale-up: setting group <cluster id> size to 3`. That started a
`KUBERNETES.CLUSTER.SCALE` job in CloudStack.
- Two workers were added. The cluster went `Running` -> `Scaling` ->
`Running` with size 3, and all 3 pods were Running on 3 workers 232 s after the
Deployment was created.
- **Scale-down:**
- I deleted the Deployment.
- About 10 minutes later the autoscaler logged `Scale-down: removing empty
node` for both new workers. The cluster went `Scaling` (3) -> `Scaling` (2) ->
`Running` with size 1, 654 s after the deletion.
- The upstream example doesn't set `--scale-down-unneeded-time` or
`--scale-down-delay-after-add`, so the defaults applied. I didn't edit the
manifest.
**Upgrade redeploys the autoscaler**
- `upgradeKubernetesCluster` from the 1.32.13 ISO to the 1.33.13 ISO took
323 s, and ended Running with both nodes on v1.33.13.
- Afterwards the autoscaler deployment had a new pod:
```
before: registry.k8s.io/autoscaling/cluster-autoscaler:v1.32.7
--nodes=1:3:<cluster id> (started 00:27:02)
after: registry.k8s.io/autoscaling/cluster-autoscaler:v1.33.6
--nodes=1:3:<cluster id> (started 00:52:25)
```
- The new pod's log had no `forbidden`, watch or list, or cache-sync errors.
- The first attempt on the control node failed with `kubectl apply` of the
network manifest:
```
... is forbidden: User "kubernetes-admin" cannot get resource
"customresourcedefinitions" ...
RBAC: [clusterrole.rbac.authorization.k8s.io "cluster-admin" not found,
...]
```
This was right after `kubeadm upgrade apply` restarted the API server,
before its RBAC cache had synced. The upgrade worker's existing retry succeeded
on the next attempt. It's not related to the autoscaler change.
**Cleanup:** the cluster and both Kubernetes versions were deleted. Neither
left a VM, network or ISO behind.
**Result:** pass. The ISO records what it holds and bundles the autoscaler
that matches its Kubernetes version and CCM v1.2.0. The autoscaler they deploy
scales a CKS cluster up and down, and an upgrade moves it to the new version's
autoscaler.
**Not covered**
- Overrides (`AUTOSCALER_VERSION`, `CCM_VERSION`), aarch64, and building
with etcd. Only default builds were run on the live cluster.
- Kubernetes 1.35 and later, where the DRA read access in the extra
ClusterRole matters most. The CKS template here was used with 1.32 and 1.33.
- Upgrading a cluster onto an ISO built by the old script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
