bhouse-nexthop commented on PR #14634:
URL: https://github.com/apache/cloudstack/pull/14634#issuecomment-6752444666
### Test report: scaling up and upgrading a cluster created before 4.21,
without the DB workaround
**Summary:** a CKS cluster created on 4.20 and upgraded to 4.22 and then
24.0, with its per-node-type template columns still NULL, now scales up.
There's no NPE, and the new worker is deployed from the default CKS template,
joins and is Ready. Launch permission for that template is granted to the owner
for the scale and revoked at its end. Scaling back down, and a Kubernetes
upgrade of the same cluster, also work. The template columns stay NULL
throughout.
**Environment**
- A CloudStack 24.0 (main) build with this PR, alongside some of my other
open PRs. Among them are #14635 and #14637, which touch other parts of the same
plugin. The management server and all three agents were upgraded in place to
this build.
- The PR targets 4.22; it wasn't run on a 4.22 build. The code it changes is
the same on both branches: 4.22's `KubernetesClusterActionWorker.init()` reads
the node templates from the per-node-type columns only, and its
`isDefaultTemplateUsed()` only compares those columns with `template_id`.
- Three KVM hosts, Ceph RBD primary storage, one management server.
- Cluster B was created on Apache CloudStack 4.20.3.1, then upgraded to
4.22.1.1 and to 24.0. It has 1 control node and 1 worker, Kubernetes v1.33.1,
and an isolated network.
- Before the run, from `kubernetes_cluster`:
```
template_id control_node_template_id worker_node_template_id
etcd_node_template_id
3 NULL NULL NULL
```
- Template 3 is the 4.20.2 system VM template that was current when the
cluster was created. Both existing nodes run from it.
- The default CKS template on this build is `systemvm-kvm-4.22.0-x86_64`
(id 207).
- The DB workaround from the issue (setting the columns to `template_id`)
was **not** applied. The other two pre-4.21 clusters in the lab have the same
NULL columns.
**Results**
| Step | Result |
|---|---|
| `scaleKubernetesCluster size=2` (1 -> 2 workers) | pass, 171 s. No NPE;
the new worker is deployed from template 207, joins, and is Ready |
| Launch permission during the scale-up | pass. A row for template 207 and
the owner appears 1 s after the call and is removed when the scale ends |
| `scaleKubernetesCluster size=1` (back to 1 worker) | pass, 20 s. The new
worker is removed |
| `upgradeKubernetesCluster` v1.33.1 -> v1.33.13 on the same cluster | pass,
334 s. Both nodes upgraded and Ready |
| Template columns after all of the above | still NULL, unchanged |
**Scale-up**
- Cluster state: `Running` -> `Scaling` (3 s after the call) -> `Running`
with size 2 (171 s).
- The management server log for the call has 0 `NullPointerException` and 0
`Unexpected exception` lines. Without this PR, this call failed after 3 s with
`Cannot invoke "VirtualMachineTemplate.getUniqueName()" because "template" is
null` (#14628).
- The new worker's template, compared with the existing nodes:
```
node vm_template_id template
kubernetes_node_version
control-1 3 SystemVM Template (KVM) v1.33.1\n
worker-1 3 SystemVM Template (KVM) v1.33.1\n
worker-2 207 systemvm-kvm-4.22.0-x86_64 1.33.1 <-
new
```
The two old nodes' version values were written by an earlier upgrade,
before #14635.
- Log, trimmed:
```
20:27:42,625 INFO KubernetesClusterScaleWorker Provisioned node VM: ...
worker-2
20:27:50,497 INFO KubernetesClusterScaleWorker Attached binaries ISO for
VM: ... worker-2
20:29:56,121 INFO KubernetesClusterUtil Kubernetes cluster
cluster B has 3 ready nodes now
20:29:57,705 INFO KubernetesClusterScaleWorker Detached Kubernetes
binaries from VM: ... worker-2
20:29:57,705 INFO KubernetesClusterScaleWorker Revoking launch
permission for systemVM Template
```
- `kubectl get nodes` afterwards:
```
control-1 Ready v1.33.1
worker-1 Ready v1.33.1
worker-2 Ready v1.33.1 <- new
```
**Launch permission**
- I sampled `launch_permission` every second during the scale-up. Only
changes are shown:
```
20:27:08 [1:t3:a2 2:t3:a2 3:t3:a2] scale requested at
20:27:08
20:27:09 [1:t3:a2 2:t3:a2 3:t3:a2 21:t207:a2] granted: template 207,
the owner's account
20:29:58 [1:t3:a2 2:t3:a2 3:t3:a2] revoked, right after
the "Revoking" log line
```
- The three rows for template 3 were there before the run and are left as
they were. The revoke only removes rows for the default template.
- Before this PR, `isDefaultTemplateUsed()` was false for this cluster, so
nothing would have been granted or revoked.
**Scale-down and upgrade**
- `size=1` removed worker-2 in 20 s, with no NPE. Cluster B was back to 1
control node and 1 worker, both Ready, and still Running.
- The upgrade to v1.33.13 (`upgradeKubernetesCluster`; covered in more
detail in the #14637 and #14635 reports) ran `init()` on the same cluster. It
upgraded both nodes and ended in Running, with both nodes on v1.33.13. The
template columns were still NULL afterwards.
**Result:** pass. A pre-4.21 cluster with NULL node-type template columns
scales up, down and upgrades without the workaround. New nodes come from the
default CKS template, and the owner holds launch permission for it only while
the scale runs.
**Not covered**
- **A cluster owned by a regular user.** All three pre-4.21 clusters in the
lab are owned by the root admin account, which can deploy from any template. So
the run shows that the permission is granted and revoked at the right times,
but not that a regular user's scale-up depends on it. A cluster created now
can't stand in for this case, since it has its node-type templates set, so I
didn't try to fake one in the DB.
- A pre-4.21 cluster with etcd nodes: these clusters had none before 4.21.
- After the scale-up, the cluster mixes nodes from template 3 and template
207. That is the expected effect of deploying new nodes from the default
template; the new node joined without problems. Whether operators want the old
nodes moved to the new template is outside this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
