andrijapanicsb opened a new issue, #14552:
URL: https://github.com/apache/cloudstack/issues/14552
### problem
#### What happens, in plain words
CloudStack Kubernetes Service (CKS) never resizes a single node. It resizes
whole node groups: all control nodes together, all worker nodes together (and
all dedicated etcd nodes, if the cluster has any). Since 4.21 the scale API can
give each group its own offering (the `nodeofferings` option), so a cluster
stores its compute offering in two places:
- one cluster-wide offering (`serviceofferingid`). It dates from before
4.21, when every node used one offering. Today it is the fallback for any group
without its own offering, and it is what "scale everything"
(`serviceofferingid=X` without `nodeofferings`) sets; and
- separate offerings for control nodes and for worker nodes
(`controlofferingid`, `workerofferingid`).
When an operator resizes only the workers or only the control nodes,
CloudStack updates that group's offering but never updates the cluster-wide
one. From then on the cluster-wide offering is out of date: it still shows
whatever was set when the cluster was created or last resized as a whole, and
no longer matches anything the nodes actually run.
Every `scaleKubernetesCluster` request that does **not** name an offering is
then treated as "move all nodes to the cluster-wide offering". That includes:
- turning on autoscaling (`autoscalingenabled=true minsize=… maxsize=…`),
- adding workers by hand (`size=N`),
- **every scale-up done by the cluster-autoscaler itself**. Its CloudStack
provider adds workers by calling exactly `scaleKubernetesCluster id=… size=N`
(and removes them with `nodeids=…`).
So on a cluster where the workers or the control nodes were ever resized on
their own, CloudStack first resizes **all existing nodes, control nodes
included**, to the old offering and records it as the worker offering, and only
then adds the new worker, at the old size. The operator (or the autoscaler,
unattended) asked for one more worker and got a resize of the whole cluster. If
the old offering is smaller than what the nodes run now, the request is most
likely refused instead, so the cluster cannot be scaled out at all.
What should happen: a request without an offering changes no node's size,
and new workers use the cluster's current **worker** offering, i.e. the size
the existing workers were last scaled to. If the cluster-wide offering happens
to be up to date (for example right after a cluster-wide resize), this is
exactly what happens today.
A second, smaller problem shows up when enabling autoscaling: the setting is
not saved correctly. The cluster-autoscaler is deployed and running, but
CloudStack still records autoscaling as off, so a later
`autoscalingenabled=false` does nothing and the autoscaler has to be removed by
hand.
The CloudStack UI is not affected: once a cluster has per-group offerings,
its Scale dialog pre-fills the worker and control offering fields with their
current values and sends them as `nodeofferings`, so the out-of-date
cluster-wide offering is never used. The problem hits requests that name no
offering: API and `cmk` calls, automation, and — most importantly — the
cluster-autoscaler, which scales up with `size=N` only.
#### Steps to reproduce
1. Create a CloudManaged CKS cluster on offering **A** (1 control, 2
workers) and wait for `Running`.
2. Resize per node group, for example workers to **B** and control to **C**
(from the UI or the API):
`scaleKubernetesCluster id=<id> nodeofferings[0].node=worker
nodeofferings[0].offering=<B> nodeofferings[1].node=control
nodeofferings[1].offering=<C>`
→ `listKubernetesClusters` still shows `serviceofferingid` = **A**.
(Variant: do a cluster-wide scale to **X** first, then per-group scales;
the cluster-wide field stays **X**.)
3. Through the API (not the UI), either
a) enable autoscaling: `scaleKubernetesCluster id=<id>
autoscalingenabled=true minsize=2 maxsize=3`, or
b) add a worker: `scaleKubernetesCluster id=<id> size=3`.
4. Observe:
- existing control and worker VMs are moved to offering **A** (on a
running cluster live, one by one; a VM may be migrated for capacity), and
`workerofferingid` becomes **A**;
- in b), the new worker is deployed with **A**;
- in a), the cluster-autoscaler is deployed but `autoscalingenabled` is
not set, and `autoscalingenabled=false` afterwards leaves the autoscaler
running.
Observed for 3a) during QA of #14223; 3b) follows the same code path (traced
below). Manual scale-out with `size=N` was also tested in that QA run and
behaved correctly, but only while the cluster-wide offering was up to date.
#### Expected
- A request that names no offering (`size`, `nodeids`, autoscaling
parameters) changes no node offering and resizes no existing VM, neither
control nor worker nodes.
- Workers added by `size=N`, whether sent by an operator or by the
cluster-autoscaler, use the current `workerofferingid`.
- `autoscalingenabled`, `minsize` and `maxsize` are saved, and disabling
autoscaling removes the cluster-autoscaler.
#### Technical details
- **Where the outdated offering is injected:**
`KubernetesClusterManagerImpl#scaleKubernetesCluster` always calls
`createNodeTypeToServiceOfferingMap(cmd.getServiceOfferingNodeTypeMap(),
cmd.getServiceOfferingId(), cluster)`. When the request has no `nodeofferings`
map, that method always adds a `DEFAULT` entry: the request's
`serviceofferingid` if given, otherwise `cluster.getServiceOfferingId()`. A
request without any offering therefore reaches the scale worker as "scale
everything to the cluster-wide offering".
- **Where the resize happens:** `KubernetesClusterScaleWorker#scaleCluster`
sets `scaleClusterDefaultOffering = true` when `DEFAULT` is present and loops
over `CONTROL, ETCD, WORKER`. Every node group whose current offering differs
from `DEFAULT` gets `scaleKubernetesClusterOffering`, which resizes its VMs and
commits the offering via `updateKubernetesClusterEntryForNodeType`. CONTROL and
ETCD are handled before WORKER. For WORKER with a larger `size`, the offering
is changed first and then `scaleKubernetesClusterSize` adds VMs; in the
autoscaling branch, `autoScaled && isNodeOfferingScalingNeeded` also resizes
the workers.
- **Which offering new VMs get:**
`KubernetesClusterResourceModifierActionWorker#createKubernetesNode` uses
`getServiceOfferingForNodeTypeOnCluster(WORKER, cluster)` (worker offering,
else cluster-wide). That part is correct; the problem is that the step above
has already overwritten the worker offering.
- **How the cluster-autoscaler calls CloudStack:** `kubernetes/autoscaler`,
`cluster-autoscaler/cloudprovider/cloudstack/service/cks.go`: scale-up
`scaleKubernetesCluster {id, size}`, scale-down `scaleKubernetesCluster {id,
nodeids}`. Neither passes an offering.
- **Why the UI is not affected:**
`ui/src/views/compute/ScaleKubernetesCluster.vue` shows the cluster-wide
`serviceofferingid` field only while the cluster has no `workerofferingid` /
`controlofferingid` / `etcdofferingid`, and pre-fills every offering field with
the current value; `handleSubmit` sends them as `serviceofferingid` or
`nodeofferings[]`, so a UI request always names an offering.
- **Why the cluster-wide offering goes stale:** per-group scales update
`control_service_offering_id` / `worker_service_offering_id` only;
`service_offering_id` changes only on a cluster-wide scale. The UI "Compute
Offering" field shows this value, so it is out of date as well.
- **Likely cause of the lost autoscaling flag:** `autoscaleCluster` saves
the autoscaling fields, then `scaleKubernetesClusterOffering` →
`updateKubernetesClusterEntryForNodeType` saves the worker's cluster object,
which was loaded earlier and still holds the old autoscaling values.
- **Suggested fix:**
1. Do not add a `DEFAULT` entry when the request contains neither
`serviceofferingid` nor `nodeofferings`; `size`, `nodeids` and autoscaling-only
requests then plan no offering change, and new workers keep using
`workerofferingid`.
2. Reload the cluster entry after `autoscaleCluster` (or update only the
offering, count and capacity columns in
`updateKubernetesClusterEntryForNodeType`) so later saves cannot revert the
autoscaling fields.
3. Optional: keep `service_offering_id` consistent after per-group scales,
or stop using it as a fallback once per-group offerings exist.
#### Environment
Found during functional testing of #14223 (CKS live-resize capacity
reconciliation) on a 2 management server + 2 KVM host lab, Kubernetes 1.33.1.
The code paths above are unchanged on `main` (checked at `af9764b`); #14223
does not touch them.
### versions
MAIN (24.0.0-SNAPSHOT)
### The steps to reproduce the bug
_No response_
### What to do about it?
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
