andrijapanicsb opened a new issue, #14213: URL: https://github.com/apache/cloudstack/issues/14213
## Context [PR #13226](https://github.com/apache/cloudstack/pull/13226) removed the remaining CKS restriction that prevented changing node compute offerings while a KVM cluster was running. Its implementation calls the CloudStack VM upgrade workflow for each CKS VM, and its validation covered the KVM/libvirt domain definition. Follow-up discussion confirmed that the PR scope ends at the KVM/hypervisor layer and that Kubernetes-specific steps may still be required. The Kubernetes documentation explicitly states that, for CPU and/or memory updates, calling the kubelet allocatable-resources endpoint is not sufficient and the kubelet must be restarted so that resource `capacity` and `allocatable` are correct: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#monitoring-device-plugin-resources Related upstream Kubernetes reports: - https://github.com/kubernetes/kubernetes/issues/124650 - https://github.com/kubernetes/enhancements/issues/4609 ## Problem `KubernetesClusterScaleWorker.scaleKubernetesClusterOffering()` currently performs, for each CKS VM: ```java userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(), ...); ``` After a successful live KVM resize, CloudStack can report the new offering and libvirt can expose the additional vCPU/RAM to the VM, while Kubernetes still advertises the node's old values in: ```text .status.capacity.cpu .status.capacity.memory .status.allocatable.cpu .status.allocatable.memory ``` That leaves three potentially different views of the same node: 1. CloudStack service offering / VM metadata; 2. resources visible in the guest OS; 3. resources advertised by the Kubernetes Node object and used by the scheduler. The current workflow neither verifies the guest OS nor refreshes and verifies the Kubernetes view. A successful CKS scale job can therefore report success even though the Kubernetes scheduler cannot use the newly added resources. ## Proposed behavior For a running Cloud-managed CKS cluster, when a node offering change increases the CPU count and/or RAM, reconcile each Kubernetes node sequentially: 1. Read and record the node's current schedulability, Kubernetes capacity, and allocatable values. 2. If it was schedulable, run `kubectl cordon <node>`. 3. Live-resize the VM through the existing CloudStack VM upgrade workflow. 4. Verify inside the guest OS that the expected online CPU count and memory are visible. 5. Restart `kubelet` on that node. 6. Wait for `systemctl is-active kubelet`, Kubernetes Node `Ready=True`, and updated `capacity` / `allocatable` values. 7. Restore schedulability only if CKS cordoned the node; preserve a node that was already cordoned by the operator. 8. Continue with the next node only after the current node is healthy and verified. This must be a rolling operation. Do not resize all nodes first and restart all kubelets afterwards. ## Cordon versus drain A kubelet service restart does not power off the VM and does not itself require workload eviction. For this live-resize path, `cordon` is the default safety mechanism: it blocks new scheduling while the node is being resized and while Kubernetes still has stale capacity, but leaves existing Pods running. `drain` should not be added unconditionally because it evicts workloads, is subject to PodDisruptionBudgets, and materially changes the impact of a live operation. Drain remains appropriate for a workflow that will stop/reboot the VM or for a future explicit operator policy. The Kubernetes node documentation describes `cordon` / unschedulable as a preparatory maintenance step that prevents new Pods without affecting existing Pods: https://kubernetes.io/docs/concepts/architecture/nodes/#manual-node-administration The drain documentation describes drain as the operation used before bringing down/deleting a machine: https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/ ## When no extra kubelet restart is needed - A `Created` cluster has no running node to reconcile. - For a `Stopped` cluster, the normal VM start also starts kubelet, which discovers the new resources; verify after cluster start rather than adding a second restart. - An offering change that modifies only CPU cap/speed or another non-capacity attribute, while CPU count and RAM remain unchanged, does not need a kubelet capacity refresh. - Dedicated external etcd VMs are not Kubernetes Node objects and do not run kubelet in the CKS image. Verify guest resources, but do not run `kubectl cordon` or restart kubelet for those VMs. ## Verification requirements The scale job must not succeed based only on the CloudStack VM record or libvirt XML. For worker and control-plane Kubernetes nodes: - guest online CPU count equals the target offering's CPU count; - guest total memory increased and is consistent with the target offering, allowing normal kernel/virtualization overhead; - kubelet is active after restart; - the Node is `Ready=True`; - `.status.capacity.cpu` equals the online guest CPU count; - `.status.capacity.memory` is consistent with guest total memory; - `.status.allocatable.cpu` and/or `.status.allocatable.memory` increased when the corresponding capacity increased; - the node returns to its original schedulability state. Use Kubernetes resource-quantity parsing or structured JSON/JSONPath output. Do not compare formatted `kubectl describe` text. ## Failure behavior - If cordon fails, do not resize that node. - If the VM resize succeeds but guest verification fails, fail the CKS scale operation with the node and observed/expected values. - If kubelet restart or Kubernetes verification fails, leave the node cordoned when that is safer, report the exact recovery action, and do not proceed to the next node. - Always best-effort restore a node that CKS itself cordoned when verification proves it is healthy. - Never uncordon a node that was already unschedulable before this operation. - A retry must recognize a VM that already has the target offering and continue the missing guest/Kubernetes reconciliation rather than treating it as fully complete. ## Suggested implementation Add a focused orchestration helper used by `KubernetesClusterScaleWorker`, for example `KubernetesClusterNodeCapacityReconciler`, responsible for: - reading Node status through the control-plane SSH/kubectl path; - resolving the target node's SSH endpoint for isolated, VPC, and direct-access networks; - cordon/preserve-schedulability handling; - guest CPU/RAM verification; - kubelet restart; - Ready/capacity/allocatable verification; - bounded retries and actionable diagnostics. The scale worker should compare the old and target offering before changing each VM and invoke this reconciler only for a running Kubernetes node whose CPU count or RAM changes. ## Acceptance criteria 1. A live CPU increase on a running KVM CKS worker finishes only after guest CPU and Node `capacity`/`allocatable` reflect the increase. 2. A live RAM increase finishes only after guest memory and Node `capacity`/`allocatable` reflect the increase. 3. Kubelet is restarted exactly once per affected Kubernetes node during a successful attempt. 4. Nodes are processed sequentially and are cordoned during the inconsistent window. 5. A previously schedulable node is uncordoned after successful verification. 6. A previously cordoned node remains cordoned. 7. Existing Pods are not drained/evicted merely to restart kubelet. 8. Control-plane nodes are reconciled sequentially; external etcd-only VMs are not treated as Kubernetes Nodes. 9. Stopped/Created clusters do not receive an unnecessary kubelet restart. 10. Cap-only offering changes do not receive an unnecessary kubelet restart. 11. Failures show which of CloudStack, guest OS, kubelet, Node readiness, capacity, or allocatable verification failed. 12. Unit and integration tests verify the complete CloudStack → guest → Kubernetes state transition, not only libvirt XML. ## Suggested labels - `component:cks` - `component:kubernetes` - `type:bug` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
