andrijapanicsb opened a new issue, #14704:
URL: https://github.com/apache/cloudstack/issues/14704
#### In short
#13226 added live offering changes for running CKS clusters on KVM. On a
default install that feature never works: CKS nodes use the SystemVM template,
which is registered as not dynamically scalable, so every node is created as
not scalable and the scale fails. Proposal: register KVM SystemVM templates as
dynamically scalable by default. We tested the side effects and found nothing
that should stop this.
#### How to reproduce
On a default KVM zone with `enable.dynamic.scale.vm=true`, create a CKS
cluster without choosing a node template, then change its service offering
while it is running. The scale fails with `Unable to scale VM instance ... as
it does not have tools to support dynamic scaling`.
#### Why it matters
The CKS documentation says *"A running or stopped Kubernetes cluster can be
scaled ... to a desired cluster size and service offering"*. On a default
install, only stopped clusters can. To use the live feature, an operator has to
find out on their own that the SystemVM template's flag must be turned on. They
also have to turn it on for each existing node while the cluster is stopped,
because a VM's flag can only be enabled when its template allows it: `dynamic
scaling cannot be enabled for instances the template does not have scaling
enabled`.
#### Cause
- `vm_template.dynamically_scalable` defaults to 0, and
`SystemVmTemplateRegistration` never sets it, so the KVM SystemVM template is
always `false`. The column dates from 2013, when it meant "the template has
XenServer/VMware tools". KVM needs no guest tools for hot-plug.
- A user VM, including a CKS node, is scalable only when its template, its
offering and `enable.dynamic.scale.vm` all allow it. So the template flag
blocks every CKS node.
#### What we tested (standard x86_64 KVM hosts)
| Situation | Result |
|---|---|
| Default install: live offering change of a running CKS cluster | Fails:
"does not have tools to support dynamic scaling" |
| Template flag on, node flags on, same change | Works: all nodes resized
live, no reboot, guests see the added CPUs |
| Template flag on, global setting on: newly created router, SSVM, console
proxy | They become dynamically scalable too, and lose about 67 MB of usable
memory each (a 64 MB kernel I/O buffer for the hot-plug memory range): 457 →
390 MiB on 512 MiB routers and SSVM, 960 → 894 MiB on the 1 GiB console proxy |
| Template flag on, global setting off | No change for system VMs or CKS
nodes; nothing becomes scalable (checked in the code) |
| Existing VMs after the template flag is changed | Unchanged; each VM keeps
the flag it was created with, so nothing breaks |
| VM whose vCPU maximum exceeds the machine type limit | Refused by libvirt
on every host; CloudStack tries the other hosts and returns an error, no crash |
The memory cost applies only where an operator has turned on
`enable.dynamic.scale.vm`, which is already required to use live scaling.
Anyone who has made the SystemVM template scalable by hand, to get live CKS
scaling, already has it today.
<details>
<summary>vCPU maximum limits seen on the hosts we checked</summary>
A dynamically scalable VM gets the host's CPU count as its vCPU maximum.
These are the machine type limits libvirt reports:
| Host | QEMU / libvirt | `pc` (i440fx) | Q35 |
|---|---|---|---|
| AlmaLinux 9.1 | 7.0.0 / 8.5.0 | 240 | 710 |
| Ubuntu 24.04.4 | 8.2.2 / 10.0.0 | 255 | 1024 |
| Ubuntu 26.04.1 | 10.2.1 / 12.0.0 | 255 | 4096 |
| Debian 13 | 11.0.2 / 11.3.0 | 255 | 4096 |
Only hosts with more threads than these limits would be affected, and
`kvm.cpu.dynamic.scaling.capacity` caps the maximum per cluster. On Q35, more
than 255 vCPUs also needs an IOMMU in extended interrupt mode (`<ioapic
driver='qemu'/>` plus `<iommu model='intel'><driver intremap='on'
eim='on'/></iommu>`).
</details>
#### Proposal
Make KVM SystemVM templates dynamically scalable by default, from the next
release on:
1. In `SystemVmTemplateRegistration`, set the flag when a KVM SystemVM
template is registered, for fresh installs and new template versions.
2. In the upgrade SQL, set it on all existing KVM SystemVM templates, since
installs upgraded across several releases keep several of them:
`UPDATE cloud.vm_template SET dynamically_scalable = 1 WHERE type =
'SYSTEM' AND hypervisor_type = 'KVM' AND removed IS NULL;`
The SQL runs before the new template row is created, so both 1 and 2 are
needed.
3. Mention in the release notes and the CKS docs that existing CKS nodes
keep their old flag until they are updated or recreated.
#### Side note: system VMs ignore their offering's dynamic-scaling flag
Routers, SSVM and console proxy copy the template's flag and never look at
their service offering's dynamic-scaling setting (`NetworkHelperImpl`,
`SecondaryStorageManagerImpl`, `ConsoleProxyManagerImpl`). We set it to off on
those system offerings, recreated the VMs, and they still came up scalable.
User VMs check template, offering and the global setting together since #4643;
system VMs were not updated then. @harikrishna-patnala, as the author of #4643
(and of the original system VM scaling change, CLOUDSTACK-3042), could you
confirm whether leaving system VMs out was an omission, or deliberate for a
reason we're missing? If it's an omission, it's a separate small bug, not part
of this proposal.
#### Suggested testing for the fix
- Fresh install: the KVM SystemVM template is registered with
`dynamically_scalable = 1`.
- Upgrade from 4.20 / 4.22 / 4.23: every existing KVM SystemVM template row
is set to 1, and the new one too.
- A new CKS cluster on the default template scales live, with
`enable.dynamic.scale.vm` on.
- Recreated routers, SSVM and console proxy start normally, with the global
setting on and off.
- Existing VMs and CKS nodes are unchanged after the upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]