andrijapanicsb opened a new issue, #14704:
URL: https://github.com/apache/cloudstack/issues/14704

   #### In short
   
   #13226 added live offering changes for running CKS clusters on KVM. On a 
default install that feature never works: CKS nodes use the SystemVM template, 
which is registered as not dynamically scalable, so every node is created as 
not scalable and the scale fails. Proposal: register KVM SystemVM templates as 
dynamically scalable by default. We tested the side effects and found nothing 
that should stop this.
   
   #### How to reproduce
   
   On a default KVM zone with `enable.dynamic.scale.vm=true`, create a CKS 
cluster without choosing a node template, then change its service offering 
while it is running. The scale fails with `Unable to scale VM instance ... as 
it does not have tools to support dynamic scaling`.
   
   #### Why it matters
   
   The CKS documentation says *"A running or stopped Kubernetes cluster can be 
scaled ... to a desired cluster size and service offering"*. On a default 
install, only stopped clusters can. To use the live feature, an operator has to 
find out on their own that the SystemVM template's flag must be turned on. They 
also have to turn it on for each existing node while the cluster is stopped, 
because a VM's flag can only be enabled when its template allows it: `dynamic 
scaling cannot be enabled for instances the template does not have scaling 
enabled`.
   
   #### Cause
   
   - `vm_template.dynamically_scalable` defaults to 0, and 
`SystemVmTemplateRegistration` never sets it, so the KVM SystemVM template is 
always `false`. The column dates from 2013, when it meant "the template has 
XenServer/VMware tools". KVM needs no guest tools for hot-plug.
   - A user VM, including a CKS node, is scalable only when its template, its 
offering and `enable.dynamic.scale.vm` all allow it. So the template flag 
blocks every CKS node.
   
   #### What we tested (standard x86_64 KVM hosts)
   
   | Situation | Result |
   |---|---|
   | Default install: live offering change of a running CKS cluster | Fails: 
"does not have tools to support dynamic scaling" |
   | Template flag on, node flags on, same change | Works: all nodes resized 
live, no reboot, guests see the added CPUs |
   | Template flag on, global setting on: newly created router, SSVM, console 
proxy | They become dynamically scalable too, and lose about 67 MB of usable 
memory each (a 64 MB kernel I/O buffer for the hot-plug memory range): 457 → 
390 MiB on 512 MiB routers and SSVM, 960 → 894 MiB on the 1 GiB console proxy |
   | Template flag on, global setting off | No change for system VMs or CKS 
nodes; nothing becomes scalable (checked in the code) |
   | Existing VMs after the template flag is changed | Unchanged; each VM keeps 
the flag it was created with, so nothing breaks |
   | VM whose vCPU maximum exceeds the machine type limit | Refused by libvirt 
on every host; CloudStack tries the other hosts and returns an error, no crash |
   
   The memory cost applies only where an operator has turned on 
`enable.dynamic.scale.vm`, which is already required to use live scaling. 
Anyone who has made the SystemVM template scalable by hand, to get live CKS 
scaling, already has it today.
   
   <details>
   <summary>vCPU maximum limits seen on the hosts we checked</summary>
   
   A dynamically scalable VM gets the host's CPU count as its vCPU maximum. 
These are the machine type limits libvirt reports:
   
   | Host | QEMU / libvirt | `pc` (i440fx) | Q35 |
   |---|---|---|---|
   | AlmaLinux 9.1 | 7.0.0 / 8.5.0 | 240 | 710 |
   | Ubuntu 24.04.4 | 8.2.2 / 10.0.0 | 255 | 1024 |
   | Ubuntu 26.04.1 | 10.2.1 / 12.0.0 | 255 | 4096 |
   | Debian 13 | 11.0.2 / 11.3.0 | 255 | 4096 |
   
   Only hosts with more threads than these limits would be affected, and 
`kvm.cpu.dynamic.scaling.capacity` caps the maximum per cluster. On Q35, more 
than 255 vCPUs also needs an IOMMU in extended interrupt mode (`<ioapic 
driver='qemu'/>` plus `<iommu model='intel'><driver intremap='on' 
eim='on'/></iommu>`).
   
   </details>
   
   #### Proposal
   
   Make KVM SystemVM templates dynamically scalable by default, from the next 
release on:
   1. In `SystemVmTemplateRegistration`, set the flag when a KVM SystemVM 
template is registered, for fresh installs and new template versions.
   2. In the upgrade SQL, set it on all existing KVM SystemVM templates, since 
installs upgraded across several releases keep several of them:
      `UPDATE cloud.vm_template SET dynamically_scalable = 1 WHERE type = 
'SYSTEM' AND hypervisor_type = 'KVM' AND removed IS NULL;`
      The SQL runs before the new template row is created, so both 1 and 2 are 
needed.
   3. Mention in the release notes and the CKS docs that existing CKS nodes 
keep their old flag until they are updated or recreated.
   
   #### Side note: system VMs ignore their offering's dynamic-scaling flag
   
   Routers, SSVM and console proxy copy the template's flag and never look at 
their service offering's dynamic-scaling setting (`NetworkHelperImpl`, 
`SecondaryStorageManagerImpl`, `ConsoleProxyManagerImpl`). We set it to off on 
those system offerings, recreated the VMs, and they still came up scalable. 
User VMs check template, offering and the global setting together since #4643; 
system VMs were not updated then. @harikrishna-patnala, as the author of #4643 
(and of the original system VM scaling change, CLOUDSTACK-3042), could you 
confirm whether leaving system VMs out was an omission, or deliberate for a 
reason we're missing? If it's an omission, it's a separate small bug, not part 
of this proposal.
   
   #### Suggested testing for the fix
   
   - Fresh install: the KVM SystemVM template is registered with 
`dynamically_scalable = 1`.
   - Upgrade from 4.20 / 4.22 / 4.23: every existing KVM SystemVM template row 
is set to 1, and the new one too.
   - A new CKS cluster on the default template scales live, with 
`enable.dynamic.scale.vm` on.
   - Recreated routers, SSVM and console proxy start normally, with the global 
setting on and off.
   - Existing VMs and CKS nodes are unchanged after the upgrade.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [I] CKS: running clusters can... andrijapanicsb (via GitHub) via commits

Reply via email to