kiranchavala opened a new issue, #14191:
URL: https://github.com/apache/cloudstack/issues/14191
### problem
A CKS cluster deployed on KVM hosts using CloudStack's **default guest CPU
configuration** comes up with
every node `NotReady` and no CNI. The cause is that the CNI's binaries
refuse to execute on the CPU
CloudStack exposes to the guest, but nothing in CloudStack surfaces this —
the only evidence is in the log
of an init container three levels down.
### versions
- ACS: 4.23
- Hypervisor: KVM on Oracle Linux 8 , Ubuntu 24
- Kubernetes: v1.37.0
- CNI: Calico v3.32.2 (`quay.io/calico/cni:v3.32.2`,
`quay.io/calico/node:v3.32.2`)
- Guest CPU: `guest.cpu.mode` unset in
`/etc/cloudstack/agent/agent.properties` (CloudStack default)
### The steps to reproduce the bug
1. Build a CKS binaries ISO with Calico as the CNI**
```bash
./create-kubernetes-binaries-iso.sh /var/www/html 1.37.0 1.9.1 1.37.0 \
https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/manifests/calico.yaml
\
0.45.0 setup-v1.37.0 amd64 3.7.0
```
(If building on EL8 or later, also apply the volume-label fix from #14180 /
#14182, or the nodes will not
mount the ISO at all and you will see a different failure first.)
2. Register it and create a cluster
```bash
cmk add kubernetessupportedversion name=cks-1.37.0 semanticversion=1.37.0 \
url=http://<host>/setup-v1.37.0-x86_64.iso zoneid=<zone> mincpunumber=2
minmemory=2048
cmk create kubernetescluster name=cks-test zoneid=<zone> \
kubernetesversionid=<id> serviceofferingid=<so> size=1
```
3. Confirm the guest CPU model is the default
On a KVM host, before deploying:
```bash
grep -n "^guest.cpu" /etc/cloudstack/agent/agent.properties # expect: no
output
```
Observed on the cluster nodes:
```
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
test-cks-control-1a0a9b3c3cb NotReady control-plane 5m38s v1.37.0
test-cks-node-1a0a9b420dc NotReady <none> 5m27s v1.37.0
$ kubectl get pods -A
NAMESPACE NAME READY STATUS
RESTARTS AGE
kube-system calico-kube-controllers-db57f7644-m6kln 0/1 Pending
0 6m49s
kube-system calico-node-s8xs2 0/1 Init:Error
6 6m45s
kube-system calico-node-sqxp9 0/1 Init:Error
6 6m49s
kube-system coredns-559f6c778d-v8l2m 0/1 Pending
0 6m50s
kube-system coredns-559f6c778d-wdr4l 0/1 Pending
0 6m50s
kube-system headlamp-65978d54db-psb98 0/1 Pending
0 6m48s
kube-system kube-apiserver-test-cks-control-... 1/1 Running
0 6m56s
kube-system kube-controller-manager-... 1/1 Running
0 6m56s
kube-system kube-proxy-srbqg 1/1 Running
0 6m45s
kube-system kube-scheduler-... 1/1 Running
0 6m56s
```
The control plane is healthy; only the CNI fails. Everything `Pending` is
downstream of having no pod
network.
The actual error is in the first init container of `calico-node`:
```
$ kubectl -n kube-system logs calico-node-sqxp9 -c upgrade-ipam --previous
This program can only be run on AMD64 processors with v2 microarchitecture
support.
```
That message comes from the Go runtime's startup CPU check, emitted when a
binary is compiled with
`GOAMD64=v2`. Calico v3.32's binaries are built for the **x86-64-v2**
microarchitecture level, which
requires `popcnt`, `sse3`, `ssse3`, `sse4_1`, `sse4_2`, `cmpxchg16b` and
`lahf_lm`.
CloudStack's KVM agent leaves the guest CPU model unset by default
(`guest.cpu.mode` in
`agent.properties`, read at `LibvirtComputingResource.java:1441-1453`), so
libvirt gives guests the generic
`qemu64` model, which is **x86-64-v1 only**. The binary starts, checks
CPUID, and exits 1.
Workaround
Confirm the cause by fixing the CPU model
On each KVM host:
```bash
echo "guest.cpu.mode=host-model" >> /etc/cloudstack/agent/agent.properties
systemctl restart cloudstack-agent
```
Recreate the cluster (the CPU model is applied at VM start, so existing
nodes keep the old one). The nodes
now expose the v2 feature set and Calico starts normally:
### What to do about it?
The workaround is `guest.cpu.mode=host-model` (or `host-passthrough`) on
every KVM host, but CloudStack
gives operators no way to discover that from the symptom. Suggestions,
roughly in order of value:
1. **Document the requirement for CKS.** The CKS documentation should state
that KVM hosts need a guest
CPU model exposing x86-64-v2 (i.e. `guest.cpu.mode=host-model` or
`host-passthrough`), because current
CNI images require it. This alone would have saved the whole
investigation.
2. **Surface the guest CPU model in CloudStack.** `guest.cpu.mode` being
per-host `agent.properties` with
no global/cluster setting and no UI surface is the core usability
problem: an admin cannot see or change
what CPU their guests get without SSH-ing to every host. Exposing it as a
host detail (alongside
`Host.OS.Kernel.Version` etc., which are already reported) would make
mismatched hosts visible.
3. CKS Code improvement for the api
https://cloudstack.apache.org/api/apidocs-4.23/apis/createKubernetesCluster.html
**`extraconfig` cannot be used.** `createKubernetesCluster` exposes no
`extraconfig` parameter; CKS
deploys its nodes internally via
`userVmService.createAdvancedVirtualMachine(...)`
(`KubernetesClusterResourceModifierActionWorker.java:439`) without one; and
even if it did, the KVM
agent's `addExtraConfigComponent` (`LibvirtComputingResource.java:3468`)
only *appends* raw XML to the
domain via `vm.addComp(comp)`. Since CloudStack already emits a `<cpu>`
element, a second one would be
invalid domain XML rather than an override.
**Service offering details do not reach the CPU definition either.**
`HypervisorGuruBase` routes service
offering details into the extraConfig map
(`addServiceOfferingExtraConfiguration`, line ~257), not into
the VM details map, so they hit the same additive-XML limitation.
**A per-VM detail *does* override it, but cannot be set at CKS deployment
time.** The KVM agent honours a
VM detail ahead of the host-wide setting:
```java
// LibvirtComputingResource.java:3310-3314
private CpuModeDef createCpuModeDef(VirtualMachineTO vmTO, int vcpus) {
final CpuModeDef cmd = new CpuModeDef();
String cpuMode = details.get(VmDetailConstants.GUEST_CPU_MODE) != null
? details.get(VmDetailConstants.GUEST_CPU_MODE) : guestCpuMode;
String cpuModel = details.get(VmDetailConstants.GUEST_CPU_MODEL) != null
? details.get(VmDetailConstants.GUEST_CPU_MODEL) : guestCpuModel;
```
`VmDetailConstants.GUEST_CPU_MODE` is the string `"guest.cpu.mode"`, and the
details map comes from
`vm_instance_details` (`HypervisorGuruBase.java:329`). So
`updateVirtualMachine details[0].guest.cpu.mode=host-model`
followed by a stop/start does fix an individual node — but only *after* the
VM exists, i.e. after cluster
creation has already failed. There is no way to supply it up front.
This suggests a small, concrete fix.** CKS already passes VM details through
`customParameterMap` when
creating nodes:
```java
// KubernetesClusterResourceModifierActionWorker.java:408-413
Map<String, String> customParameterMap = new HashMap<String, String>();
customParameterMap.put("rootdisksize", String.valueOf(rootDiskSize));
customParameterMap.put(VmDetailConstants.ROOT_DISK_CONTROLLER, "scsi");
```
Adding `VmDetailConstants.GUEST_CPU_MODE` to that map — either
unconditionally as a sensible default for
CKS nodes, or as an optional `createKubernetesCluster` parameter — would let
operators deploy working CKS
clusters without touching `agent.properties` on every KVM host.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
