bhouse-nexthop opened a new issue, #14630:
URL: https://github.com/apache/cloudstack/issues/14630
### problem
On main, a CKS upgrade, scale-up or start fails at once if a node VM still
has an ISO attached, for example the Kubernetes binaries ISO left by an
operation that didn't finish:
```
ERROR KubernetesClusterUpgradeWorker Failed to attach binaries ISO for VM :
<control node> in the Kubernetes cluster name: <cluster>
com.cloud.exception.InvalidParameterValueException: The specified ISO is
already attached to this Instance.
at
com.cloud.template.TemplateManagerImpl.enforceCdromAttachLimits(TemplateManagerImpl.java)
at
com.cloud.template.TemplateManagerImpl.attachIso(TemplateManagerImpl.java)
at
com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterActionWorker.attachIsoKubernetesVMs(KubernetesClusterActionWorker.java)
```
The operation fails before it changes anything, and the cluster goes to
Alert. It keeps failing until the ISO is detached from each node by hand.
Cause:
- `KubernetesClusterActionWorker.attachIsoKubernetesVMs` calls
`templateService.attachIso(isoId, vmId, true)` on every node, and relies on the
attach replacing whatever ISO the node has. That is how `attachIso` behaves on
4.22.
- #13101 (multiple CD-ROMs per VM) added
`TemplateManagerImpl.enforceCdromAttachLimits`. It refuses an ISO that is
already attached, and, with the default of one CD-ROM, any second ISO
("Instance has reached the maximum of 1 attached CD-ROM(s)").
A node can be left holding an ISO when the management server stops during a
CKS operation, or when an operator attached one by hand.
### versions
main (since #13101). 4.22 is not affected, as its `attachIso` replaces the
attached ISO.
### The steps to reproduce the bug
1. Create a CKS cluster.
2. Attach the binaries ISO of the cluster's Kubernetes version to one of its
node VMs (`attachIso`), or stop the management server while an upgrade of the
cluster is running, after the ISO is attached.
3. Upgrade or scale up the cluster (after any recovery of the cluster state).
4. The job fails with "The specified ISO is already attached to this
Instance." With a different ISO attached, it fails with "Instance has reached
the maximum of 1 attached CD-ROM(s)".
### What to do about it?
In `attachIsoKubernetesVMs`, leave the binaries ISO in place on a node that
has it already, and detach any other ISO from the node before attaching, as the
attach used to replace it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
