Tn9hia opened a new issue, #14654: URL: https://github.com/apache/cloudstack/issues/14654
### problem Since kubeadm runs a `CreateJob` health check during `upgrade apply`/`upgrade plan`, it creates a Job `upgrade-health-check-*` in kube-system using the pause image of the TARGET version (`registry.k8s.io/pause:3.10.2` for 1.36.5). CKS upgrades node 0 (control) first. The upgrade script imports ISO images only on the node it runs on, so at this point the workers still only have `pause:3.10.1`. The control node has also just been drained (cordoned), so the Job pod is scheduled on a worker, tries to pull from registry.k8s.io, fails (air-gapped), the Job hits its deadline, and `kubeadm upgrade apply` fails. The fallback retry only ignores `CoreDNSUnsupportedPlugins`, so it fails again. The upgrade fails after all retries. [kubeadm-plan.log](https://github.com/user-attachments/files/34610271/kubeadm-plan.log) ### versions ### COMPONENT NAME Kubernetes Service (CKS) – KubernetesClusterUpgradeWorker / upgrade-kubernetes.sh ### CLOUDSTACK VERSION 4.23 ### ENVIRONMENT - Upgrade: v1.35.9 -> v1.36.5 (offline ISO, no internet on nodes) - containerd 1.7.12, 1 control + 2 workers, Cilium CNI ### OS - Ubuntu 24.04 ### The steps to reproduce the bug ### STEPS TO REPRODUCE 1. Deploy an offline CKS cluster on a version whose pause tag differs from the target (1.35.x -> pause 3.10.1) 2. Register the 1.36.5 ISO (contains pause 3.10.2) and upgrade the cluster 3. `kubectl -n kube-system get events | grep upgrade-health-check` ### EXPECTED RESULTS Upgrade succeeds using only images from the ISO. ### ACTUAL RESULTS Failed to pull image "registry.k8s.io/pause:3.10.2": ... lookup registry.k8s.io ... i/o timeout Job was active longer than specified deadline (control node left cordoned, still v1.35.9; worker has only pause:3.10.1) `kubeadm upgrade plan v1.36.5` reproduces it (hangs on "Job ... is not yet complete"). ### LOGS **1. Health-check Job events (one full cycle, the same pattern repeated for 8 Jobs between 56m and 46m)** ``` $ kubectl -n kube-system get events --sort-by=.lastTimestamp | grep -iE 'upgrade-health|pause|pull' 56m Normal SuccessfulCreate job/upgrade-health-check-1794277494890 Created pod: upgrade-health-check-1794277494890-vbh5b 56m Normal Scheduled pod/upgrade-health-check-1794277494890-vbh5b Successfully assigned kube-system/upgrade-health-check-1794277494890-vbh5b to k8s-upgrade-node-1a1c33de33f 56m Warning Failed pod/upgrade-health-check-1794277494890-vbh5b Failed to pull image "registry.k8s.io/pause:3.10.2": failed to pull and unpack image "registry.k8s.io/pause:3.10.2": failed to resolve reference "registry.k8s.io/pause:3.10.2": failed to do request: Head "https://registry.k8s.io/v2/pause/manifests/3.10.2": dial tcp: lookup registry.k8s.io on 127.0.0.53:53: read udp 127.0.0.1:45086->127.0.0.53:53: i/o timeout 55m Warning Failed pod/upgrade-health-check-1794277494890-vbh5b Error: ErrImagePull 55m Normal BackOff pod/upgrade-health-check-1794277494890-vbh5b Back-off pulling image "registry.k8s.io/pause:3.10.2" 55m Warning Failed pod/upgrade-health-check-1794277494890-vbh5b Error: ImagePullBackOff 55m Normal SuccessfulDelete job/upgrade-health-check-1794277494890 Deleted pod: upgrade-health-check-1794277494890-vbh5b 55m Warning DeadlineExceeded job/upgrade-health-check-1794277494890 Job was active longer than specified deadline ``` Jobs created: 1794277494890, 1794277555081, 1794277651281, 1794277711462, 1794277807947, 1794277868124, 1794277969440, 1794278029604. All were scheduled on the worker `k8s-upgrade-node-1a1c33de33f` and all failed the same way. 8 Jobs = 4 script runs (upgrade retries) x 2 `kubeadm upgrade apply` attempts per run. **2. Reproduce with `kubeadm upgrade plan` (control node)** ``` $ kubeadm upgrade plan v1.36.5 --v=5 [upgrade] Running cluster health checks I1110 03:21:49.429624 health.go:182] Creating a Job "upgrade-health-check-1794280909429" in the namespace "kube-system" I1110 03:21:49.449846 health.go:210] Job "upgrade-health-check-1794280909429" in the namespace "kube-system" is not yet complete, retrying ... (repeats every 1s until deadline) ``` **3. Image state: control has the image, worker does not** ``` # control node (images imported from 1.36.5 ISO) $ kubeadm config images list --kubernetes-version v1.36.5 | grep pause registry.k8s.io/pause:3.10.2 $ crictl images | grep pause registry.k8s.io/pause 3.10.1 cd073f4c5f6a8 320kB registry.k8s.io/pause 3.10.2 4a83b15d3ecfe 321kB # worker node (not yet upgraded) $ crictl images | grep pause registry.k8s.io/pause 3.10.1 cd073f4c5f6a8 320kB ``` **4. Cluster state after failure** ``` $ kubectl get node NAME STATUS ROLES AGE VERSION k8s-upgrade-control-1a1c33cc713 Ready,SchedulingDisabled control-plane 57m v1.35.9 k8s-upgrade-node-1a1c33d569a Ready <none> 56m v1.35.9 k8s-upgrade-node-1a1c33de33f Ready <none> 56m v1.35.9 ``` kubeadm binary on control is already v1.36.5, but static pods and kubelet are still v1.35.9 (the script exits at `kubeadm upgrade apply`). Environment: containerd 1.7.12, `imageRepository: registry.k8s.io`, `sandbox_image = "registry.k8s.io/pause:3.10.2"` (control). ### WORKAROUND Export pause:3.10.2 from the control node, import it on all workers, then retry the upgrade. ### What to do about it? ### SUGGESTED FIX 1. Pre-stage phase: after `attachIsoKubernetesVMs`, import ISO images on ALL cluster VMs before upgrading node 0 (e.g. split image import out of upgrade-kubernetes.sh into a separate script/flag run on every node first). 2. Log `result.second()` (script stdout/stderr) on failure in `upgradeKubernetesClusterNodes()`. It is currently discarded, which makes the failure invisible in management-server.log. 3. Minor: `upgrade-kubernetes.sh` checks `$# -lt 4` but its usage text and the Java caller use 5 args; `PAUSE_IMAGE` selection via `ctr images ls | grep pause | sort | tail -1` is fragile. Use `kubeadm config images list | grep pause` instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
