ewerton-silva00 opened a new issue, #13544:
URL: https://github.com/apache/cloudstack/issues/13544
### problem
### CloudStack version
4.22.1.0
### Environment
- Hypervisor: KVM
- CKS Kubernetes version: v1.36.1
- CSI driver: csi.cloudstack.apache.org
- Cluster type: CloudManaged (1 control node, size 4)
### Summary
Deleting a CKS Kubernetes cluster can leave it stuck in `Destroying` state
indefinitely (observed 25h+, reproduced twice) because
`KubernetesClusterDestroyWorker` performs PV cleanup via a **blocking SSH
exec with no timeout**, and a second, separate bug silently ignores a
state-machine error on retry, causing repeated attempts to hang the exact
same way instead of failing fast.
### Root cause #1 — no timeout on SSH exec used for PV cleanup
`KubernetesClusterDestroyWorker.destroy()` → `destroyClusterVMs()` calls
`KubernetesClusterResourceModifierActionWorker.deletePVsWithReclaimPolicyDelete()`,
which SSHes into the cluster's control-plane node and runs `kubectl` to
delete PVs with `ReclaimPolicy=Delete` **before** any cluster VM is
stopped/destroyed.
This uses `com.cloud.utils.ssh.SshHelper.sshExecute()`, which has **no
execution timeout** on this code path. If the remote `kubectl` command
blocks — e.g. because a PVC is stuck in `Terminating` (its
`kubernetes.io/pvc-protection` finalizer is waiting on a Pod that is still
using it and was never deleted) — the SSH channel read blocks forever and
so does the whole async destroy job.
Thread dump confirming the hang (management server JVM, thread alive for
91851s / ~25.5h):
"API-Job-Executor-16" ... in Object.wait()
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:338)
at com.trilead.ssh2.channel.FifoBuffer.read(FifoBuffer.java:212)
- locked <...> (a com.trilead.ssh2.channel.Channel)
at com.trilead.ssh2.channel.Channel$Output.read(Channel.java:127)
at
com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:935)
at
com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58)
at
com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:70)
at com.cloud.utils.ssh.SshHelper.sshExecute(SshHelper.java:292)
at
com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterResourceModifierActionWorker.deletePVsWithReclaimPolicyDelete(KubernetesClusterResourceModifierActionWorker.java:966)
at
com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterDestroyWorker.destroyClusterVMs(KubernetesClusterDestroyWorker.java:110)
at
com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterDestroyWorker.destroy(KubernetesClusterDestroyWorker.java:311)
at
com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.destroyKubernetesCluster(KubernetesClusterManagerImpl.java:2439)
TCP-level connectivity to the control node was fine throughout (SSH port
and the k8s API port both accepted new connections instantly) — this was
never a network/node-availability issue, purely a stuck remote command
with an unbounded client-side wait.
Symptom visible to the operator: the cluster stays `Destroying` forever in
`listKubernetesClusters`, and the UI eventually shows a generic **"Error
encountered while fetching async job result"** after giving up polling —
which is misleading, since the job is not erroring, it simply never
completes.
### Root cause #2 — state-transition failure is logged and ignored, not
surfaced
Every retry of `deleteKubernetesCluster` against a cluster already in
`Destroying` hits:
WARN [c.c.k.c.a.KubernetesClusterDestroyWorker] Failed to transition
state of the Kubernetes cluster : <name> in state Destroying on event
DestroyRequested
com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new
state from Destroying via DestroyRequested
at
com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterActionWorker.stateTransitTo(KubernetesClusterActionWorker.java:672)
at
com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterDestroyWorker.destroy(KubernetesClusterDestroyWorker.java:310)
This exception is only logged as a `WARN` — execution continues into a
**brand new** attempt at PV cleanup / VM teardown, instead of failing the
API call immediately or resuming/checking the existing in-flight job. In
practice this means: once a cluster gets wedged by root cause #1, every
subsequent delete click by the operator just reproduces the identical
hang, with no useful error surfaced and no indication that a previous job
is already stuck.
### versions
Apache CloudStack 4.22.1.0
### The steps to reproduce the bug
### Steps to reproduce
1. Create a CKS cluster with at least one PVC provisioned via the
CloudStack CSI driver, bound to a running Pod.
2. Delete that PVC directly (`kubectl delete pvc <name>`) while the Pod is
still running and using it. It will sit in `Terminating` forever
(expected k8s behavior — `pvc-protection` finalizer waiting on the
Pod).
3. Call `deleteKubernetesCluster` on the cluster.
4. Observe: the async job never completes. `jstack` the management server
and confirm the `API-Job-Executor-N` thread for that job is blocked in
`SshHelper.sshExecute` → `deletePVsWithReclaimPolicyDelete`.
5. Retry `deleteKubernetesCluster` — observe the same
`NoTransitionException` WARN followed by the exact same hang.
### Expected behavior
- SSH-based remote command execution for PV cleanup should have a
configurable timeout, after which the cleanup step fails cleanly (and,
ideally, falls back to destroying the VMs anyway rather than blocking
the whole cluster teardown on stuck in-cluster storage state).
- A `NoTransitionException` on a destroy retry should cause the API call
to fail fast with a clear message (e.g. "a destroy operation is already
in progress for this cluster, job <jobid>"), not silently proceed to
re-run the entire destroy workflow.
### Workaround used
No workaround inside CloudStack itself was available short of a code fix.
Had to: SSH directly into the control-plane node (using the cluster's own
node keypair) to find and manually resolve the stuck PVC/Pod, restart
`cloudstack-management` to kill the wedged thread, then retry the delete.
This is not something a typical operator/admin without JVM-level access
(thread dumps, log correlation across `apilog.log` /
`management-server.log`) could reasonably self-diagnose.
### What to do about it?
**Fix #1 — bound the SSH exec used for PV cleanup**
`KubernetesClusterResourceModifierActionWorker.deletePVsWithReclaimPolicyDelete()`
should call `SshHelper.sshExecute(...)` with an explicit timeout instead of
the unbounded read used today. Concretely:
- Add a timeout parameter (config key, e.g.
`cloud.kubernetes.cluster.pv.cleanup.timeout`, default something like
120s) and pass it through to the SSH exec call.
- On timeout, log a clear WARN/ERROR ("PV cleanup on node X timed out
after Ns, proceeding with VM teardown regardless") and **continue** into
`destroyClusterVMs()`'s VM stop/expunge steps rather than blocking the
whole job. Losing best-effort PV cleanup on a wedged node is strictly
better than an unkillable job — the VMs (and their disks) are about to
be destroyed anyway, so a CSI-level PV delete that didn't get to run is
not a correctness problem for the *cluster* teardown, only a possible
orphaned-volume cleanup task afterward (worth its own follow-up: a
"list volumes with no owning VM/cluster" admin command would help here).
- `trilead.ssh2` supports read timeouts on the channel/session directly
(`Connection.connect(..., timeout)` for connect, and
`InputStream.read` respects the underlying socket's `SO_TIMEOUT` if the
session is opened with one) — this should be a small, contained change
in `SshHelper`, not a rearchitecture.
**Fix #2 — fail fast instead of silently retrying a wedged destroy**
In `KubernetesClusterActionWorker.stateTransitTo()` / the call site in
`KubernetesClusterDestroyWorker.destroy()`, a `NoTransitionException`
should not be caught-and-continue. Concretely:
- If the cluster is already in `Destroying` (or any non-terminal state
that isn't a valid source for `DestroyRequested`), the API command
should throw `InvalidParameterValueException` (or similar) back to the
caller immediately, ideally including the existing job's UUID if one is
still tracked, e.g.: *"Cluster <name> already has a destroy operation
in progress (job <jobid>), started at <timestamp>. Wait for it to
complete or check its status directly."*
- This alone would have turned this incident from "silent 25h hang,
repeated on every retry" into "immediate, actionable error on the very
first retry."
**Suggested priority:** Fix #2 is small/low-risk and should be
straightforward to backport across supported branches. Fix #1 is the
actual root cause of the indefinite hang and is more valuable long-term,
but touches shared SSH plumbing (`SshHelper`) used elsewhere in the
codebase, so it deserves a bit more test coverage/review.
Happy to submit a PR for either if that's useful — flagging here first
since I don't have full context on branch/version support policy for this
project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
