bhouse-nexthop opened a new issue, #14610:
URL: https://github.com/apache/cloudstack/issues/14610
### problem
Both scale-down and upgrade of a CloudStack-managed Kubernetes cluster drain
a node before they remove or upgrade it, and both handle a drain that fails, or
doesn't finish quickly, badly.
**Scale-down** (`KubernetesClusterScaleWorker.removeKubernetesClusterNode`):
- When `kubectl drain` exits non-zero, for example because a
PodDisruptionBudget doesn't allow an eviction yet, it gives up at once with no
retry.
- When it does retry, after an exception, each attempt is counted twice.
- The node is left cordoned.
- The cluster goes to **Alert**, although no node was removed and the
cluster still works.
**Upgrade** (`KubernetesClusterUpgradeWorker`):
- A failed drain is retried at once, with no wait for the eviction to become
possible, and on the last failure the node is left cordoned.
- Only the first control node is waited on to be Ready. The next node can be
drained while the previous one is still coming back.
- The script for node *i* is run over SSH on port `sshPort + i`. The
port-forwarding rules aren't created in the order of the upgrade's VM list:
manually upgraded nodes are left out of that list, and nodes added later get
the next free port. So the script can run on the wrong node.
**Both:** `kubectl drain` runs without `--timeout`, so a fixed 60s SSH
timeout cuts it off rather than a deadline of its own.
**After a failed upgrade** the cluster is left in Alert.
`upgradeKubernetesCluster` only accepts a Running cluster, so the upgrade can't
be retried or finished, although the nodes already upgraded are known
(`kubernetesnodeversion` in the cluster's VM list).
### versions
4.22 (the code is the same on main).
### The steps to reproduce the bug
1. Create a CKS cluster with 2 or more worker nodes.
2. Deploy a workload with a PodDisruptionBudget that allows no disruption
(for example `minAvailable` equal to the replica count) on the worker that a
scale-down would remove.
3. Scale the cluster down by one worker.
4. The drain fails at once, the cluster goes to Alert, and the node stays
cordoned. No node is removed.
For the upgrade: let the drain or the upgrade script fail on any node after
the first one. The cluster goes to Alert with some nodes at the new version,
and `upgradeKubernetesCluster` refuses to run again because the cluster isn't
Running.
### What to do about it?
- Use one drain helper for both paths:
- each attempt counts once, and a non-zero exit or an exception is retried
after a wait;
- `kubectl drain --timeout` comes from a setting, and the SSH wait is
derived from it;
- on the last failure the node is uncordoned and the drain output is
reported.
- On scale-down, cordon and drain every node to remove before removing any.
If one can't be drained, uncordon them all and leave the cluster as it was
rather than in Alert.
- On upgrade, wait for every node to be Ready, and reach each node through
its own SSH port-forwarding rule.
- Let `upgradeKubernetesCluster` be retried on a cluster in Alert after an
upgrade to the same version failed part way, skipping the nodes already
upgraded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
