bhouse-nexthop opened a new pull request, #14666: URL: https://github.com/apache/cloudstack/pull/14666
### Description This PR adds a check that `kubernetes_cluster_vm_map` still matches the cluster it describes. Detection only: the map, the VMs and the cluster are left as they are. Fixes #14665 #### What it checks Every 5 minutes the Kubernetes cluster status scanner starts a check on its own thread, so the scanner and its lock are not held up by SSH. The check runs under its own global lock, so only one management server checks at a time. For each CloudManaged cluster in **Running** or **Alert** state it compares three views: 1. the VM map; 2. the cluster's VMs in CloudStack: the owner's VMs (account or project) whose host name follows the names CKS gives node VMs (`<prefix>-control-<hex>`, `<prefix>-node-<hex>`, `<etcd prefix>-<n>-<hex>`, including the 40-character prefix used before #10778), with their state; 3. the Kubernetes nodes, from `kubectl get nodes -o name` on the control node, over SSH in the same way the other operations do. It reports: - mapped VMs that are destroyed, expunged, in Error state or missing; - VMs named as nodes of the cluster, not removed, destroyed or in Error, that are not in the map of this or any other cluster (VMs created less than 15 minutes ago are left out, as an operation may not have mapped them yet); - Kubernetes nodes that match no VM of the cluster (a node registered under its VM's FQDN matches); - mapped running VMs that are not Kubernetes nodes, after a 15 minute grace period from the VM's creation or last state change, so a node that is still joining is not flagged. etcd nodes are left out. If kubectl can't be run (no SSH address, SSH or kubectl failing), only the map is compared with the VMs. What was found on the nodes before is kept, neither cleared nor reported again. Clusters in any other state (Starting, Scaling, Upgrading, Stopping, RemovingNodes, ...) and ExternalManaged clusters are skipped. The cluster's state and map are read again before reporting, and nothing is reported if either changed during the check, for example because an operation ran meanwhile. #### Alerts and dedupe Each mismatch raises a WARN event (`KUBERNETES.CLUSTER.VM.MAP.CHECK`, on the cluster, not displayed to end users) and an alert naming the cluster and the VM or node. The alert type is `ALERT_TYPE_USERVM`, an existing type, so 4.22 gets no new alert type. Each alert is a separate alert. The keys of the mismatches reported are kept in two cluster details (`vmMapCheckReportedVmDrift`, `vmMapCheckReportedNodeDrift`, not displayed), so the dedupe also holds across restarts and management servers: - a mismatch is reported once, and again only if it went away and came back; - a mapped VM keeps one key as it goes from destroyed to expunged to gone; - when the set of mismatches changes, only the new ones are reported; - once a cluster matches again, the detail is removed and an INFO event says so. The details are written before the alerts are sent. More than 5 new mismatches in one cluster at once give 5 alerts and one summing up the rest, and each mismatch still gets its event. #### Setting `cloud.kubernetes.cluster.vm.map.check.enabled` (global, dynamic, default `true`) turns the check off. No schema change. #### Notes The API and UI already refuse to destroy a VM of a CKS cluster. The path through which a failed destroy on scale-down left the map behind is fixed in #14611. What is left is drift from manual node rebuilds and database edits, which this reports. ### Types of changes - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] New feature (non-breaking change which adds functionality) - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] Enhancement (improves an existing feature and functionality) - [ ] Cleanup (Code refactoring and cleanup, that may add test cases) - [ ] Build/CI - [ ] Test (unit or integration test code) ### Feature/Enhancement Scale or Bug Severity #### Feature/Enhancement Scale - [ ] Major - [x] Minor ### How Has This Been Tested? Unit tests, `KubernetesClusterVmMapCheckerTest` (32) and `KubernetesClusterVmMapCheckWorkerTest` (8), cover: - each mismatch type: mapped VM missing, destroyed, expunged or in Error; unmapped VM, with the cases that must not be reported; node without a VM; running VM that is not a node; - the grace periods, FQDN node names, etcd and stopped VMs, and a cluster with no nodes; - the dedupe: reported once, only new mismatches on change, one key through the destroyed/expunged/gone lifecycle, cleared and reported again, the alert cap, duplicate map rows, keys stored as a hash when there are very many; - skipping clusters in other states, ExternalManaged clusters, and clusters whose state or map changes during the check; - SSH/kubectl failure: only the VM comparison runs and earlier node findings are kept; - the interval and the setting; - the host name pattern, including names that must not match, and parsing of the kubectl output. All `Kubernetes*Test` tests in the plugin pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
