adestis-mf opened a new issue, #14673:
URL: https://github.com/apache/cloudstack/issues/14673
### problem
When a CKS cluster has nodes added with `addNodesToKubernetesCluster`, a
later scale-up that
increases the number of managed nodes fails while rebuilding the SSH
firewall rules, because the
new rule range overlaps the existing rule of an imported node. By the time
it fails, the old SSH
rule and the SSH port forwarding rules of the managed nodes are already
removed.
From the code:
1. `addFirewallRulesForNodes` opens one range for the managed nodes, 2222 to
2222+M-1 (M =
managed non-etcd nodes), and one single-port rule per imported node after
that range
([KubernetesClusterActionWorker.java#L1116-L1146](https://github.com/apache/cloudstack/blob/a7be64af7ad8fdefe1948efa175e87d4852d3453/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterActionWorker.java#L1116-L1146)).
`KubernetesClusterAddWorker` creates the same single-port rule when a
node is imported
([KubernetesClusterAddWorker.java#L229](https://github.com/apache/cloudstack/blob/a7be64af7ad8fdefe1948efa175e87d4852d3453/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterAddWorker.java#L229)).
All rules use the source CIDR `0.0.0.0/0`.
2. On scale-up, `scaleKubernetesClusterIsolatedNetworkRules` revokes a
single rule with
`removeSshFirewallRule` (the one starting at 2222), removes the port
forwarding rules of the
old range, and calls `setupKubernetesClusterIsolatedNetworkRules`, which
calls
`addFirewallRulesForNodes` again
([KubernetesClusterScaleWorker.java#L128-L149](https://github.com/apache/cloudstack/blob/a7be64af7ad8fdefe1948efa175e87d4852d3453/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java#L128-L149)).
The single-port rules of the imported nodes are not revoked.
3. With one more managed node the new range ends on the port of the first
imported node.
`FirewallManagerImpl.detectRulesConflict` rejects two firewall rules with
the same protocol,
the same source CIDR and overlapping ports, so `addFirewallRulesForNodes`
fails with "Failed to
provision firewall rules for SSH access", and the scale-up fails with
"unable to update network
rules"
([KubernetesClusterScaleWorker.java#L518-L525](https://github.com/apache/cloudstack/blob/a7be64af7ad8fdefe1948efa175e87d4852d3453/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java#L518-L525)).
The new VMs are cleaned up and the cluster goes back to Running
(`Scaling` + `OperationFailed`
→ `Running`), but the revoked SSH rule and the removed port forwarding
rules of the managed
nodes are not restored. The cluster looks healthy while the management
server can no longer
reach the managed nodes over SSH, which later operations such as an
upgrade need.
Example: 1 control node and 1 worker (rule 2222-2223), one imported node
(rule 2224), scale to 2
workers: the new range 2222-2224 conflicts with the rule for 2224.
This affects isolated networks today. With apache/cloudstack#14670, which
adds the same firewall
rules on VPC tiers whose VPC has the Firewall service, it affects those VPC
tiers as well. VPC
tiers without the Firewall service have no firewall rules and are not
affected by this part.
### versions
- Apache CloudStack `main` (a7be64af7a), 4.23.0.0 and the `4.22` branch: the
code paths above
are the same in all three
- Isolated network with the virtual router; CKS cluster with at least one
node added via
`addNodesToKubernetesCluster`
Found by reading the code during the review of #14670, not reproduced on a
live system yet.
### The steps to reproduce the bug
1. Create a CKS cluster on an isolated network with 1 control node and 1
worker. CKS creates the
firewall rule TCP 2222-2223 on the source NAT IP.
2. Add a prepared VM with `addNodesToKubernetesCluster`. CKS creates a
firewall rule and a port
forwarding rule for TCP 2224.
3. Scale the cluster to 2 workers with `scaleKubernetesCluster`.
4. Expected: the cluster is Running with 2 managed workers plus the imported
node. From the code,
the scale-up fails instead and the cluster is back in Running with 1
worker, but the firewall
rule for 2222-2223 and the port forwarding rules 2222-2223 are gone.
### What to do about it?
When the SSH rules are rebuilt for a new node count, revoke the single-port
rules of the imported
nodes together with the range rule before calling
`addFirewallRulesForNodes`, which recreates them
after the new range (and move their port forwarding rules accordingly). A
regression test that
imports a node and then scales up would cover it.
`removeSshFirewallRule` is also being changed in #14165 (scale-down with
separate etcd nodes), so
a fix should be coordinated with that PR. Related: #14670 (VPC tiers with
the Firewall service).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
