adestis-mf opened a new pull request, #14670:
URL: https://github.com/apache/cloudstack/pull/14670
### Description
This PR fixes CKS clusters on VPC tiers whose VPC uses the Firewall service
for public IPs
(added in #12706). In such a VPC the VPC virtual router drops ingress
traffic to a public IP
unless a firewall rule allows it. CKS only created network ACL items, port
forwarding and the
API load balancer rule on VPC tiers. The management server could not reach
the nodes (SSH 2222+,
API 6443), the control node could not reach its own API through the public IP
(`controlPlaneEndpoint`), and cluster creation failed after the start
timeout.
When the VPC virtual router provides the Firewall service of the cluster's
VPC, CKS now creates
the same ingress firewall rules as on isolated networks:
- create: SSH range and 6443 on the cluster public IP
(`setupKubernetesClusterVpcTierRules`),
and the etcd SSH ports (`setupKubernetesEtcdNetworkRules`);
- scale: the SSH firewall rule is replaced to match the new node count
(`scaleKubernetesClusterVpcTierRules`), as for isolated networks;
- add/remove node: the per-node SSH firewall rule is created and revoked
(`KubernetesClusterAddWorker`, `KubernetesClusterRemoveWorker`).
In a VPC the firewall rules are created **after** the load balancing and
port forwarding
rules. Applying firewall rules on a VPC public IP does not associate the IP
with the VR
(`IpAddressManagerImpl.applyRules` skips it when the rule has no network).
If the firewall rules
come first, the later LB/PF rules do not associate it either and the IP
never reaches the VR
(#14669). On isolated networks the order stays as it was.
Firewall rules on VPC public IPs are stored with `vpc_id` and without
`network_id`, so the
per-node lookup now goes by public IP and ports for VPC tiers
(`findIngressFirewallRuleForPorts`); isolated networks keep using
`findByNetworkIdAndPorts`.
On cluster deletion the rules are revoked together with the release of the
cluster public IP
(`cleanupIpResources` → `revokeFirewallRulesForIp`), so the destroy worker
is unchanged.
The check (`isVpcFirewallEnabled`) is the one that sets
`vpc_firewall_enabled` on the VR:
`VpcServiceMapDao.canProviderSupportServiceInVpc(vpcId, Firewall,
VPCVirtualRouter)`. VPCs
without the Firewall service, isolated networks and the NSX/Netris paths
behave as before.
`FirewallRulesDao` is now injected once in `KubernetesClusterActionWorker`
instead of separately
in three subclasses, so that the shared lookup can use it.
This targets `main`: the Firewall service for VPC public IPs is not in
`4.22`, and there is no
`4.23` branch. The commit also applies cleanly on top of the 4.23.0.0 tag.
Fixes: #14668
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Feature/Enhancement Scale
- [ ] Major
- [ ] Minor
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [x] Major
- [ ] Minor
- [ ] Trivial
### Screenshots (if appropriate):
### How Has This Been Tested?
Unit tests (new): `KubernetesClusterActionWorkerTest` covers
`isVpcFirewallEnabled`,
`isPublicIpFirewallRequired` and `findIngressFirewallRuleForPorts` for
isolated networks and VPC
tiers with and without the Firewall service;
`KubernetesClusterResourceModifierActionWorkerTest`
checks that `setupKubernetesClusterVpcTierRules` creates the firewall rules
only when the VPC
firewall is enabled, and only after the LB and SSH port forwarding rules.
```
mvn -pl plugins/integrations/kubernetes-service -am test
-Dtest='**/kubernetes/**/*Test' \
-Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false
```
Result on JDK 11 / Maven 3.10: BUILD SUCCESS, 141 tests in the
kubernetes-service module,
0 failures, 0 errors; checkstyle 0 violations.
Live, on 4.23.0.0 (KVM) without the patch, adding by hand the firewall rules
the patch creates
(same VPC with the Firewall service, same tier, v1.35.4, 1 control + 1
worker):
| Run | Firewall rules | Result |
|---|---|---|
| 1 | none, then added after 30 min | `Error` after 3600 s: first `kubeadm
init` had failed, control node never recovers |
| 2 | within 7 s of the IP, before LB/PF | `Error`: IP never configured on
the VR (#14669) |
| 3 | about 10 s after CKS created LB and PF | `Running` in under 3 min;
scale-up to 2 workers `Running` |
Deleting each cluster released the public IP and left no firewall rule
behind. The patched
build itself was not deployed to a live management server.
#### How did you try to break this feature and the system with this change?
- Isolated networks: the same calls in the same order as before; the lookup
still uses
`findByNetworkIdAndPorts`.
- VPC without the Firewall service: no firewall call is made, only ACL, PF
and LB as before.
- Order in a VPC: firewall rules before LB/PF break the IP association (run
2 above), so the
patch creates them afterwards; a unit test pins the order.
- A missing firewall rule during node revert/removal is now skipped with a
warning instead of a
`NullPointerException`.
- Out of scope: the source CIDR of the rules stays `0.0.0.0/0` as on
isolated networks
(#13970, #14212); `removeSshFirewallRule` still matches by port, as
before. On scale the VPC
path goes through the same `removeSshFirewallRule`, so the fix for
clusters with separate etcd
nodes in #14165 (#14159) applies to VPC tiers as well once it is merged
forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
