vishesh92 opened a new issue, #107:
URL: https://github.com/apache/cloudstack-kubernetes-provider/issues/107
## Summary
On a VPC tier the controller opens a port with a Network ACL entry matched
only on protocol and
port. The entry is shared by every LoadBalancer Service on that tier, and by
anything an operator
added by hand, but cleanup deletes it whenever any one Service stops using
that port. The other
Services lose ingress on that port until their next reconcile recreates the
entry.
## Impact
- Two Services on one tier both exposing TCP/443 on their own public IPs
share one ACL entry.
Deleting either Service, or moving it off port 443, removes the entry.
Traffic to the other
Service on 443 is dropped until its next reconcile.
- An entry created by an operator for the same protocol and port is adopted
rather than
duplicated, then deleted on the same trigger. If the operator's entry had
a restricted CIDR, the
controller recreates it as `0.0.0.0/0` on the next reconcile, silently
widening exposure.
Firewall rules on isolated networks have the same ownership model but are
scoped to the
Service's own public IP, so they only affect that Service.
## Reproduction
1. VPC tier with a custom ACL list attached (not `default_allow` /
`default_deny`).
2. Create Service A, `type: LoadBalancer`, port 443.
3. Create Service B, `type: LoadBalancer`, port 443. `listNetworkACLs` shows
a single TCP/443
entry on the tier.
4. Delete Service A.
5. The TCP/443 entry is gone. Service B does not answer on 443 until its
next reconcile.
## Cause
`updateNetworkACL` lists the tier's entries and skips creation if any entry
matches the IP
protocol and port, so entries are shared and carry no marker of who created
them.
`deleteNetworkACLRule` lists the same way and deletes the first match. It is
called from:
- `EnsureLoadBalancerDeleted`, for every rule of the Service being deleted;
- the prune path in `EnsureLoadBalancer` (`pruneNetworkACLRule`), for a rule
the Service no
longer needs. Since #104 an entry the *same* Service still claims is kept
and the delete is
scoped to the rule's own network, but nothing considers other Services.
This is the design the ACL path shipped with in #69 (July 2024). The cleanup
loop it replaced in
#104 deleted the same way, so #104 narrowed the problem without fixing it.
Raised by review on
#104: <link to the review thread>.
## Proposed fix
Tag what the controller creates and delete only what it tagged. CloudStack
resource tags support
`NetworkACL`, `FirewallRule` and `PublicIpAddress`.
- On create, tag the ACL entry (and firewall rule) with the owning Service,
for example
`cloudstack-ccm/service=<service UID>`.
- Stop sharing entries: create one entry per Service even when a matching
one exists, so each
entry has exactly one owner. Verify CloudStack accepts same-protocol/port
items in one ACL list;
if it does not, fall back to one tag key per Service on the shared entry
and delete only when the
last key is removed.
- On cleanup, delete only entries tagged with this Service. Untagged entries
are left alone, which
also stops the controller from deleting or widening operator-created
entries.
- Existing untagged entries created by older controller versions need a
one-time adoption rule,
for example tag on first match during reconcile.
A cheaper mitigation without tags is to list the tier's load balancer rules
before deleting and
skip the delete if any other rule uses the same protocol and port. It closes
the cross-Service
case but not the operator-created case, and it races with concurrent
reconciles.
## Related
The same tagging covers the public IP ownership gaps left over from #105: an
IP whose rule was
deleted but whose release failed is never retried, and the pre-release check
counts only load
balancer rules so an IP carrying port forwarding or static NAT can be
released. Those can be
folded into this issue or filed separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
