bhouse-nexthop commented on PR #14350:
URL: https://github.com/apache/cloudstack/pull/14350#issuecomment-6727080107
### Test report: multi-CIDR VPC ACL rules on a running router (iptables,
IPv6 nft, routed nft)
**Environment**
- A CloudStack 24.0 (main) build with this PR applied, alongside some of my
other open PRs.
- Why the result carries over to 4.22:
- Main still ships system VM template 4.22.0, and the parts of
`configure.py` this PR touches are the same on 4.22 and main.
- The VPC offering has no Firewall service, as on 4.22.
- Three KVM hosts on Ubuntu 24.04, nested virtualisation; Ceph RBD primary
storage.
- System VM template 4.22.0 (Debian 12, iptables 1.8.9 (nf_tables), nft
1.0.6). The routers were created on this build and run its `configure.py`.
- A dual-stack VPC:
- tier T1 holds the server V1 (Ubuntu 24.04), with the test list;
- tier T2 holds the client V2.
- A routed-mode VPC with one tier and a small VM.
- An external host on the public network (IPv4 and IPv6).
- For the "without the PR" runs, I copied apache/main's `configure.py` onto
the router, re-applied the list, and then restored the PR's script with
`restartVPC livepatch=true`.
**1. The table from the PR description**
The list:
- rule 1: tcp 22 from `1.2.3.4/32,<T2>`;
- rule 2: tcp 8080 from `3.4.5.6/32`;
- rule 3: all, deny.
| | Router chain | ssh (tcp 22) from V2 |
|---|---|---|
| with the PR | `-s 1.2.3.4/32 --dport 22 ACCEPT`, `-s 172.16.2.0/24 --dport
22 ACCEPT`, `-s 3.4.5.6/32 --dport 8080 ACCEPT`, `DROP`, `DROP` | connects |
| stock | `-s 172.16.2.0/24 ...22 ACCEPT`, `-s 3.4.5.6/32 ...8080 ACCEPT`,
`DROP`, **`-s 1.2.3.4/32 ...22 ACCEPT`**, `DROP` | connects (its CIDR is the
last one, which stock keeps in place) |
So that the traffic shows the bug, I put the client's CIDR first: rule 1 =
`<T2>,1.2.3.4/32`.
| | Router chain | ssh from V2 |
|---|---|---|
| with the PR | `-s 172.16.2.0/24 ...22 ACCEPT`, `-s 1.2.3.4/32 ...22
ACCEPT`, `-s 3.4.5.6/32 ...8080 ACCEPT`, `DROP`, `DROP` | connects |
| stock | `-s 1.2.3.4/32 ...22 ACCEPT`, `-s 3.4.5.6/32 ...8080 ACCEPT`,
`DROP`, **`-s 172.16.2.0/24 ...22 ACCEPT`**, `DROP` | **times out** |
| after `restartVPC livepatch=true` | back in list order, as in the first
row | connects |
So a chain that the old script had already scrambled is repaired once the
new script re-applies the list.
**2. Deny ahead of allow**
- rule 1: deny tcp 8080 from `<T2>,192.168.0.0/16`;
- rule 2: allow all.
The router shows:
```
-A ACL_INBOUND_eth2 -s 172.16.2.0/24 -p tcp -m tcp --dport 8080 -j DROP
-A ACL_INBOUND_eth2 -s 192.168.0.0/16 -p tcp -m tcp --dport 8080 -j DROP
-A ACL_INBOUND_eth2 -j ACCEPT
```
From V2, `curl V1:8080` fails (dropped), and tcp 22 still connects.
**3. IPv6 (dual-stack tier)**
I added `tcp 22 from <external /64>,2001:db8:1::/64`, with the reachable
prefix listed first. The rule renders as:
```
chain eth2_ingress_policy {
ip6 saddr { 2001:db8:1::/64, fd00:30::/64 } tcp dport 22 accept
counter packets 0 bytes 0 drop
}
```
SSH (tcp 22) from the external host over IPv6 connects, and another port is
dropped. The router's `cloud.log` shows the `nft add rule ... { fd00:30::/64,
2001:db8:1::/64 } ...` line and no failed `nft add rule`.
**4. Routed IPv4 (nft)**
The rule `tcp 22 from 192.0.2.0/24,<external host>/32` renders in `ip
ip4_acl` as:
```
ip saddr { 172.30.0.30, 192.0.2.0/24 } tcp dport 22 accept
```
- From the external host, tcp 22 reaches the VM, which answers with a reset
(its ssh daemon was not listening yet), and tcp 23 is dropped.
- From a host not in the rule, tcp 22 is dropped.
- With the stock script the rule doesn't load. `cloud.log` shows:
```
ERROR Command 'nft add rule ip ip4_acl eth2_ingress_policy ip saddr
192.0.2.0/24,172.30.0.30/32 tcp dport 22 accept' returned non-zero exit status
1.
```
The chain is left with only the drop, and the external host's tcp 22 times
out.
**5. A rule with 300 CIDRs**
I created a rule for tcp 443 from 300 `/32` addresses in 198.18.0.0/15.
- All 300 lines are in list order, after the earlier rules and ahead of the
final `DROP`: `iptables -S | grep -c 198.18.` is 300.
- The first apply took about 1.3 s in the router's `cloud.log`, and a
re-apply about 5.9 s. This is in line with the PR's 1.0 s and 6.6 s.
- There was no `Message too long`.
**6. Re-apply is stable**
After `restartVPC cleanup=false`, the chain was identical (`diff` was empty)
for the three-rule list and for the 300-CIDR list.
**Result:** pass.
- Each CIDR of a rule becomes its own rule, in list order, ahead of the DROP.
- The traffic results match the rules.
- The nft sets load for IPv6 and for routed IPv4.
- The 300-CIDR rule applies, re-apply is stable, and a chain scrambled by
the old script is repaired.
**Not covered**
- Egress multi-CIDR rules (only ingress was tested).
- Redundant VPC routers.
- Templates older than 4.22.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
