prashanthr2 opened a new issue, #13942:
URL: https://github.com/apache/cloudstack/issues/13942
### problem
On a VPC with public IPs from more than one range/VLAN, the virtual router
installs an unscoped source NAT rule on every public interface using the single
VPC source NAT address. Because the rule has no source match, it also rewrites
traffic the router itself originates, so the VR cannot emit packets with the
correct source address from any public interface other than the source-NAT one.
Generated in systemvm/debian/opt/cloud/bin/cs/CsAddress.py:
```
if self.get_type() == "public" and self.config.is_vpc() and method == "add"
and not self.config.is_routed():
if self.address["source_nat"]:
vpccidr = cmdline.get_vpccidr()
self.fw.append(["filter", 3, "-A FORWARD -s %s ! -d %s -j ACCEPT" %
(vpccidr, vpccidr)])
self.fw.append(["nat", "", "-A POSTROUTING -j SNAT -o %s --to-source
%s" % (self.dev, self.address['public_ip'])])
elif cmdline.get_source_nat_ip() and not self.is_private_gateway():
self.fw.append(["nat", "", "-A POSTROUTING -j SNAT -o %s --to-source
%s" % (self.dev, cmdline.get_source_nat_ip())])
```
The if branch pairs its SNAT with a FORWARD -s <vpccidr> rule, showing the
intent is forwarded guest traffic. The elif branch has no equivalent scoping
and no -s match
**VPC router r-21-VM — two public ranges on separate VLANs:**
```
eth1 10.1.30.5/24 public range 1 (source NAT IP), gateway 10.1.30.1
eth2 10.1.31.2/24 public range 2, gateway 10.1.31.1
eth3 10.0.0.1/24 guest tier
eth4 10.1.1.1/24 guest tier
```
Resulting NAT rules:
```
-A POSTROUTING -o eth1 -j SNAT --to-source 10.1.30.5
-A POSTROUTING -o eth2 -j SNAT --to-source 10.1.30.5 ← eth2 is
10.1.31.2; wrong subnet
-A POSTROUTING -s 10.1.1.0/24 -d 10.1.1.52/32 -o eth4 -p tcp -m tcp --dport
80 -j SNAT --to-source 10.1.1.1
```
/etc/cloudstack/ips.json is correct.
```
root@r-21-VM:~# cat /etc/cloudstack/ips.json | grep -iE
'eth|public_ip|source_nat|network|gateway'
"eth0": [
"device": "eth0",
"gateway": "",
"network": "169.254.0.0/16",
"public_ip": "169.254.82.31",
"source_nat": false
"eth1": [
"device": "eth1",
"gateway": "10.1.30.1",
"is_private_gateway": false,
"network": "10.1.30.0/24",
"public_ip": "10.1.30.5",
"source_nat": true,
"eth2": [
"device": "eth2",
"gateway": "10.1.31.1",
"is_private_gateway": false,
"network": "10.1.31.0/24",
"public_ip": "10.1.31.2",
"source_nat": false,
"eth3": [
"device": "eth3",
"gateway": "10.0.0.1",
"network": "10.0.0.0/24",
"public_ip": "10.0.0.1",
"source_nat": false
"eth4": [
"device": "eth4",
"gateway": "10.1.1.1",
"network": "10.1.1.0/24",
"public_ip": "10.1.1.1",
"source_nat": false
```
The eth2 entry has source_nat: false and public_ip: 10.1.31.2 so this is not
bad input from the management server; the elif deliberately substitutes the
VPC-wide source NAT IP.
**Packet capture, VR pinging its own eth2 gateway (10.1.31.1):**
```
root@r-21-VM:~# tcpdump -i any -nnnvvv 'host 10.1.31.1 and icmp'
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot
length 262144 bytes
10:01:07.711338 eth2 Out IP (tos 0x0, ttl 64, id 10156, offset 0, flags
[DF], proto ICMP (1), length 84)
10.1.30.5 > 10.1.31.1: ICMP echo request, id 6524, seq 0, length 64
10:01:08.712408 eth2 Out IP (tos 0x0, ttl 64, id 10259, offset 0, flags
[DF], proto ICMP (1), length 84)
10.1.30.5 > 10.1.31.1: ICMP echo request, id 6524, seq 1, length 64
10:01:09.713031 eth2 Out IP (tos 0x0, ttl 64, id 10338, offset 0, flags
[DF], proto ICMP (1), length 84)
10.1.30.5 > 10.1.31.1: ICMP echo request, id 6524, seq 2, length 64
```
Requests and replies traverse different interfaces. On a permissive network
the ping still succeeds via that asymmetry; on any gateway enforcing
reverse-path forwarding the packet is dropped at ingress and no reply is
generated.
**User-visible impact**: /root/health_checks/gateways_check.py pings every
NIC's gateway, so 10.1.31.1 is reported unreachable despite being perfectly
reachable, producing repeated Health checks failed alerts. If
router.health.checks.failures.to.recreate.vr includes gateways_check.py,
CloudStack will also repeatedly recreate a healthy VR.
More generally, any VR-originated traffic leaving a non-source-NAT public
interface carries an address that does not belong on that segment.
**Not affected**: guest egress. A VPC has a single source NAT IP by design
and guest traffic follows the default route out the source-NAT interface, so it
is unchanged.
### versions
CloudStack : 4.22.1
Hypervisor: KVM
Gateway: pfSense (reverse-path filtering enabled)
CsAddress.py is byte-identical in 4.20.1.0, 4.20.3.0, 4.22.1.0 and main
long-standing behaviour, not a regression.
### The steps to reproduce the bug
1. Create a VPC with the default VPC offering and at least one tier.
2. Acquire a public IP from a second public range on a different VLAN/subnet
and associate it with the VPC (e.g. add a port-forwarding rule), so the VR
gains a second public NIC.
3. On the VR, confirm the interface and its SNAT rule:
```
ip addr show eth2
iptables -t nat -S POSTROUTING | grep eth2
```
The --to-source value is the VPC source NAT IP (10.1.30.5), not the eth2
address (10.1.31.2).
4. Ping the second range's gateway from the VR while capturing both public
interfaces:
```
tcpdump -lnni eth1 -s 96 'icmp and host 10.1.31.1' > /tmp/eth1.txt 2>&1 &
tcpdump -lnni eth2 -s 96 'icmp and host 10.1.31.1' > /tmp/eth2.txt 2>&1 &
sleep 1; ping -c 3 10.1.31.1; sleep 2; kill %1 %2
cat /tmp/eth1.txt /tmp/eth2.txt
```
4. Run the health check directly:
`python3 /root/health_checks/gateways_check.py basic`
**Expected**: requests leave eth2 sourced from 10.1.31.2, replies return on
eth2, and the check reports all gateways reachable.
**Actual**: requests leave eth2 sourced from 10.1.30.5; replies arrive on
eth1, or are dropped entirely by a reverse-path-enforcing gateway, and the
check reports 10.1.31.1 unreachable.
### What to do about it?
Scope the elif rule so it does not match locally-generated packets:
```
elif cmdline.get_source_nat_ip() and not self.is_private_gateway():
self.fw.append(["nat", "", "-A POSTROUTING -m addrtype ! --src-type
LOCAL -j SNAT -o %s --to-source %s" % (self.dev, cmdline.get_source_nat_ip())])
```
! --src-type LOCAL excludes packets whose source is an address on the
router, while forwarded guest traffic still matches and is SNATed as before.
Verified by replacing the rule manually on the VR:
```
iptables -t nat -D POSTROUTING -o eth2 -j SNAT --to-source 10.1.30.5
iptables -t nat -A POSTROUTING -o eth2 -m addrtype ! --src-type LOCAL -j
SNAT --to-source 10.1.30.5
```
| Behaviour | Before | After |
| --- | --- | ---|
| VR → 10.1.31.1| 10.1.30.5 > 10.1.31.1, reply on eth1 / dropped|10.1.31.2 >
10.1.31.1, reply on eth2 |
| Guest → internet | SNAT to 10.1.30.5 via eth1 | unchanged |
A CIDR-based scope (-s <vpccidr>) was considered and rejected: it fails when
the public ranges fall inside the VPC CIDR.
Workaround for affected deployments — a manual iptables edit is reverted by
CloudStack on any config refresh, so suppress the check instead:
```
cmk update configuration name=router.health.checks.to.exclude
value=gateways_check.py zoneid=<zone-uuid>
cmk list configurations name=router.health.checks.failures.to.recreate.vr
# ensure gateways_check.py is NOT listed
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
