skorchir opened a new issue, #14184:
URL: https://github.com/apache/cloudstack/issues/14184
### problem
On a VPC virtual router, the per-VM static-NAT rule
```
-A POSTROUTING -o <public dev> -s <vm>/32 -j SNAT --to-source <public ip>
```
and the site-to-site VPN NAT exemption
```
-A POSTROUTING -o <public dev> -m mark --mark 0x525 -j ACCEPT
```
are both inserted at the front of the `nat` `POSTROUTING` chain, and
whichever is created later ends up first. When static NAT is enabled on a VM
**after** a site-to-site VPN connection already exists, the SNAT rule precedes
the exemption and every packet that VM sends into the tunnel carries its
**public** address instead of its private one. The remote side sees traffic
from the public IP, its rules for the VPC's guest CIDR do not match, and that
VM cannot reach the remote network while VMs without static NAT can. Nothing in
the API or UI indicates a problem: the connection stays `Connected`.
Where the other end of the tunnel is does not matter (another VPC, another
zone, another cloud, a customer gateway); the translation happens on the local
VPC router before encryption.
**Observed:**
- Ping and TCP/22 from the static-NAT VM (A) to a VM behind the peer (B)
fail. B's kernel receives no echo requests (`/proc/net/snmp` `InEchos` does not
move).
- Temporarily allowing all ingress on B's tier makes the traffic arrive, and
a listener on B reports the connection from A's static-NAT **public** address,
not from the VPC's guest CIDR.
- Traffic initiated by B toward A is not affected (connection tracking keeps
the reply un-translated), so the failure is one-directional and only for the VM
with static NAT.
- The order of operations decides the outcome. Measured on the same setup:
- static NAT enabled on A **before** the VPN gateway, customer gateway and
connection are created: traffic crosses with private addresses;
- static NAT disabled and re-enabled **after** the connection exists: ICMP
and TCP/22 from A both fail;
- the connection deleted and recreated with static NAT left in place:
traffic crosses again.
- Replacing the static NAT with a port-forwarding rule also avoids it, since
port forwarding only source-translates hairpin traffic to the VM's own public
IP.
**Analysis (4.22.1.1 sources):**
- `systemvm/debian/opt/cloud/bin/configure.py`,
`CsForwardingRules.processStaticNatRule`, lines 1559-1560 (4.23.0.0: line 1685):
`["nat", "front", "-A POSTROUTING -o %s -s %s/32 -j SNAT --to-source %s"]`
- `configure.py`, `CsSite2SiteVpn.configure_iptables`, line 1059 (4.23.0.0:
line 1184):
`["nat", "front", "-A POSTROUTING -t nat -o %s -m mark --mark 0x525 -j
ACCEPT"]`
- `systemvm/debian/opt/cloud/bin/cs/CsNetfilter.py`, lines 164-176: `front`
becomes `iptables -I <chain> <rule>` (insert at position 1), and a rule that
already exists is skipped (`has_rule`).
So on a fresh apply the databags are processed `forwardingrules` first and
`site2sitevpn` second (`configure.py` lines 1591-1595), the exemption is
inserted last and lands on top, and everything works. On an incremental apply
only the new rule is inserted: enabling static NAT after the connection exists
puts the SNAT rule at position 1 above the existing exemption, so the VM's
VPN-bound packets are translated before the exemption can match them. The order
then persists until the connection is deleted and recreated (measured) or the
router is rebuilt (expected from the fresh-apply order).
The VPN mark itself is set correctly (`mangle FORWARD -s <vpc cidr> -d <peer
cidr> -j MARK --set-xmark 0x525`), so the exemption matches whenever it is
reached. The VPC-wide source-NAT rule is appended rather than front-inserted
and is not affected; only per-VM static NAT is.
### versions
- Apache CloudStack 4.22.1.1, KVM, advanced zone, VPC with "Default VPC
offering" (NAT mode, VPC virtual router),
`DefaultIsolatedNetworkOfferingForVpcNetworks` tiers, IKEv2 site-to-site
connections between two VPC virtual routers.
- The same rules and insertion logic are present in 4.23.0.0 (line numbers
above).
### The steps to reproduce the bug
1. Two VPCs in one zone with non-overlapping CIDRs, e.g. `10.90.0.0/22` (A)
and `10.91.0.0/22` (B), one tier each, one VM each. Tier ACLs on both sides
allow ICMP and TCP/22 from the peer VPC CIDR. (A second VPC is only the
simplest peer; any site-to-site endpoint reproduces it.)
2. `createVpnGateway` on each VPC, a `createVpnCustomerGateway` on each side
describing the other (peer gateway IP, peer VPC CIDR, IKEv2), and
`createVpnConnection` on each side. Send traffic once; both connections reach
`Connected`.
3. `associateIpAddress` on VPC A and `enableStaticNat` it to VM A.
4. From VM A, ping VM B's private address and open TCP/22 to it.
Expected: traffic between VM A and VM B crosses the tunnel with private
source and destination addresses, as it does for a VM without static NAT.
Static NAT should apply to internet-bound traffic only.
Actual: both fail; with the peer tier temporarily open, VM B sees the
connection arriving from VM A's public address.
### What to do about it?
Make the static-NAT rule itself skip VPN-marked traffic, so chain order
stops mattering:
```
-A POSTROUTING -o <dev> -s <vm>/32 -m mark ! --mark 0x525 -j SNAT
--to-source <public ip>
```
Alternatives: re-assert the VPN exemption at position 1 whenever forwarding
rules are (re)applied, or stop inserting the per-VM SNAT rule at the front. The
first option is the smallest change, and it matches what a Neutron router does
for floating IPs, which are exempted from source translation by an IPsec policy
match.
Remote-access VPN was not checked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
