Tn9hia opened a new issue, #14432:
URL: https://github.com/apache/cloudstack/issues/14432
### problem
Since #10594 ("security groups: Do not use conntrack when it is not
required"), a VM can reach **any port** of another VM that runs **on the same
KVM host and the same bridge**, regardless of the destination VM's ingress
rules. Traffic from outside the host is still filtered correctly.
This breaks tenant isolation. The source and destination VMs can belong to
different accounts.
**Root cause**
Each bridge chain evaluates the source VM first (`BF-<br>-IN`, `--physdev-in
<src vif>`) and the destination VM second (`BF-<br>-OUT`, `--physdev-out <dst
vif>`):
```
-A BF-br -m physdev --physdev-is-in --physdev-is-bridged -j BF-br-IN
-A BF-br -m physdev --physdev-is-out --physdev-is-bridged -j BF-br-OUT
```
Since #10594, the source VM's egress chain ends in a terminating `ACCEPT`
when the security group has no egress rules (and allowed egress rules also use
`ACCEPT`):
```python
# add_network_rules()
if egressrule_v4 == 0 :
execute('iptables -A ' + egress_vmchain + ' -j ACCEPT')
```
The packet is accepted inside `BF-<br>-IN`, so `BF-<br>-OUT` is never
reached and the destination VM's ingress chain is never evaluated. In 4.20 the
egress chain ended with `RETURN`, which let the packet go on to the destination
VM's chains.
| | 4.20 | 4.23 / main |
|---|---|---|
| Egress rule matched | `RETURN` | `ACCEPT` |
| No egress rules (default allow) | `RETURN` | `ACCEPT` |
| Egress rules present, no match | `DROP` | `RETURN` → `DROP` at the end of
`-def` |
| End of `<vm>-def` for `--physdev-in` | none (implicit return) | `DROP` |
| Destination ingress enforced for same-host traffic | yes | **no** |
When the two VMs run on different hosts, the destination host receives the
packet on the uplink and evaluates `BF-<br>-OUT`, so ingress is enforced there.
The bug only affects VMs on the same host and bridge.
### versions
CLOUDSTACK VERSION: 4.23.0.0 (cloudstack-agent 4.23.0.0). Also present on
current main.
COMPONENT: KVM agent – scripts/vm/network/security_group.py (Security Groups)
OS: KVM host: Ubuntu 24.04, iptables-nft
CONFIGURATION:
- Advanced zone with security groups enabled
- Shared guest network (VLAN), Linux bridge (brXXX-<vlan>)
- net.bridge.bridge-nf-call-iptables = 1
### The steps to reproduce the bug
#### STEPS TO REPRODUCE
1. In an advanced zone with security groups, create two security groups:
* `SG-A`: no rules at all (default egress allow).
* `SG-B`: one ingress rule, TCP 22 from `1.1.1.1/32` only.
2. Deploy `VM-A` with `SG-A` and `VM-B` with `SG-B` on the same shared
network, and make sure both run on the **same KVM host**. For example, use a
single host, or migrate one VM to the other's host.
3. From a machine outside the host (not `1.1.1.1`), run `ssh <VM-B IP>`. The
connection is blocked, as expected.
4. From inside `VM-A`, run `ssh <VM-B IP>`. **The connection succeeds.**
#### EXPECTED RESULTS
```
VM-B's ingress rules are enforced for all sources, including VMs on the same
host.
SSH from VM-A to VM-B is dropped.
```
#### ACTUAL RESULTS
```
SSH from VM-A to VM-B succeeds. Only VM-A's egress rules are evaluated;
VM-B's ingress chain is skipped for same-host, same-bridge traffic.
```
### What to do about it?
#### PROPOSED FIX
Use `RETURN` instead of `ACCEPT` for allowed egress, so the packet goes back
to `BF-<br>` and continues to `BF-<br>-OUT`. This is the same pattern the
system VM chains use (`-A r-XXX-VM -m physdev --physdev-in vnetX -j RETURN`).
Traffic to the uplink is still accepted by `-A BF-<br> -m physdev --physdev-out
<uplink> -j ACCEPT`.
Explicit egress denial moves into the egress chain. `ESTABLISHED,RELATED` is
kept there so replies to allowed inbound connections are not dropped by the
egress rules.
```diff
@@ default_network_rules()
- execute("iptables -A " + vmchain_default + " -m physdev
--physdev-is-bridged --physdev-in " + vif + " -j DROP")
+ execute("iptables -A " + vmchain_default + " -m physdev
--physdev-is-bridged --physdev-in " + vif + " -j RETURN")
@@
- execute('ip6tables -A ' + vmchain_default + ' -m physdev
--physdev-is-bridged --physdev-in ' + vif + ' -j DROP')
+ execute('ip6tables -A ' + vmchain_default + ' -m physdev
--physdev-is-bridged --physdev-in ' + vif + ' -j RETURN')
@@ add_network_rules()
if rule['ruletype'] == 'E':
vmchain = egress_vmchain
direction = "-d"
- action = "ACCEPT"
+ action = "RETURN"
@@
if egressrule_v4 == 0 :
- execute('iptables -A ' + egress_vmchain + ' -j ACCEPT')
+ execute('iptables -A ' + egress_vmchain + ' -j RETURN')
else:
- execute('iptables -A ' + egress_vmchain + ' -j RETURN')
+ execute('iptables -A ' + egress_vmchain + ' -m state --state
ESTABLISHED,RELATED -j RETURN')
+ execute('iptables -A ' + egress_vmchain + ' -j DROP')
if egressrule_v6 == 0 :
- execute('ip6tables -A ' + egress_vmchain + ' -j ACCEPT')
+ execute('ip6tables -A ' + egress_vmchain + ' -j RETURN')
else:
- execute('ip6tables -A ' + egress_vmchain + ' -j RETURN')
+ execute('ip6tables -A ' + egress_vmchain + ' -m state --state
ESTABLISHED,RELATED -j RETURN')
+ execute('ip6tables -A ' + egress_vmchain + ' -j DROP')
```
We tested the IPv4 part on 4.23.0.0. After a stop/start of the VMs so the
`-def` chains are rebuilt:
* Same-host traffic from VM-A to VM-B is dropped by VM-B's ingress chain.
* External access that VM-B's ingress rules allow still works.
* VM outbound and internet access is unaffected.
The IPv6 part mirrors the IPv4 change but has not been tested.
Note: VMs added to `cs_notrack` (any rule with `protocol=all` from
`0.0.0.0/0`) have no conntrack state, so the `ESTABLISHED,RELATED` rule does
not match for them. That behaviour is unchanged from 4.23. Also, the
`conntrack*_not_needed` check does not look at the rule type, so an *egress*
`all 0.0.0.0/0` rule also disables conntrack for the VM. This may deserve a
separate look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
