MitchDrage opened a new pull request, #14607: URL: https://github.com/apache/cloudstack/pull/14607
### Description This PR applies `network.usage.exempt.cidrs` to VMs on shared networks on KVM. It builds on #14583, which adds the setting for virtual routers, so this branch includes that commit until #14583 merges. On shared networks the host reports usage from libvirt, which reads each VM NIC's tap counters. Those count every byte, so the agent counts exempt traffic separately on the same tap and reports the libvirt total minus that. How it works: 1. The management server sends each KVM host its zone's exempt list in `SetHostParamsCommand`, on connect and whenever the setting is updated or reset. 2. The agent keeps an nftables `netdev` table with the exempt addresses in two sets (IPv4 and IPv6). Each user VM's tap gets two chains: the ingress hook counts what the VM sends to exempt addresses, and the egress hook counts what it receives from them. A sync every 10 seconds adds chains for new taps (VM start, NIC plug, migration in) and repairs the table if it's flushed. 3. When usage is collected, the agent reads those counters and takes them off the libvirt totals for each NIC. A few details keep the numbers safe to bill. These are in `UsageExemptBilling`: - The ingress hook doesn't see the 14 byte Ethernet header, which the tap counters include, so 14 bytes per packet are added back to what the VM sends. - The management server treats a lower number as a counter reset and bills the previous value again. So the reported numbers for a tap never go down. The record lives in `/run`, so it survives an agent restart and a reboot clears it along with the tap counters. - If a tap's chains are wiped, the bytes they counted are kept. A tap with a new ifindex starts from zero, the same as libvirt. Why nftables: hooking the tap counts both directions whatever bridge or switch it's plugged into, Linux bridge or OVS. iptables only sees bridged traffic with `br_netfilter`, which changes filtering for every VM. ebtables only covers Linux bridges. tc would sit alongside the qdiscs libvirt already uses for rate limiting. This adds `nftables` to the agent's package dependencies. The egress hook needs Linux 5.16 or later, or a backport: | Host OS | Kernel | Works | |---|---|---| | EL9 (AlmaLinux 9.8 tested) | 5.14 with backport | Yes | | Ubuntu 24.04, Debian 12 | 6.8 / 6.1 | Yes (expected) | | Ubuntu 22.04 (GA kernel) | 5.15 | No (tested). The HWE kernel works | | Ubuntu 20.04, Debian 11, EL8 | 5.4 / 5.10 / 4.18 | No (expected) | On kernels without the hook, or without `nft`, the agent logs one warning and reports usage as before. Other changes and limits: - `configureHostParams` only refreshes the reconcile command logs when the reconcile setting is sent, which only happens on connect. Before this, any setting pushed to a running agent (for example `migrate.wait`) marked commands still in progress as interrupted. - Only user VMs' taps get counters. User VMs on isolated and VPC networks still get counters nobody reads, as the host can't tell which kind of network a NIC is on. That costs a minuscule amount of CPU per packet on those VMs. Fixing it needs the server to send extra NIC details on VM start, NIC plug and migration, so I've decided to leave that out of this PR and accept the trade-off. - OVS-DPDK NICs aren't covered. They have no tap, and from reading the code, usage isn't recorded for them today anyway. - If a VM adds its own VLAN tags to its traffic, exempt traffic is billed a little more: 4 bytes per packet with one tag, and the whole packet with two tags (Q-in-Q). - The sync runs every 10 seconds because the agent has no reliable hook for new taps, especially on a migration target. That's shorter than a VM takes to boot and send traffic, and a migrating VM's tap usually gets its counters during the memory copy. An idle sync is two short nft command. - After a migration the VM gets a new tap on the new host and its counters start from zero, the same as libvirt's today. - #14142 fixes skipped NICs in the same stats code. The two don't conflict. Fixes: #14578 ### Types of changes - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] New feature (non-breaking change which adds functionality) - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] Enhancement (improves an existing feature and functionality) - [ ] Cleanup (Code refactoring and cleanup, that may add test cases) - [ ] Build/CI - [ ] Test (unit or integration test code) ### Feature/Enhancement Scale or Bug Severity #### Feature/Enhancement Scale - [ ] Major - [x] Minor ### Screenshots (if appropriate): ### How Has This Been Tested? - Unit tests: `UsageExemptBillingTest` (the arithmetic), `UsageExemptCountersTest` (the nftables side against a fake `nft`) and `AgentManagerImplTest` (sending the list to one zone or all zones). - AlmaLinux 9.8 (kernel 5.14), real nftables and a tap driven by a test program: exempt traffic came off the totals to the byte in both directions. - Ubuntu 22.04 (kernel 5.15): the missing egress hook is detected, one warning is logged and usage is reported unchanged. The same happens without `nft` installed. - On a real KVM host (CloudStack 4.22.1.1, AlmaLinux 9.8): backported this PR and #14583 to 4.22.1.1, replaced the KVM plugin and engine-api jars on one host, and left the management servers stock. With a test VM on a shared network, 200 exempt pings (208,400 bytes each way) came off the usage the management server stored for that VM, to the byte. After rolling back, the stored figures kept rising with no reset. Sending the list from the server is covered by unit tests. #### How did you try to break this feature and the system with this change? - Ran `nft flush ruleset` between stats collections. The sync rebuilt the table and the exempt bytes already counted stayed off. - Restarted the agent before the server sent the list. It kept using the existing table, so the reported figures didn't jump up and then drop. - Removed a tap and created a new one with the same name. The new tap started from zero and the old chains were removed. - Fed the agent unreadable `nft` output. That collection kept the previous exempt counts and the stats request still succeeded. - Cleared the setting. Traffic already exempted stayed exempt and nothing new was counted. - With the setting unused, the agent doesn't run `nft` at all. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
