kiwiflyer opened a new pull request, #14735: URL: https://github.com/apache/cloudstack/pull/14735
### Description This PR adds an OVN network extension: guest networking for KVM on an [OVN](https://www.ovn.org/) logical fabric, driven entirely by a script through `ovn-nbctl`. It needs **no changes to CloudStack** and nothing on the hypervisor beyond `ovn-controller`. It is a new extension under `extensions/OVN/`, alongside the existing `Proxmox`, `HyperV` and `MaaS` samples, plus one line in the Debian package file list. Nothing outside `extensions/` is touched, so no existing behaviour changes: a deployment that does not create and register this extension is unaffected. **Why the data path needs nothing new.** `OvsVifDriver` already handles `BroadcastDomainType.Lswitch` by emitting libvirt's `<virtualport type='openvswitch'><parameters interfaceid='...'/></virtualport>`, which libvirt turns into `external_ids:iface-id` on the `br-int` port, and `ovn-controller` binds that to the `Logical_Switch_Port` of the same name. `HypervisorGuruBase.toNicTO()` copies `NicVO.getUuid()` into `NicTO` — under a comment still referring to Nicira integration — so the uuid the extension is handed in `prepare-nic` is the same one the agent stamps on the VIF. OVN inherited NVP's port-binding convention, so the plumbing CloudStack shipped for NSX-MH is exactly what OVN needs. **Services implemented**, across 25 script commands: | Service | Implemented with | |---|---| | `Connectivity`, `Dhcp` | a logical switch per network, native OVN DHCP | | `SourceNat`, `StaticNat` | `NAT` rows on a per-network logical router | | `PortForwarding` | `Load_Balancer` VIPs — OVN's NAT has no L4 translation | | `Firewall` | `ACL` rows on the guest switch | | VPC + `NetworkACL` | one logical router per VPC, a leg and ACLs per tier | | EVPN (optional, off by default) | `external_ids` tags read by a companion `ovn-bgp-agent` | Also included: - `test-with-ovn.sh` — an integration test that runs the extension against a real OVN northbound database. It is not a unit test and does not run in CI; it needs an OVN to talk to. - `ovn-host-setup.sh` — the host-side prerequisites, which have no CloudStack API behind them: installing OVN, pointing `ovn-controller` at the southbound database, giving a chassis a stable name, the provider bridge and its `ovn-bridge-mappings` entry, and switching the agent to `OvsVifDriver`. Three roles, idempotent, with `--dry-run` and `--verify`. **Private gateways, VPC static routes and private gateway ACLs** are implemented in the script but depend on a companion change to `NetworkExtensionElement`, which throws `UnsupportedOperationException` for all four today. On stock CloudStack those commands are simply never dispatched and the script behaves exactly as it does without them. There is a PR already out there for the Network Extension additions: https://github.com/apache/cloudstack/pull/14734 <!-- Fixes: # --> ### Types of changes - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] New feature (non-breaking change which adds functionality) - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] Enhancement (improves an existing feature and functionality) - [ ] Cleanup (Code refactoring and cleanup, that may add test cases) - [ ] Build/CI - [ ] Test (unit or integration test code) ### Feature/Enhancement Scale or Bug Severity #### Feature/Enhancement Scale - [x] Major - [ ] Minor #### Bug Severity - [ ] BLOCKER - [ ] Critical - [ ] Major - [ ] Minor - [ ] Trivial ### Screenshots (if appropriate): n/a ### How Has This Been Tested? Against a real OVN 24.03 — northbound database, `ovn-northd`, and live `ovn-controller` chassis — not against a mock. `./test-with-ovn.sh` reproduces all of it. **Full lifecycle, 87 assertions.** Every one of the 25 commands, asserting on the resulting NB objects *and* on `ovn-northd`'s log, because the NB database will happily store a configuration northd cannot translate into logical flows — that last check is the one that proves anything. 0 northd errors, 326 logical flows generated. **Port binding on a real chassis.** `ovs-vswitchd` and `ovn-controller` were brought up and registered as a chassis. Two OVS ports carrying nothing but `external_ids:iface-id=<NIC uuid>` — precisely what libvirt writes — were bound by `ovn-controller` to the switch ports the script created, and the NB ports went `up`: ``` nic-bind-0001 Port_Binding.chassis=chassis-a Logical_Switch_Port up=up nic-bind-0002 Port_Binding.chassis=chassis-a Logical_Switch_Port up=up ``` The controller also claimed the distributed gateway port on the chassis `ovn_gateway_chassis` named, without prompting. **Real packets.** Frames injected with `ovs-ofctl packet-out` and the flow counters watched, so this is the installed datapath rather than a simulation of it: - three ICMP echo requests on one VIF's port came out the other's (`n_packets=0 -> 3`, 183 bytes, which is 3 x the 61-byte packet) - two DHCP DISCOVERs of 292 bytes produced two **328**-byte replies back to the client port. The other VIF's counter never moved, so those are OVN's own DHCP OFFERs rather than the broadcast being flooded — there is no DHCP server in the topology at all `ovn-trace` confirms the options come from the script's own configuration: `put_dhcp_opts(offerip = 10.5.1.10, dns_server = {8.8.8.8}, lease_time = 86400, mtu = 1442, netmask = 255.255.255.0, router = 10.5.1.1, server_id = 10.5.1.1)`, and that VM-to-internet reaches `ct_snat` on the gateway port and leaves via the localnet port. **Two chassis.** A second chassis on a real Geneve tunnel, BFD up both ways with `forwarding=true`. `ovn_gateway_chassis` produced `Gateway_Chassis` rows at priority 100 and 90 in the order given, northd built the `HA_Chassis_Group`, and the highest-priority chassis claimed the distributed gateway port. On host loss the survivor's BFD reported `Control Detection Time Expired`. **Payload replay, 21 assertions.** With the companion `NetworkExtensionElement` change, the payloads that class actually produces are captured by a unit test and replayed here against real OVN, substituting only `physical-network-extension-details`. Unit tests on one side and script tests on the other are each checked against fixtures written by the same hand, so both would pass if a key name drifted; this closes that. **Concurrency, 13 assertions.** Covered below. **Not tested: :** no CloudStack management server driving a real zone, no libvirt-managed VIF (packets were injected at the OpenFlow layer, so this exercises the convention the agent relies on rather than the agent itself), no `ovn-bgp-agent` so EVPN remains tag-only, shared networks, and IPv6. #### How did you try to break this feature and the system with this change? Testing against a real database found four bugs that a mock would not have. All are fixed in this PR, with the reproduction kept as a regression test. **1. OVSDB garbage-collects rows in non-root tables.** The script created an `ACL` or `Logical_Router_Static_Route` row and attached it to its parent in a second `ovn-nbctl` call. A row in a non-root table that nothing references by the end of the transaction that created it is deleted, so it was gone before the second call ran: ``` ovn-nbctl: applying "create" command to table ACL without --id option will have no effect transaction error: ... references nonexistent row ... "error":"referential integrity violation" ``` That broke **every firewall rule, every VPC network ACL and every static route**. Both sites now create and attach in one transaction with `--id=@name`. `DHCP_Options` and `Load_Balancer` *are* root tables, which is why the identical pattern was correct for them and the fault stayed invisible against a stub. **2. `--may-exist` is not atomic.** Only `Logical_Switch_Port` and `Logical_Router_Port` carry a name index in the NB schema. `Logical_Switch`, `Logical_Router`, `DHCP_Options`, `Load_Balancer` and `ACL` do not, so OVSDB will not refuse a duplicate: ``` 8 x ovn-nbctl --may-exist ls-add dupetest -> 3 switches named dupetest ``` `ensure_dhcp_options` is called from both `implement-network` and `prepare-nic`, so a network coming up while instances are already deploying hit this routinely — up to 8 `DHCP_Options` rows for one network with instances bound across 2 or 3 of them. That failure is silent: a later `config-dhcp-subnet` updates one row and the instances on the others keep being handed stale DNS, MTU and lease settings indefinitely. Fixed with a per-command file lock keyed on the network — or on the VPC, since tiers share a router — plus a second lock for the external switch, which is shared by every network on a public VLAN. Duplicate `DHCP_Options` rows are additionally collapsed deterministically, for the case a file lock cannot reach: a second management server. **3. My first version of that fix was also wrong,** and the test caught it. `dhcpv4_options` is a *weak* reference, so destroying a row silently empties the column on every port pointing at it. Because OVSDB uuids are random, two racers can briefly disagree about which row wins — 3 of 10 instances ended with no DHCP at all. Ports are now re-pointed before any row is destroyed, and a sweep repairs anything a concurrent collapse orphaned. **4. `ovs-vsctl add-port` succeeds for a device that does not exist.** It records the port and leaves the interface in an error state, so a typo in `ovn-host-setup.sh --public-nic` reported success and would have surfaced much later as "north-south traffic does not work". It now checks the interface's `error` column and fails with the `del-port` command to undo it. What is still **not** safe: Two management servers creating the same logical switch at the same instant can still produce two of them, which splits the network. Nothing `ovn-nbctl` offers can prevent that — it needs a compare-and-swap, which means OVSDB `wait` operations and a native client. The script now logs it loudly instead of leaving it to be discovered. In practice CloudStack serialises most per-network work with its own locks, so this is defence in depth rather than a routine hazard. Other adversarial cases, all regression tests: missing `ovn_nb_db`; missing required payload fields; malformed payload JSON; an unwritable log directory; `ovn-nbctl` failing; `ovn-nbctl` writing noise to stdout *and* stderr (the management server merges them and JSON-parses the result, so one stray line breaks `implement-network` — the script writes its result to a saved fd 3 that a redirecting call site cannot swallow); NAT arriving before the router exists; port-forward ranges that are over-cap, mismatched or use an unsupported protocol; both egress default policies; and unknown commands, which return success rather than wedging a deployment with an over-broad offering. `ovn-host-setup.sh` was exercised too: dry run changes nothing, a real run converges, a second run reports everything already set, bridge mappings merge rather than overwrite another provider, and it refuses to enslave the NIC carrying the host's default route without `--force`, because doing so cuts the host off. **Operational note for reviewers who test failover:** BFD is answered by `ovs-vswitchd`, not `ovn-controller`. Stopping `ovn-controller` on the active gateway chassis fails nothing over — the peer still sees BFD up and correctly leaves the port where it is. The switch or the host has to go down. This is easy to mistake for the feature being broken. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
