kiwiflyer opened a new pull request, #14735:
URL: https://github.com/apache/cloudstack/pull/14735

   ### Description
   
   This PR adds an OVN network extension: guest networking for KVM on an
   [OVN](https://www.ovn.org/) logical fabric, driven entirely by a script 
through
   `ovn-nbctl`.
   
   It needs **no changes to CloudStack** and nothing on the hypervisor beyond
   `ovn-controller`. It is a new extension under `extensions/OVN/`, alongside 
the existing
   `Proxmox`, `HyperV` and `MaaS` samples, plus one line in the Debian package 
file list.
   Nothing outside `extensions/` is touched, so no existing behaviour changes: a
   deployment that does not create and register this extension is unaffected.
   
   **Why the data path needs nothing new.** `OvsVifDriver` already handles
   `BroadcastDomainType.Lswitch` by emitting libvirt's
   `<virtualport type='openvswitch'><parameters 
interfaceid='...'/></virtualport>`, which
   libvirt turns into `external_ids:iface-id` on the `br-int` port, and 
`ovn-controller`
   binds that to the `Logical_Switch_Port` of the same name. 
`HypervisorGuruBase.toNicTO()`
   copies `NicVO.getUuid()` into `NicTO` — under a comment still referring to 
Nicira
   integration — so the uuid the extension is handed in `prepare-nic` is the 
same one the
   agent stamps on the VIF. OVN inherited NVP's port-binding convention, so the 
plumbing
   CloudStack shipped for NSX-MH is exactly what OVN needs.
   
   **Services implemented**, across 25 script commands:
   
   | Service | Implemented with |
   |---|---|
   | `Connectivity`, `Dhcp` | a logical switch per network, native OVN DHCP |
   | `SourceNat`, `StaticNat` | `NAT` rows on a per-network logical router |
   | `PortForwarding` | `Load_Balancer` VIPs — OVN's NAT has no L4 translation |
   | `Firewall` | `ACL` rows on the guest switch |
   | VPC + `NetworkACL` | one logical router per VPC, a leg and ACLs per tier |
   | EVPN (optional, off by default) | `external_ids` tags read by a companion 
`ovn-bgp-agent` |
   
   Also included:
   
   - `test-with-ovn.sh` — an integration test that runs the extension against a 
real OVN
     northbound database. It is not a unit test and does not run in CI; it 
needs an OVN to
     talk to.
   - `ovn-host-setup.sh` — the host-side prerequisites, which have no 
CloudStack API
     behind them: installing OVN, pointing `ovn-controller` at the southbound 
database,
     giving a chassis a stable name, the provider bridge and its 
`ovn-bridge-mappings`
     entry, and switching the agent to `OvsVifDriver`. Three roles, idempotent, 
with
     `--dry-run` and `--verify`.
   
   **Private gateways, VPC static routes and private gateway ACLs** are 
implemented in the
   script but depend on a companion change to `NetworkExtensionElement`, which 
throws
   `UnsupportedOperationException` for all four today. On stock CloudStack 
those commands
   are simply never dispatched and the script behaves exactly as it does 
without them.
   
   There is a PR already out there for the Network Extension additions: 
https://github.com/apache/cloudstack/pull/14734
   
   <!-- Fixes: # -->
   
   ### Types of changes
   
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to change)
   - [x] New feature (non-breaking change which adds functionality)
   - [ ] Bug fix (non-breaking change which fixes an issue)
   - [ ] Enhancement (improves an existing feature and functionality)
   - [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
   - [ ] Build/CI
   - [ ] Test (unit or integration test code)
   
   ### Feature/Enhancement Scale or Bug Severity
   
   #### Feature/Enhancement Scale
   
   - [x] Major
   - [ ] Minor
   
   #### Bug Severity
   
   - [ ] BLOCKER
   - [ ] Critical
   - [ ] Major
   - [ ] Minor
   - [ ] Trivial
   
   ### Screenshots (if appropriate):
   
   n/a
   
   ### How Has This Been Tested?
   
   Against a real OVN 24.03 — northbound database, `ovn-northd`, and live 
`ovn-controller`
   chassis — not against a mock. `./test-with-ovn.sh` reproduces all of it.
   
   **Full lifecycle, 87 assertions.** Every one of the 25 commands, asserting 
on the
   resulting NB objects *and* on `ovn-northd`'s log, because the NB database 
will happily
   store a configuration northd cannot translate into logical flows — that last 
check is
   the one that proves anything. 0 northd errors, 326 logical flows generated.
   
   **Port binding on a real chassis.** `ovs-vswitchd` and `ovn-controller` were 
brought up
   and registered as a chassis. Two OVS ports carrying nothing but
   `external_ids:iface-id=<NIC uuid>` — precisely what libvirt writes — were 
bound by
   `ovn-controller` to the switch ports the script created, and the NB ports 
went `up`:
   
   ```
   nic-bind-0001    Port_Binding.chassis=chassis-a    Logical_Switch_Port up=up
   nic-bind-0002    Port_Binding.chassis=chassis-a    Logical_Switch_Port up=up
   ```
   
   The controller also claimed the distributed gateway port on the chassis
   `ovn_gateway_chassis` named, without prompting.
   
   **Real packets.** Frames injected with `ovs-ofctl packet-out` and the flow 
counters
   watched, so this is the installed datapath rather than a simulation of it:
   
   - three ICMP echo requests on one VIF's port came out the other's 
(`n_packets=0 -> 3`,
     183 bytes, which is 3 x the 61-byte packet)
   - two DHCP DISCOVERs of 292 bytes produced two **328**-byte replies back to 
the client
     port. The other VIF's counter never moved, so those are OVN's own DHCP 
OFFERs rather
     than the broadcast being flooded — there is no DHCP server in the topology 
at all
   
   `ovn-trace` confirms the options come from the script's own configuration:
   `put_dhcp_opts(offerip = 10.5.1.10, dns_server = {8.8.8.8}, lease_time = 
86400,
   mtu = 1442, netmask = 255.255.255.0, router = 10.5.1.1, server_id = 
10.5.1.1)`, and that
   VM-to-internet reaches `ct_snat` on the gateway port and leaves via the 
localnet port.
   
   **Two chassis.** A second chassis on a real Geneve tunnel, BFD up both ways 
with
   `forwarding=true`. `ovn_gateway_chassis` produced `Gateway_Chassis` rows at 
priority 100
   and 90 in the order given, northd built the `HA_Chassis_Group`, and the 
highest-priority
   chassis claimed the distributed gateway port. On host loss the survivor's 
BFD reported
   `Control Detection Time Expired`.
   
   **Payload replay, 21 assertions.** With the companion 
`NetworkExtensionElement` change,
   the payloads that class actually produces are captured by a unit test and 
replayed here
   against real OVN, substituting only `physical-network-extension-details`. 
Unit tests on
   one side and script tests on the other are each checked against fixtures 
written by the
   same hand, so both would pass if a key name drifted; this closes that.
   
   **Concurrency, 13 assertions.** Covered below.
   
   **Not tested: :** no CloudStack management server driving a real
   zone, no libvirt-managed VIF (packets were injected at the OpenFlow layer, 
so this
   exercises the convention the agent relies on rather than the agent itself), 
no
   `ovn-bgp-agent` so EVPN remains tag-only, shared networks, and IPv6.
   
   #### How did you try to break this feature and the system with this change?
   
   Testing against a real database found four bugs that a mock would not have. 
All are
   fixed in this PR, with the reproduction kept as a regression test.
   
   **1. OVSDB garbage-collects rows in non-root tables.** The script created an 
`ACL` or
   `Logical_Router_Static_Route` row and attached it to its parent in a second
   `ovn-nbctl` call. A row in a non-root table that nothing references by the 
end of the
   transaction that created it is deleted, so it was gone before the second 
call ran:
   
   ```
   ovn-nbctl: applying "create" command to table ACL without --id option will 
have no effect
   transaction error: ... references nonexistent row ... "error":"referential 
integrity violation"
   ```
   
   That broke **every firewall rule, every VPC network ACL and every static 
route**. Both
   sites now create and attach in one transaction with `--id=@name`. 
`DHCP_Options` and
   `Load_Balancer` *are* root tables, which is why the identical pattern was 
correct for
   them and the fault stayed invisible against a stub.
   
   **2. `--may-exist` is not atomic.** Only `Logical_Switch_Port` and
   `Logical_Router_Port` carry a name index in the NB schema. `Logical_Switch`,
   `Logical_Router`, `DHCP_Options`, `Load_Balancer` and `ACL` do not, so OVSDB 
will not
   refuse a duplicate:
   
   ```
   8 x ovn-nbctl --may-exist ls-add dupetest   ->   3 switches named dupetest
   ```
   
   `ensure_dhcp_options` is called from both `implement-network` and 
`prepare-nic`, so a
   network coming up while instances are already deploying hit this routinely — 
up to
   8 `DHCP_Options` rows for one network with instances bound across 2 or 3 of 
them. That
   failure is silent: a later `config-dhcp-subnet` updates one row and the 
instances on the
   others keep being handed stale DNS, MTU and lease settings indefinitely.
   
   Fixed with a per-command file lock keyed on the network — or on the VPC, 
since tiers
   share a router — plus a second lock for the external switch, which is shared 
by every
   network on a public VLAN. Duplicate `DHCP_Options` rows are additionally 
collapsed
   deterministically, for the case a file lock cannot reach: a second 
management server.
   
   **3. My first version of that fix was also wrong,** and the test caught it.
   `dhcpv4_options` is a *weak* reference, so destroying a row silently empties 
the column
   on every port pointing at it. Because OVSDB uuids are random, two racers can 
briefly
   disagree about which row wins — 3 of 10 instances ended with no DHCP at all. 
Ports are
   now re-pointed before any row is destroyed, and a sweep repairs anything a 
concurrent
   collapse orphaned.
   
   **4. `ovs-vsctl add-port` succeeds for a device that does not exist.** It 
records the
   port and leaves the interface in an error state, so a typo in 
`ovn-host-setup.sh
   --public-nic` reported success and would have surfaced much later as 
"north-south
   traffic does not work". It now checks the interface's `error` column and 
fails with the
   `del-port` command to undo it.
   
   What is still **not** safe: 
   Two management servers creating the same logical switch at the same instant 
can still
   produce two of them, which splits the network. Nothing `ovn-nbctl` offers 
can prevent
   that — it needs a compare-and-swap, which means OVSDB `wait` operations and 
a native
   client. The script now logs it loudly instead of leaving it to be 
discovered. In
   practice CloudStack serialises most per-network work with its own locks, so 
this is
   defence in depth rather than a routine hazard.
   
   Other adversarial cases, all regression tests: missing `ovn_nb_db`; missing 
required
   payload fields; malformed payload JSON; an unwritable log directory; 
`ovn-nbctl`
   failing; `ovn-nbctl` writing noise to stdout *and* stderr (the management 
server merges
   them and JSON-parses the result, so one stray line breaks 
`implement-network` — the
   script writes its result to a saved fd 3 that a redirecting call site cannot 
swallow);
   NAT arriving before the router exists; port-forward ranges that are over-cap,
   mismatched or use an unsupported protocol; both egress default policies; and 
unknown
   commands, which return success rather than wedging a deployment with an 
over-broad
   offering.
   
   `ovn-host-setup.sh` was exercised too: dry run changes nothing, a real run 
converges, a
   second run reports everything already set, bridge mappings merge rather than 
overwrite
   another provider, and it refuses to enslave the NIC carrying the host's 
default route
   without `--force`, because doing so cuts the host off.
   
   **Operational note for reviewers who test failover:** BFD is answered by
   `ovs-vswitchd`, not `ovn-controller`. Stopping `ovn-controller` on the 
active gateway
   chassis fails nothing over — the peer still sees BFD up and correctly leaves 
the port
   where it is. The switch or the host has to go down. This is easy to mistake 
for the
   feature being broken.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to