On Fri, Sep 18, 2026 at 04:23:03PM +0900, Jaygue Lee wrote:
> Hi,
>
> This series fixes two bugs found while investigating a production incident
> on OVN 25.09.0, and asks for direction on a third gap that the two fixes do
> not cover.
>
> Summary
> -------
>
> A load balancer backend is probed only by the ovn-controller of the chassis
> its port is bound to, and only that ovn-controller writes the
> Service_Monitor status. When the chassis dies, nothing expires the last
> "online": the Service_Monitor row keeps status=online until the chassis
> comes back and probes again, and northd keeps sending traffic to the dead
> backend for as long as that takes.
>
> This is not specific to gateway chassis. Every chassis that hosts a
> backend probes it with its own pinctrl, so a compute-only hypervisor dying
> produces the same result. In the deployment below the controllers also
> host VMs, which is only why the same host's timeline shows both the BFD
> failover of its cr-lrps and the Service_Monitor that nobody probed anymore:
> OVN did notice the chassis was gone within seconds, but that knowledge is
> not connected to service monitors.
>
> What happened
> -------------
>
> hv1 hosts VM B, a backend of a TCP load balancer with a health check. hv1
> is also a gateway chassis, which is incidental here. The host suffered
> I/O starvation, then died, then was hard-reset. In stages:
>
> 1. ovn-controller stalled, host alive. The main loop blocked for up to
> 275 s at a time, the pinctrl thread for up to 246 s, and the SB
> connection was lost. Nothing probed B. Service_Monitor for B:
> status=online, chassis_name=hv1.
>
> 2. Host dead. hv2 logged
> Changing chassis for lport cr-lrp-... from hv1 to hv2
> i.e. BFD detected the loss and the gateway failed over. The
> Service_Monitor for B kept status=online. VIP requests failed ~50%:
> the active backends were B (dead) and one healthy one.
>
> 3. Host rebooted, VM B not started. Its tap does not exist, so
> ovs-vswitchd: could not open network device tap<B> (No such device)
> and the fresh ovn-controller released the ports still bound to it:
> binding|INFO|Removing iface tap<B> ovn-installed in OVS
> binding|INFO|Releasing lport <B> from this chassis (sb_readonly=0)
> if_status|WARN|Trying to release unknown interface <X>
> No "Setting lport <B> down in Southbound" followed. SB:
> Port_Binding <B>: chassis=[] up=[true]
> Service_Monitor <B>: chassis_name=hv1 status=online
> northd derived NB Logical_Switch_Port.up=false from the empty chassis,
> so the CMS saw the port DOWN while the Service_Monitor kept
> status=online.
>
> 4. VM B started, hv1 re-claimed the port, a probe succeeded and the
> Service_Monitor status finally changed. Nothing in between touched it.
>
> Why
> ---
>
> 1. Only the owning chassis probes and writes the status.
> controller/pinctrl.c sync_svc_monitors():
> if (pb->chassis != our_chassis) continue;
> Nothing else writes "online" and nothing expires the status.
>
> 2. northd's safety net only looks at "up". Commit 23e203a3f ("northd: set
> svc_mon status to offline if port_binding released") sets
> Service_Monitor.status to offline when Port_Binding.up is false, but
> not when the port is unbound (chassis empty, up still true).
> chassis_name is also never cleared, so it keeps naming a chassis that
> stopped probing. The same northd derives NB Logical_Switch_Port.up=false
> from an empty chassis, so the CMS and the Service_Monitor disagree about
> the same port.
>
> 3. ovn-controller can leave a port unbound-but-up. A port that was already
> bound to this chassis when ovn-controller started is not tracked by
> if-status. When the recompute path releases it (no local binding
> because the interface cannot be opened), release_lport() clears the
> chassis but if_status_mgr_release_iface() returns early on an unknown
> interface, so "up" is never cleared. The incremental path
> (release_binding_lport()) does set it down.
>
> 4. Chassis liveness is not connected to service monitors. BFD is enabled
> only between HA chassis group members and their ref_chassis
> (controller/bfd.c bfd_calculate_chassis()), its result lives in the
> local OVS Interface.bfd_status only, and it is consumed only for cr-lrp
> and external port election. northd does not use Chassis_Private
> nb_cfg/nb_cfg_timestamp for anything but hv_cfg.
>
> This series
> -----------
>
> Patch 1 (northd) treats an unbound backend port like a port that is down:
> Service_Monitor.status is set to offline and chassis_name is cleared.
> This covers stage 3 above and also "ovn-appctl exit" without --restart and
> "ovn-sbctl chassis-del", both of which clear the chassis while leaving "up"
> true. ovn-controller still owns the transition back to "online" once the
> port is bound again and a probe succeeds; northd never writes "online".
> The ovn-ic service monitor test emulated probed backends by setting "up"
> without a chassis; it now binds the ports.
>
> Patch 2 (ovn-controller) sets "up" to false when release_lport() releases a
> port that if-status does not track, matching the tracked path. With it the
> unbound-but-up state of stage 3 is not produced in the first place.
>
> The two are independent and can be applied in either order. We would like
> both considered for branch-25.09: patch 2 applies there as is, patch 1
> needs a small adaptation (the LSP health check path does not exist there)
> which we can send separately.
>
> What this series does not fix, and a question
> ---------------------------------------------
>
> If the chassis simply dies (or its ovn-controller is stopped with SIGTERM,
> which skips the cleanup path), Port_Binding.chassis and up stay as they
> were, so neither patch applies and the Service_Monitor keeps status=online
> until the chassis comes back. This is stage 2 above, and it applies to
> any hypervisor that hosts a backend.
>
> The information needed to close this gap already exists: every chassis
> hosting a backend has a BFD session with the gateway chassis (northd puts
> it in ref_chassis as soon as a port is bound; we measured the session
> coming up within about 1 s of the binding), and BFD reported hv1 down
> within seconds. What is missing is a consumer. Two shapes we considered:
>
> A. ovn-controller on a gateway chassis sets Service_Monitor rows whose
> chassis_name is a BFD-down peer to "offline" (edge-triggered, only
> while it still sees a majority of its own BFD peers up). No schema
> change, but a non-owner then writes the status, and true quorum is
> not possible because observers do not share their view.
>
> B. Each chassis publishes the set of BFD-down peers (e.g. a new Chassis
> column), and northd sets a Service_Monitor offline when a majority of
> the HA chassis of the groups that reference the backend's chassis
> report it down. Single writer and a real quorum, at the cost of a
> schema change.
>
> Before writing either we would like to know: is it acceptable for northd
> (or a non-owning chassis) to set a Service_Monitor offline based on chassis
> liveness at all, and if so which of the two shapes fits the project?
> Should it be opt-in? A stalled ovn-controller on a live host (stage 1)
> is not visible to BFD and would need a prober liveness signal instead; we
> consider that out of scope for now.
>
> Jaygue Lee (2):
> northd: Mark unbound ports' service monitors offline.
> controller: Set untracked ports down on release.
>
> controller/binding.c | 8 +++++
> northd/northd.c | 25 ++++++++------
> tests/ovn-controller.at | 42 +++++++++++++++++++++++
> tests/ovn-ic.at | 18 +++++-----
> tests/ovn-northd.at | 75 +++++++++++++++++++++++++++++++++++++++++
> 5 files changed, 150 insertions(+), 18 deletions(-)
>
> --
> 2.49.0
>
> _______________________________________________
> dev mailing list
> [email protected]
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
Series LGTM.
Acked-by: Mairtin O'Loingsigh <[email protected]>
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
