Hi,
This series fixes two bugs found while investigating a production incident
on OVN 25.09.0, and asks for direction on a third gap that the two fixes do
not cover.
Summary
-------
A load balancer backend is probed only by the ovn-controller of the chassis
its port is bound to, and only that ovn-controller writes the
Service_Monitor status. When the chassis dies, nothing expires the last
"online": the Service_Monitor row keeps status=online until the chassis
comes back and probes again, and northd keeps sending traffic to the dead
backend for as long as that takes.
This is not specific to gateway chassis. Every chassis that hosts a
backend probes it with its own pinctrl, so a compute-only hypervisor dying
produces the same result. In the deployment below the controllers also
host VMs, which is only why the same host's timeline shows both the BFD
failover of its cr-lrps and the Service_Monitor that nobody probed anymore:
OVN did notice the chassis was gone within seconds, but that knowledge is
not connected to service monitors.
What happened
-------------
hv1 hosts VM B, a backend of a TCP load balancer with a health check. hv1
is also a gateway chassis, which is incidental here. The host suffered
I/O starvation, then died, then was hard-reset. In stages:
1. ovn-controller stalled, host alive. The main loop blocked for up to
275 s at a time, the pinctrl thread for up to 246 s, and the SB
connection was lost. Nothing probed B. Service_Monitor for B:
status=online, chassis_name=hv1.
2. Host dead. hv2 logged
Changing chassis for lport cr-lrp-... from hv1 to hv2
i.e. BFD detected the loss and the gateway failed over. The
Service_Monitor for B kept status=online. VIP requests failed ~50%:
the active backends were B (dead) and one healthy one.
3. Host rebooted, VM B not started. Its tap does not exist, so
ovs-vswitchd: could not open network device tap<B> (No such device)
and the fresh ovn-controller released the ports still bound to it:
binding|INFO|Removing iface tap<B> ovn-installed in OVS
binding|INFO|Releasing lport <B> from this chassis (sb_readonly=0)
if_status|WARN|Trying to release unknown interface <X>
No "Setting lport <B> down in Southbound" followed. SB:
Port_Binding <B>: chassis=[] up=[true]
Service_Monitor <B>: chassis_name=hv1 status=online
northd derived NB Logical_Switch_Port.up=false from the empty chassis,
so the CMS saw the port DOWN while the Service_Monitor kept
status=online.
4. VM B started, hv1 re-claimed the port, a probe succeeded and the
Service_Monitor status finally changed. Nothing in between touched it.
Why
---
1. Only the owning chassis probes and writes the status.
controller/pinctrl.c sync_svc_monitors():
if (pb->chassis != our_chassis) continue;
Nothing else writes "online" and nothing expires the status.
2. northd's safety net only looks at "up". Commit 23e203a3f ("northd: set
svc_mon status to offline if port_binding released") sets
Service_Monitor.status to offline when Port_Binding.up is false, but
not when the port is unbound (chassis empty, up still true).
chassis_name is also never cleared, so it keeps naming a chassis that
stopped probing. The same northd derives NB Logical_Switch_Port.up=false
from an empty chassis, so the CMS and the Service_Monitor disagree about
the same port.
3. ovn-controller can leave a port unbound-but-up. A port that was already
bound to this chassis when ovn-controller started is not tracked by
if-status. When the recompute path releases it (no local binding
because the interface cannot be opened), release_lport() clears the
chassis but if_status_mgr_release_iface() returns early on an unknown
interface, so "up" is never cleared. The incremental path
(release_binding_lport()) does set it down.
4. Chassis liveness is not connected to service monitors. BFD is enabled
only between HA chassis group members and their ref_chassis
(controller/bfd.c bfd_calculate_chassis()), its result lives in the
local OVS Interface.bfd_status only, and it is consumed only for cr-lrp
and external port election. northd does not use Chassis_Private
nb_cfg/nb_cfg_timestamp for anything but hv_cfg.
This series
-----------
Patch 1 (northd) treats an unbound backend port like a port that is down:
Service_Monitor.status is set to offline and chassis_name is cleared.
This covers stage 3 above and also "ovn-appctl exit" without --restart and
"ovn-sbctl chassis-del", both of which clear the chassis while leaving "up"
true. ovn-controller still owns the transition back to "online" once the
port is bound again and a probe succeeds; northd never writes "online".
The ovn-ic service monitor test emulated probed backends by setting "up"
without a chassis; it now binds the ports.
Patch 2 (ovn-controller) sets "up" to false when release_lport() releases a
port that if-status does not track, matching the tracked path. With it the
unbound-but-up state of stage 3 is not produced in the first place.
The two are independent and can be applied in either order. We would like
both considered for branch-25.09: patch 2 applies there as is, patch 1
needs a small adaptation (the LSP health check path does not exist there)
which we can send separately.
What this series does not fix, and a question
---------------------------------------------
If the chassis simply dies (or its ovn-controller is stopped with SIGTERM,
which skips the cleanup path), Port_Binding.chassis and up stay as they
were, so neither patch applies and the Service_Monitor keeps status=online
until the chassis comes back. This is stage 2 above, and it applies to
any hypervisor that hosts a backend.
The information needed to close this gap already exists: every chassis
hosting a backend has a BFD session with the gateway chassis (northd puts
it in ref_chassis as soon as a port is bound; we measured the session
coming up within about 1 s of the binding), and BFD reported hv1 down
within seconds. What is missing is a consumer. Two shapes we considered:
A. ovn-controller on a gateway chassis sets Service_Monitor rows whose
chassis_name is a BFD-down peer to "offline" (edge-triggered, only
while it still sees a majority of its own BFD peers up). No schema
change, but a non-owner then writes the status, and true quorum is
not possible because observers do not share their view.
B. Each chassis publishes the set of BFD-down peers (e.g. a new Chassis
column), and northd sets a Service_Monitor offline when a majority of
the HA chassis of the groups that reference the backend's chassis
report it down. Single writer and a real quorum, at the cost of a
schema change.
Before writing either we would like to know: is it acceptable for northd
(or a non-owning chassis) to set a Service_Monitor offline based on chassis
liveness at all, and if so which of the two shapes fits the project?
Should it be opt-in? A stalled ovn-controller on a live host (stage 1)
is not visible to BFD and would need a prober liveness signal instead; we
consider that out of scope for now.
Jaygue Lee (2):
northd: Mark unbound ports' service monitors offline.
controller: Set untracked ports down on release.
controller/binding.c | 8 +++++
northd/northd.c | 25 ++++++++------
tests/ovn-controller.at | 42 +++++++++++++++++++++++
tests/ovn-ic.at | 18 +++++-----
tests/ovn-northd.at | 75 +++++++++++++++++++++++++++++++++++++++++
5 files changed, 150 insertions(+), 18 deletions(-)
--
2.49.0
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
