On 11/4/26 5:56 AM, Rosemarie O'Riorden via dev wrote:
> Hi Ruairi, thanks for the bug fix!
> 
> Acked-by: Rosemarie O'Riorden <[email protected]>
> 

Hi Ruairi, Rosemarie,

Thanks for the fix and review!

> On 11/2/26 9:37 AM, Ruairi Carroll via dev wrote:
>> A port can be added to the postponed set from two places:
>> claim_lport() and the OIF_MARK_UP/OIF_INSTALL_FLOWS handling in
>> if-status.c, both via lport_maybe_postpone().  In the if-status case,
>> once the claim threshold has passed, if-status sets pb->chassis itself
>> through local_binding_set_pb() but never removes the port from the
>> postponed set.
>>
>> From then on the postponed_ports engine node is non-empty forever, so
>> every engine run processes the port again through
>> runtime_data_postponed_ports_handler() -> handle_updated_port() ->
>> claim_lport().  Since pb->chassis is already this chassis,
>> claim_lport() takes the "already claimed" branch, which does not remove
>> the port from the set either.  The port is added to the tracked
>> bindings each time, so all port groups containing it are marked
>> updated and every logical flow referencing those port groups is
>> re-processed on every iteration.
>>
>> The sequence that leaves a port in the set is:
>>
>> 1. This chassis claims a port, and its interface is still in the
>>    OIF_INSTALL_FLOWS or OIF_MARK_UP state.
>> 2. Within the claim threshold (500ms), another chassis sets
>>    pb->chassis to itself.
>> 3. Both binding (claim_lport()) and if-status postpone claiming the
>>    port back, and it is added to the postponed set.
>> 4. Once the threshold has passed, if-status sets pb->chassis back to
>>    this chassis before binding handles the postponed claim.  Binding
>>    normally runs first, as engine_run() is called before
>>    if_status_mgr_update() in the main loop, but if the engine run is
>>    cancelled (e.g. a recompute is pending while recomputes are not
>>    allowed because of an OpenFlow backlog), if-status claims the port
>>    on its own.
>> 5. From then on claim_lport() only sees a port that is already bound
>>    to this chassis, so the port is never removed from the set.
>>
>> Step 2 typically happens when two chassis briefly claim the same port,
>> e.g. around a VM live migration, a VM re-created on another server
>> with the same port, or a stale interface with the same iface-id left
>> behind on another server.  The postponed state survives until
>> ovn-controller is restarted.
>>
>> On servers with large ACL port groups this kept ovn-controller at
>> 100% CPU indefinitely (observed for weeks, until restart), with each
>> iteration spending ~200ms re-processing ACL flows.  Combined with the
>> conjunction ordering issue fixed separately in ofctrl, every iteration
>> also removed and re-added thousands of unchanged OpenFlow flows.
>>
>> Fix it by removing the port from the postponed set in the "already
>> claimed by this chassis" branch of claim_lport(), as the claim is
>> complete at that point.
>>
>> The new test reproduces the stuck state deterministically: it makes
>> ovn-controller postpone a claim by binding the port to another chassis
>> in the SB DB, then binds it back to this chassis while ovn-controller
>> is paused, as if-status would, and checks that the postponed_ports node
>> no longer reports changes on subsequent engine runs.
>>
>> Fixes: f64818d43933 ("controller: throttle port claim attempts from 
>> if-status")
>> Assisted-by: Claude Opus 5.5, Claude Code
>> Signed-off-by: Ruairi Carroll <[email protected]>
>> ---
>>  controller/binding.c |  5 ++++
>>  tests/ovn.at         | 58 ++++++++++++++++++++++++++++++++++++++++++++
>>  2 files changed, 63 insertions(+)
>>
>> diff --git a/controller/binding.c b/controller/binding.c
>> index 3e8114e7ec..c14f9ee2f6 100644
>> --- a/controller/binding.c
>> +++ b/controller/binding.c
>> @@ -1473,6 +1473,11 @@ claim_lport(const struct sbrec_port_binding *pb,
>>              register_claim_timestamp(pb->logical_port, now);
>>              sset_find_and_delete(postponed_ports, pb->logical_port);
>>          } else {
>> +            /* The port may have been postponed by if-status, which then set
>> +             * pb->chassis itself without going through this function.  The
>> +             * claim is done, so the port must not stay postponed, otherwise
>> +             * it is re-processed on every engine run. */

Nit: This comment is a bit superfluous, especially since we don't
comment anything on the other branch of the "if" statement.  I'd just
remove it.

>> +            sset_find_and_delete(postponed_ports, pb->logical_port);
>>              update_tracked = true;
>>              if ((pb->n_up && !pb->up[0]) ||
>>                  (is_vif && !smap_get_bool(&iface_rec->external_ids,
>> diff --git a/tests/ovn.at b/tests/ovn.at
>> index 7801494674..8621766730 100644
>> --- a/tests/ovn.at
>> +++ b/tests/ovn.at
>> @@ -43462,6 +43462,64 @@ ignored_dp=ls0])
>>  AT_CLEANUP
>>  ])
>>  
>> +OVN_FOR_EACH_NORTHD([
>> +AT_SETUP([Port claimed outside of binding does not stay postponed])
>> +AT_KEYWORDS([claim postpone])
>> +ovn_start
>> +
>> +check ovn-nbctl ls-add ls0
>> +check ovn-nbctl lsp-add ls0 lsp0
>> +
>> +net_add n1
>> +sim_add hv1
>> +as hv1
>> +check ovs-vsctl add-br br-phys
>> +ovn_attach n1 br-phys 192.168.0.1
>> +
>> +# A chassis that is not running ovn-controller, so that nothing releases or
>> +# claims the ports bound to it.
>> +check ovn-sbctl chassis-add fake geneve 192.168.0.100
>> +hv1_uuid=$(fetch_column Chassis _uuid name=hv1)
>> +fake_uuid=$(fetch_column Chassis _uuid name=fake)
>> +
>> +as hv1 check ovn-appctl -t ovn-controller vlog/set binding:file:dbg
>> +as hv1 check ovs-vsctl add-port br-int vif -- \
>> +    set Interface vif external_ids:iface-id=lsp0
>> +wait_for_ports_up lsp0
>> +pb_uuid=$(fetch_column Port_Binding _uuid logical_port=lsp0)
>> +
>> +# Bind lsp0 to the other chassis shortly after hv1 claimed it, so that hv1
>> +# postpones claiming it back.  If hv1 was too late to postpone, it claims
>> +# lsp0 back right away, so just try again.
>> +OVS_WAIT_UNTIL([
>> +    check ovn-sbctl set Port_Binding $pb_uuid chassis=$fake_uuid
>> +    sleep 0.1
>> +    grep -q "Postponed claim on logical port lsp0" hv1/ovn-controller.log
>> +])
>> +
>> +# Bind lsp0 back to hv1 without going through binding, as if-status does
>> +# once the claim threshold has passed.  hv1 sees lsp0 bound to itself, so
>> +# the claim is complete.
>> +sleep_controller hv1
>> +check ovn-sbctl set Port_Binding $pb_uuid chassis=$hv1_uuid
>> +wake_up_controller hv1
>> +wait_column "$hv1_uuid" Port_Binding chassis logical_port=lsp0
>> +check ovn-nbctl --wait=hv sync
>> +
>> +# lsp0 must not stay postponed, otherwise it is processed again on every
>> +# engine run.
>> +as hv1 check ovn-appctl -t ovn-controller vlog/set inc_proc_eng:file:dbg
>> +check ovn-nbctl --wait=hv sync
>> +check ovn-nbctl --wait=hv sync
>> +as hv1 check ovn-appctl -t ovn-controller vlog/set inc_proc_eng:file:info
>> +AT_CHECK([grep -c "node: postponed_ports, old_state Stale, new_state 
>> Updated" \
>> +    hv1/ovn-controller.log], [1], [0
>> +])
>> +
>> +OVN_CLEANUP([hv1])
>> +AT_CLEANUP
>> +])
>> +
>>  dnl This test checks that the megaflows translated by ovs-vswitchd
>>  dnl don't match on IPv6 source and destination addresses for
>>  dnl simple switching.
> 

With the small nit addressed, I went ahead and applied this to main,
26.09 and 26.03.

I also added Ruairi to the AUTHORS list.

Regards,
Dumitru

_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Reply via email to