On 11/5/26 6:54 AM, Bekei Park wrote:
> ofctrl_put() runs ecmp_nexthop_monitor_run() before it checks
> ofctrl_can_put().  The monitor builds the NXT_CT_FLUSH message for a
> next-hop that went away and forgets that next-hop at once.  If ofctrl
> cannot send at that point, because the switch connection still has
> messages queued or is not in S_UPDATE_FLOWS, ofctrl_put() deletes the
> messages and returns.  The flows are sent on a later run, but the flush
> is never built again, so the conntrack entries that carry the removed
> next-hop MAC in ct_label stay until they expire.
> 
> We hit this while removing members of an ecmp-symmetric-reply route
> under load: the route was gone from the flows, but established
> connections kept the removed member in ct_label and their replies kept
> going to it.
> 
> Keep a removed next-hop in a pending set instead, and build its flush
> only once ofctrl_can_put() passed and while the switch connection is
> up (rconn_send() drops what is queued on a down connection).  A pending
> flush sets need_put, so it is still sent when no flow changes.  This
> also stops encoding a flush with rconn_get_version() == -1 before the
> first connection to the switch.
> 
> A flush only stays pending while the flows that remove the next-hop
> have not reached the switch either.  If the same next-hop comes back in
> the meantime, its conntrack entries are still valid, so the pending
> flush is dropped.
> 
> The new test stops ovs-vswitchd, fills the OpenFlow connection with a
> large flow update and removes a next-hop; without this change its flush
> never reaches the switch.  It then removes the next-hop and adds it
> back while ofctrl is still backlogged and checks that no flush is sent.
> 
> CC: Lorenzo Bianconi <[email protected]>
> CC: Dumitru Ceara <[email protected]>
> Fixes: b4cacd5c2142 ("ofctrl: Introduce ecmp_nexthop_monitor.")
> Assisted-by: Claude Opus 5.5, Claude Code
> Signed-off-by: Bekei Park <[email protected]>
> ---

Hi Bekei Park,

Thanks for the new revision!

> v2:
> - Drop a pending flush when the same next-hop comes back before the
>   flush was sent, and test that case (Jacob Tanenbaum).
> 
>  controller/ecmp-next-hop-monitor.c |  64 +++++++++++++---
>  controller/ecmp-next-hop-monitor.h |   9 ++-
>  controller/ofctrl.c                |  23 +++---
>  tests/ovn.at                       | 118 +++++++++++++++++++++++++++++
>  4 files changed, 190 insertions(+), 24 deletions(-)
> 
> diff --git a/controller/ecmp-next-hop-monitor.c 
> b/controller/ecmp-next-hop-monitor.c
> index 39ea46a..7f63e6e 100644
> --- a/controller/ecmp-next-hop-monitor.c
> +++ b/controller/ecmp-next-hop-monitor.c
> @@ -25,8 +25,16 @@
>  #include "ovn-sb-idl.h"
>  #include "controller/ecmp-next-hop-monitor.h"
>  
> +VLOG_DEFINE_THIS_MODULE(ecmp_nexthop_monitor);
> +
> +/* Next-hops of the SB ECMP_Nexthop table that are in use on local
> + * datapaths. */
>  static struct hmap ecmp_nexthop;
>  
> +/* Next-hops that went away but whose conntrack flush has not been sent to
> + * the switch yet.  They stay here until ofctrl is able to send it. */
> +static struct hmap ecmp_nexthop_pending_flush;
> +
>  struct ecmp_nexthop_data {
>      struct hmap_node hmap_node;
>      uint16_t zone_id;
> @@ -37,6 +45,7 @@ struct ecmp_nexthop_data {
>  void ecmp_nexthop_init(void)
>  {
>      hmap_init(&ecmp_nexthop);
> +    hmap_init(&ecmp_nexthop_pending_flush);
>  }
>  
>  static void
> @@ -60,6 +69,7 @@ ecmp_nexthop_destroy_map(struct hmap *map)
>  void ecmp_nexthop_destroy(void)
>  {
>      ecmp_nexthop_destroy_map(&ecmp_nexthop);
> +    ecmp_nexthop_destroy_map(&ecmp_nexthop_pending_flush);
>  }
>  
>  static uint32_t
> @@ -137,15 +147,16 @@ ecmp_nexthop_monitor_flush_ct_entry(const struct rconn 
> *swconn,
>      ovs_list_push_back(msgs, &msg->list_node);
>  }
>  
> -bool
> +/* Updates the monitored next-hops from 'enh_table'.  A monitored next-hop
> + * that is no longer there is moved to the pending flush set; its conntrack
> + * entries are flushed by ecmp_nexthop_monitor_flush(). */
> +void

We could avoid the kind of ugly ecmp_nexthop_monitor_has_pending_flush()
if we keep the return type for ecmp_nexthop_monitor_run() as bool.  That
is, true if a flush is pending, false otherwise.

We would need to change the function so that it correctly handles NULL
'local_datapaths' and make it end with:

return !hmap_is_empty(&ecmp_nexthop_pending_flush);

What do you think?

The rest of the patch looks OK to me.

Regards,
Dumitru

>  ecmp_nexthop_monitor_run(const struct sbrec_ecmp_nexthop_table *enh_table,
>                           const struct hmap *local_datapaths,
> -                         const struct shash *current_ct_zones,
> -                         const struct rconn *swconn, struct ovs_list *msgs)
> +                         const struct shash *current_ct_zones)
>  {
>      struct hmap sb_ecmp_nexthop = HMAP_INITIALIZER(&sb_ecmp_nexthop);
>      const struct sbrec_ecmp_nexthop *sbrec_ecmp_nexthop;
> -    bool ret = false;
>  
>      ovs_assert(local_datapaths);
>  
> @@ -178,6 +189,17 @@ ecmp_nexthop_monitor_run(const struct 
> sbrec_ecmp_nexthop_table *enh_table,
>                                       sbrec_ecmp_nexthop->mac,
>                                       ct_zone->zone, &ecmp_nexthop);
>          }
> +
> +        /* A next-hop that is back before its flush was sent never left the
> +         * flows on the switch, so its conntrack entries are still valid. */
> +        struct ecmp_nexthop_data *pending =
> +            ecmp_nexthop_find_entry(sbrec_ecmp_nexthop->nexthop,
> +                                    sbrec_ecmp_nexthop->mac, ct_zone->zone,
> +                                    &ecmp_nexthop_pending_flush);
> +        if (pending) {
> +            hmap_remove(&ecmp_nexthop_pending_flush, &pending->hmap_node);
> +            ecmp_nexthop_destroy_entry(pending);
> +        }
>          ecmp_nexthop_alloc_entry(sbrec_ecmp_nexthop->nexthop,
>                                   sbrec_ecmp_nexthop->mac, ct_zone->zone,
>                                   &sb_ecmp_nexthop);
> @@ -187,14 +209,38 @@ ecmp_nexthop_monitor_run(const struct 
> sbrec_ecmp_nexthop_table *enh_table,
>      HMAP_FOR_EACH_SAFE (e, hmap_node, &ecmp_nexthop) {
>          if (!ecmp_nexthop_find_entry(e->nexthop, e->mac, e->zone_id,
>                                       &sb_ecmp_nexthop)) {
> -            ecmp_nexthop_monitor_flush_ct_entry(swconn, e->mac,
> -                                                e->zone_id, msgs);
>              hmap_remove(&ecmp_nexthop, &e->hmap_node);
> -            ecmp_nexthop_destroy_entry(e);
> -            ret = true;
> +            if (ecmp_nexthop_find_entry(e->nexthop, e->mac, e->zone_id,
> +                                        &ecmp_nexthop_pending_flush)) {
> +                ecmp_nexthop_destroy_entry(e);
> +            } else {
> +                hmap_insert(&ecmp_nexthop_pending_flush, &e->hmap_node,
> +                            ecmp_nexthop_hash(e->nexthop, e->mac,
> +                                              e->zone_id));
> +            }
>          }
>      }
>      ecmp_nexthop_destroy_map(&sb_ecmp_nexthop);
> +}
> +
> +bool
> +ecmp_nexthop_monitor_has_pending_flush(void)
> +{
> +    return !hmap_is_empty(&ecmp_nexthop_pending_flush);
> +}
>  
> -    return ret;
> +/* Appends to 'msgs' a conntrack flush for each pending next-hop and forgets
> + * them.  The caller must make sure that 'msgs' are going to be sent to the
> + * switch. */
> +void
> +ecmp_nexthop_monitor_flush(const struct rconn *swconn, struct ovs_list *msgs)
> +{
> +    struct ecmp_nexthop_data *e;
> +    HMAP_FOR_EACH_POP (e, hmap_node, &ecmp_nexthop_pending_flush) {
> +        VLOG_DBG("Flushing CT for ECMP next-hop %s (MAC %s) in zone %"PRIu16,
> +                 e->nexthop, e->mac, e->zone_id);
> +        ecmp_nexthop_monitor_flush_ct_entry(swconn, e->mac, e->zone_id,
> +                                            msgs);
> +        ecmp_nexthop_destroy_entry(e);
> +    }
>  }
> diff --git a/controller/ecmp-next-hop-monitor.h 
> b/controller/ecmp-next-hop-monitor.h
> index 59a4bc7..ac514ce 100644
> --- a/controller/ecmp-next-hop-monitor.h
> +++ b/controller/ecmp-next-hop-monitor.h
> @@ -18,9 +18,10 @@
>  
>  void ecmp_nexthop_init(void);
>  void ecmp_nexthop_destroy(void);
> -bool ecmp_nexthop_monitor_run(const struct sbrec_ecmp_nexthop_table *,
> +void ecmp_nexthop_monitor_run(const struct sbrec_ecmp_nexthop_table *,
>                                const struct hmap *local_datapaths,
> -                              const struct shash *current_ct_zones,
> -                              const struct rconn *swconn,
> -                              struct ovs_list *msgs);
> +                              const struct shash *current_ct_zones);
> +bool ecmp_nexthop_monitor_has_pending_flush(void);
> +void ecmp_nexthop_monitor_flush(const struct rconn *swconn,
> +                                struct ovs_list *msgs);
>  #endif /* OVN_ECMP_NEXT_HOP_MONITOR_H */
> diff --git a/controller/ofctrl.c b/controller/ofctrl.c
> index c90b200..12da7f7 100644
> --- a/controller/ofctrl.c
> +++ b/controller/ofctrl.c
> @@ -2802,30 +2802,31 @@ ofctrl_put(struct ovn_desired_flow_table *lflow_table,
>          }
>      }
>  
> -    /* OpenFlow messages to send to the switch to bring it up-to-date. */
> -    struct ovs_list msgs = OVS_LIST_INITIALIZER(&msgs);
> -
>      if (local_datapaths) {
> -        need_put |= ecmp_nexthop_monitor_run(enh_table, local_datapaths,
> -                                             current_ct_zones, swconn, 
> &msgs);
> +        ecmp_nexthop_monitor_run(enh_table, local_datapaths,
> +                                 current_ct_zones);
>      }
> +    need_put |= ecmp_nexthop_monitor_has_pending_flush();
>  
>      if (!need_put) {
> -        ovs_assert(ovs_list_is_empty(&msgs));
>          VLOG_DBG("ofctrl_put not needed");
>          return;
>      }
>      if (!ofctrl_can_put()) {
>          VLOG_DBG("ofctrl_put can't be performed");
> -
> -        struct ofpbuf *msg;
> -        LIST_FOR_EACH_POP (msg, list_node, &msgs) {
> -            ofpbuf_delete(msg);
> -        }
>          skipped_last_time = true;
>          return;
>      }
>  
> +    /* OpenFlow messages to send to the switch to bring it up-to-date. */
> +    struct ovs_list msgs = OVS_LIST_INITIALIZER(&msgs);
> +
> +    /* Removed ECMP next-hops stay pending while the connection is down,
> +     * because rconn drops the messages queued then. */
> +    if (rconn_is_connected(swconn)) {
> +        ecmp_nexthop_monitor_flush(swconn, &msgs);
> +    }
> +
>      /* Iterate through ct zones that need to be flushed. */
>      struct shash_node *iter;
>      SHASH_FOR_EACH(iter, pending_ct_zones) {
> diff --git a/tests/ovn.at b/tests/ovn.at
> index 7801494..2ba8514 100644
> --- a/tests/ovn.at
> +++ b/tests/ovn.at
> @@ -48682,3 +48682,121 @@ AT_CHECK([ovs-ofctl dump-flows br-int 
> table=OFTABLE_PHY_TO_LOG \
>  OVN_CLEANUP([hv1])
>  AT_CLEANUP
>  ])
> +
> +OVN_FOR_EACH_NORTHD([
> +AT_SETUP([ECMP next-hop conntrack flush when ofctrl cannot send])
> +AT_KEYWORDS([ecmp ecmp_nexthop_monitor])
> +ovn_start
> +
> +net_add n1
> +sim_add hv1
> +as hv1
> +check ovs-vsctl add-br br-phys
> +ovn_attach n1 br-phys 192.168.0.1
> +check ovn-appctl -t ovn-controller vlog/set ofctrl:dbg
> +
> +check ovn-nbctl set nb_global . options:ecmp_nexthop_monitor_enable=true
> +check ovn-nbctl lr-add lr0 -- set logical_router lr0 options:chassis=hv1
> +check ovn-nbctl lrp-add lr0 lr0-public 00:00:20:20:12:13 192.168.10.1/24
> +check ovn-nbctl ls-add public
> +check ovn-nbctl lsp-add-router-port public public-lr0 lr0-public
> +check ovn-nbctl --ecmp-symmetric-reply lr-route-add lr0 1.0.0.0/24 
> 192.168.10.10
> +check ovn-nbctl --ecmp-symmetric-reply lr-route-add lr0 1.0.0.0/24 
> 192.168.10.20
> +check ovn-nbctl --wait=sb sync
> +
> +dp=$(fetch_column Datapath_Binding _uuid external_ids:name=lr0)
> +check_uuid ovn-sbctl create MAC_Binding datapath=$dp logical_port=lr0-public 
> \
> +    ip=192.168.10.10 mac='"f0:00:00:00:10:10"'
> +check_uuid ovn-sbctl create MAC_Binding datapath=$dp logical_port=lr0-public 
> \
> +    ip=192.168.10.20 mac='"f0:00:00:00:20:20"'
> +wait_row_count ECMP_Nexthop 1 nexthop=192.168.10.10 mac='"f0:00:00:00:10:10"'
> +wait_row_count ECMP_Nexthop 1 nexthop=192.168.10.20 mac='"f0:00:00:00:20:20"'
> +check ovn-nbctl --wait=hv sync
> +
> +zone=$(ovn-appctl -t ovn-controller ct-zone-list | grep "^lr0_dnat " | cut 
> -d ' ' -f 2)
> +check test -n "$zone"
> +flush_10="NXT_CT_FLUSH .* zone=$zone labels=0xf0000000101000000000/"
> +flush_20="NXT_CT_FLUSH .* zone=$zone labels=0xf0000000202000000000/"
> +
> +AS_BOX([Removed next-hop is flushed])
> +n=$(wc -l < hv1/ovs-vswitchd.log)
> +check ovn-nbctl --wait=hv lr-route-del lr0 1.0.0.0/24 192.168.10.20
> +tail -n +$((n + 1)) hv1/ovs-vswitchd.log > vswitchd-del.log
> +AT_CHECK([grep -c "$flush_20" vswitchd-del.log], [0], [1
> +])
> +AT_CHECK([grep -c "$flush_10" vswitchd-del.log], [1], [0
> +])
> +
> +AS_BOX([Removed next-hop is flushed when ofctrl could not send at removal])
> +check ovn-nbctl --ecmp-symmetric-reply lr-route-add lr0 1.0.0.0/24 
> 192.168.10.20
> +wait_row_count ECMP_Nexthop 1 nexthop=192.168.10.20 mac='"f0:00:00:00:20:20"'
> +check ovn-nbctl --wait=hv sync
> +
> +# Stop the switch and queue more flow updates than the connection can
> +# buffer, so that ofctrl has a backlog.
> +sleep_ovs hv1
> +addrs=$(for i in $(seq 0 19); do for j in $(seq 1 250); do
> +            printf '"10.0.%d.%d",' $i $j
> +        done; done)
> +stats=$(ovn-appctl -t ovn-controller inc-engine/show-stats lflow_output)
> +check_uuid ovn-nbctl --wait=sb create Address_Set name=big \
> +    addresses="${addrs%,}" -- acl-add public to-lport 1001 'ip4.src == $big' 
> allow
> +OVS_WAIT_UNTIL([test "$stats" != "$(ovn-appctl -t ovn-controller 
> inc-engine/show-stats lflow_output)"])
> +
> +# Remove the next-hop while ovn-controller sleeps, so that the removal is
> +# processed in one run while the backlog is still there.
> +sleep_controller hv1
> +check ovn-nbctl --wait=sb lr-route-del lr0 1.0.0.0/24 192.168.10.20
> +check_row_count ECMP_Nexthop 0 nexthop=192.168.10.20
> +n_skip=$(grep -c "ofctrl_put can't be performed" hv1/ovn-controller.log)
> +stats=$(ovn-appctl -t ovn-controller inc-engine/show-stats lflow_output)
> +wake_up_controller hv1
> +OVS_WAIT_UNTIL([test "$stats" != "$(ovn-appctl -t ovn-controller 
> inc-engine/show-stats lflow_output)"])
> +OVS_WAIT_UNTIL([test $(grep -c "ofctrl_put can't be performed" 
> hv1/ovn-controller.log) -gt $n_skip])
> +
> +wake_up_ovs hv1
> +check ovn-nbctl --wait=hv sync
> +AT_CHECK([grep -c "$flush_20" hv1/ovs-vswitchd.log], [0], [2
> +])
> +AT_CHECK([grep -c "$flush_10" hv1/ovs-vswitchd.log], [1], [0
> +])
> +
> +AS_BOX([Next-hop that is back before ofctrl can send is not flushed])
> +check ovn-nbctl --ecmp-symmetric-reply lr-route-add lr0 1.0.0.0/24 
> 192.168.10.20
> +wait_row_count ECMP_Nexthop 1 nexthop=192.168.10.20 mac='"f0:00:00:00:20:20"'
> +check ovn-nbctl --wait=hv sync
> +
> +# Stop the switch again and queue the removal of the big ACL.
> +sleep_ovs hv1
> +stats=$(ovn-appctl -t ovn-controller inc-engine/show-stats lflow_output)
> +check ovn-nbctl --wait=sb acl-del public -- destroy Address_Set big
> +OVS_WAIT_UNTIL([test "$stats" != "$(ovn-appctl -t ovn-controller 
> inc-engine/show-stats lflow_output)"])
> +
> +# Remove the next-hop and add it back, each in its own run, while ofctrl
> +# still cannot send.
> +sleep_controller hv1
> +check ovn-nbctl --wait=sb lr-route-del lr0 1.0.0.0/24 192.168.10.20
> +check_row_count ECMP_Nexthop 0 nexthop=192.168.10.20
> +n_skip=$(grep -c "ofctrl_put can't be performed" hv1/ovn-controller.log)
> +stats=$(ovn-appctl -t ovn-controller inc-engine/show-stats lflow_output)
> +wake_up_controller hv1
> +OVS_WAIT_UNTIL([test "$stats" != "$(ovn-appctl -t ovn-controller 
> inc-engine/show-stats lflow_output)"])
> +OVS_WAIT_UNTIL([test $(grep -c "ofctrl_put can't be performed" 
> hv1/ovn-controller.log) -gt $n_skip])
> +
> +sleep_controller hv1
> +check ovn-nbctl --wait=sb --ecmp-symmetric-reply lr-route-add lr0 1.0.0.0/24 
> 192.168.10.20
> +wait_row_count ECMP_Nexthop 1 nexthop=192.168.10.20 mac='"f0:00:00:00:20:20"'
> +n_skip=$(grep -c "ofctrl_put can't be performed" hv1/ovn-controller.log)
> +stats=$(ovn-appctl -t ovn-controller inc-engine/show-stats lflow_output)
> +wake_up_controller hv1
> +OVS_WAIT_UNTIL([test "$stats" != "$(ovn-appctl -t ovn-controller 
> inc-engine/show-stats lflow_output)"])
> +OVS_WAIT_UNTIL([test $(grep -c "ofctrl_put can't be performed" 
> hv1/ovn-controller.log) -gt $n_skip])
> +
> +wake_up_ovs hv1
> +check ovn-nbctl --wait=hv sync
> +AT_CHECK([grep -c "$flush_20" hv1/ovs-vswitchd.log], [0], [2
> +])
> +
> +OVN_CLEANUP([hv1])
> +AT_CLEANUP
> +])

_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Reply via email to