Hi Dumitru and Rosemarie, Thanks for the review and applying the patch!
On the hash check in ecmp_hash_count_get(): two different values can indeed share a bucket, but hmap_next_with_hash__() compares the full node->hash, not only the bucket, so nodes with different values are skipped. In this helper the value we insert is the key itself, so any node HMAP_FOR_EACH_WITH_HASH returns already matches, and the check is redundant, as Rosemarie said. The same goes for ecmp_id_slot_get(). It doesn't change the behavior, though, so I'm fine keeping it. I can send a small follow-up to drop it if you prefer. I'll send a backport shortly. Regards, Lucas Em seg., 9 de nov. de 2026 às 10:44, Dumitru Ceara <[email protected]> escreveu: > On 10/26/26 4:54 PM, Rosemarie O'Riorden via dev wrote: > > Hi Lucas, thanks for the new verison! I have two tiny nits below that > > could be addressed upon merge. So if I were you I'd wait for other > > comments before sending another version, as it might not be needed. > > > > Hi Lucas, Rosemarie, > > Thanks for the fix and reviews! > > > Otherwise the new addition to the algorithm looks good to me. > > > > As I said previously, other reviewers may have comments since I've been > > the main reviewer over all the versions, but: > > > > Acked-by: Rosemarie O'Riorden <[email protected]> > > > > I took care of the nits and applied the patch to main and 26.09. > The patch didn't apply cleanly to 26.03 but if you need it there > please send a 26.03-specific backport patch. > > Please also see below a reply to your other hash related comment. > > Regards, > Dumitru > > > On 10/22/26 3:52 PM, Lucas Vargas Dias wrote: > >> The ECMP group id and the member ids are written into the > >> lr_in_ip_routing and lr_in_ip_routing_ecmp logical flows, but they > >> were assigned from the order in which routes reached > >> en_group_ecmp_route: the group id was hmap_count() at creation time > >> and the member id was the insertion position. On a full recompute > >> the routes are walked in hmap order, while incremental processing > >> appends them in event order, so the same set of routes could get a > >> different numbering. Every forced or fallback recompute could then > >> delete and re-insert Logical_Flow rows in the Southbound DB even > >> though nothing had changed. > >> > >> Derive the ids from hashes instead. The group id comes from the > >> group key, i.e. prefix, prefix length, route table, source or > >> destination route and source class, which does not change as members > >> are added or removed. The member id comes from the UUID of the > >> route's source row, its nexthop and its output port: a NAT or LB row > >> produces one route per path to the peer router, so the UUID alone is > >> not unique within a group. > >> > >> When two elements derive the same id, all of them retry in the next > >> round with an id derived from their hash and the round number, so > >> that the ids only depend on the set of routes and not on the order > >> in which they were processed. Elements with equal hashes can't be > >> told apart, so they skip the retries and take the next free id. The > >> members are listed in the select() action in id order for the same > >> reason. > >> > >> Besides making a recompute produce the same flows as incremental > >> processing, adding or removing a route no longer changes the ids, > >> and so the flows, of unrelated groups or members. > >> > >> Fixes: 6979a96138a2 ("northd: Support I+P for group_ecmp_route engine.") > >> Fixes: 5395e12f247c ("northd: Drop traffic for ECMP group with > "discard" route.") > >> Suggested-by: Rosemarie O'Riorden <[email protected]> > >> Assisted-by: Claude Opus 5.5, Claude Code > >> Signed-off-by: Lucas Vargas Dias <[email protected]> > >> --- > >> v6: > >> - Elements with equal hashes skip the retry rounds in > >> ecmp_assign_ids() and go straight to the fallback. > >> - Renamed 'n' to 'n_elements' and 'n_left' to 'n_unassigned' in > >> ecmp_assign_ids(), and 'n' to 'n_members' and 'n_groups' in its > >> callers. > >> v5: > >> - Implement route_sources_ecmp_compatible() on top of > >> ecmp_group_source_class() so that the NAT/LB rule lives in one > place. > >> - Hash the whole source row UUID with hash_bytes() in > >> ecmp_member_hash(). > >> - Derive each id once per round in ecmp_assign_ids(). > >> - Cross-reference ecmp_group_hash() and ecmp_groups_find(). > >> v4: > >> - Derive the ECMP group and member ids from hashes instead of sorting, > >> so that adding or removing a route doesn't change the ids of > >> unrelated groups or members (suggested by Rosemarie). > >> - Added a test with NAT and LB routes reachable over several paths, > >> whose ECMP members share their source row. > >> - Added ovn_strip_ecmp_ids(), ecmp_ids() and ecmp_member_id() for the > >> tests. > >> v3: > >> - Dropped nullable_strcmp() and the patch converting its callers. > >> - Removed the unused 'route_count' field from struct ecmp_groups_node. > >> v2: > >> - Moved nullable_strcmp() to lib/ovn-util.h and documented its NULL > >> handling. > >> - Clarified in ecmp_groups_node_cmp() that it can't return 0 for two > >> distinct groups. > >> > >> northd/en-group-ecmp-route.c | 326 +++++++++++++++++++++++++++++++---- > >> northd/en-group-ecmp-route.h | 1 - > >> tests/ovn-macros.at | 34 ++++ > >> tests/ovn-northd.at | 300 ++++++++++++++++++++++++-------- > >> 4 files changed, 558 insertions(+), 103 deletions(-) > >> > >> diff --git a/northd/en-group-ecmp-route.c b/northd/en-group-ecmp-route.c > >> index aca197318..4785100f4 100644 > >> --- a/northd/en-group-ecmp-route.c > >> +++ b/northd/en-group-ecmp-route.c > >> @@ -24,6 +24,7 @@ > >> #include "en-advertised-route-sync.h" > >> #include "en-group-ecmp-route.h" > >> #include "en-learned-route-sync.h" > >> +#include "hash.h" > >> #include "hmapx.h" > >> #include "openvswitch/hmap.h" > >> > >> @@ -182,22 +183,26 @@ unique_routes_destroy(struct hmap *unique_routes) > >> hmap_destroy(unique_routes); > >> } > >> > >> -/* Allow LB and NAT forwarding routes for the same prefix to coexist as > >> +/* Routes for the same prefix share an ecmp group if their sources map > to the > >> + * same class. > >> + * > >> + * LB and NAT forwarding routes are in the same class so that they > coexist as > >> * ECMP members instead of one silently replacing the other. This is > an > >> * edge case: it arises only when two different peer LRs independently > >> * use the same IP as a VIP and a NAT external IP, and a CMS would > >> * typically prevent such overlap. Without this guard, one route > >> * would silently win and traffic to the other service would be > >> * black-holed. */ > >> +static enum route_source > >> +ecmp_group_source_class(enum route_source source) > >> +{ > >> + return source == ROUTE_SOURCE_LB ? ROUTE_SOURCE_NAT : source; > >> +} > >> + > >> static bool > >> route_sources_ecmp_compatible(enum route_source a, enum route_source b) > >> { > >> - if (a == b) { > >> - return true; > >> - } > >> - > >> - return (a == ROUTE_SOURCE_NAT || a == ROUTE_SOURCE_LB) && > >> - (b == ROUTE_SOURCE_NAT || b == ROUTE_SOURCE_LB); > >> + return ecmp_group_source_class(a) == ecmp_group_source_class(b); > >> } > >> > >> /* Remove the unique_routes_node from the group, and return the > parsed_route > >> @@ -228,14 +233,12 @@ ecmp_groups_add_route(struct ecmp_groups_node > *group, > >> const struct parsed_route *route) > >> { > >> static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); > >> - if (group->route_count == UINT16_MAX) { > >> + if (vector_len(&group->route_list) == UINT16_MAX) { > >> VLOG_WARN_RL(&rl, "too many routes in a single ecmp group."); > >> return; > >> } > >> > >> if (route->is_discard_route) { > >> - group->has_discard_route = true; > >> - > >> char *prefix = normalize_v46_prefix(&route->prefix, > route->plen); > >> VLOG_WARN_RL(&rl, "The ECMP route \"%s\" contains \"discard\" " > >> "route, the whole group will drop traffic.", > prefix); > >> @@ -244,22 +247,13 @@ ecmp_groups_add_route(struct ecmp_groups_node > *group, > >> > >> struct ecmp_route_list_node er = (struct ecmp_route_list_node) { > >> .route = route, > >> - .id = ++group->route_count, > >> }; > >> > >> - if (group->route_count == 1) { > >> - sset_clone(&group->selection_fields, > &route->ecmp_selection_fields); > >> - } else { > >> - sset_intersect(&group->selection_fields, > >> - &route->ecmp_selection_fields); > >> - } > >> - > >> vector_push(&group->route_list, &er); > >> } > >> > >> -/* Removes a route from an ecmp group. If the ecmp group should persist > >> - * afterwards you must call ecmp_groups_update_ids before any further > >> - * insertions. */ > >> +/* Removes a route from an ecmp group. The ids of the group and of its > >> + * members are refreshed by group_ecmp_datapath_finalize(). */ > >> static const struct parsed_route * > >> ecmp_groups_remove_route(struct ecmp_groups_node *group, > >> const struct parsed_route *pr) > >> @@ -278,15 +272,281 @@ ecmp_groups_remove_route(struct ecmp_groups_node > *group, > >> return NULL; > >> } > >> > >> +/* Hashes the key of 'eg', which is unique within a datapath. It must > cover > >> + * the fields that ecmp_groups_find() compares. Unlike the members, > the key > >> + * does not change when routes are added to or removed from the group. > */ > >> +static uint32_t > >> +ecmp_group_hash(const struct ecmp_groups_node *eg) > >> +{ > >> + uint32_t hash = hash_bytes(&eg->prefix, sizeof eg->prefix, 0); > >> + hash = hash_add(hash, eg->plen); > >> + hash = hash_add(hash, eg->route_table_id); > >> + hash = hash_add(hash, eg->is_src_route); > >> + hash = hash_add(hash, ecmp_group_source_class(eg->source)); > >> + return hash_finish(hash, 0); > >> +} > >> + > >> +/* Hashes what makes 'route' unique within its ecmp group. A NAT or > LB row > >> + * produces one route per path to the peer router, so its UUID alone > is not > >> + * enough. */ > >> +static uint32_t > >> +ecmp_member_hash(const struct parsed_route *route) > >> +{ > >> + const struct uuid *uuid = &route->source_hint->uuid; > >> + uint32_t hash = hash_bytes(uuid, sizeof *uuid, 0); > >> + if (route->nexthop) { > >> + hash = hash_bytes(route->nexthop, sizeof *route->nexthop, > hash); > >> + } > >> + if (route->out_port) { > >> + hash = hash_string(route->out_port->key, hash); > >> + } > >> + return hash; > >> +} > >> + > >> +/* Number of rounds after which ecmp_assign_ids() stops retrying > hashes that > >> + * collide and falls back to the next free id. */ > >> +#define ECMP_ID_MAX_ROUNDS 32 > >> + > >> +struct ecmp_id_slot { > >> + struct hmap_node hmap_node; > >> + uint16_t id; > >> + bool taken; /* Assigned to an element. */ > >> + size_t n_claims; /* Elements that derived 'id' in the current > round. */ > >> +}; > >> + > >> +static struct ecmp_id_slot * > >> +ecmp_id_slot_get(struct hmap *slots, uint16_t id) > >> +{ > >> + struct ecmp_id_slot *slot; > >> + HMAP_FOR_EACH_WITH_HASH (slot, hmap_node, id, slots) { > >> + if (slot->id == id) { > >> + return slot; > >> + } > >> + } > >> + slot = xzalloc(sizeof *slot); > >> + slot->id = id; > >> + hmap_insert(slots, &slot->hmap_node, id); > >> + return slot; > >> +} > >> + > >> +struct ecmp_hash_count { > >> + struct hmap_node hmap_node; > >> + uint32_t hash; > >> + size_t n; > >> +}; > >> + > >> +static struct ecmp_hash_count * > >> +ecmp_hash_count_get(struct hmap *counts, uint32_t hash) > >> +{ > >> + struct ecmp_hash_count *count; > >> + HMAP_FOR_EACH_WITH_HASH (count, hmap_node, hash, counts) { > >> + if (count->hash == hash) { > > > > Btw, this if-condition will always be true. HMAP_FOR_EACH_WITH_HASH > > already checks for the hash match. So this is doing that comparison > > twice. So I'd recommend removing the if-condition and just returning > > count when a node is found. > > I don't think you're right, Rosemarie. It's a bit confusing that > we're using the hash value as hashmap "hash" too. But you could have > two ecmp hash values that "hash" to the same bucket. > > See: > > /* Returns the first node in 'hmap' with the given 'hash', or a null > pointer if > * no nodes have that hash value. */ > static inline struct hmap_node * > hmap_first_with_hash(const struct hmap *hmap, size_t hash) > { > return hmap_next_with_hash__(hmap->buckets[hash & hmap->mask], hash); > } > > So I actually think Lucas' version is the correct one. > > > > > For context: > > > > static inline struct hmap_node * > > hmap_next_with_hash__(const struct hmap_node *node, size_t hash) > > { > > while (node != NULL && node->hash != hash) { > > node = node->next; > > } > > return CONST_CAST(struct hmap_node *, node); > > } > > > >> + return count; > >> + } > >> + } > >> + count = xzalloc(sizeof *count); > >> + count->hash = hash; > >> + hmap_insert(counts, &count->hmap_node, hash); > >> + return count; > >> +} > >> + > >> +/* Sets 'same_hash[i]' for each of the 'n_elements' elements that has > no id > >> + * yet and shares its hash with another such element. */ > >> static void > >> -ecmp_group_update_ids(struct ecmp_groups_node *group) > >> +ecmp_find_same_hash(const uint32_t *hashes, const uint16_t *ids, > >> + bool *same_hash, size_t n_elements) > >> { > >> + struct hmap counts = HMAP_INITIALIZER(&counts); > >> + struct ecmp_hash_count *count; > >> + > >> + for (size_t i = 0; i < n_elements; i++) { > >> + if (!ids[i]) { > >> + ecmp_hash_count_get(&counts, hashes[i])->n++; > >> + } > >> + } > >> + for (size_t i = 0; i < n_elements; i++) { > >> + if (!ids[i]) { > >> + same_hash[i] = ecmp_hash_count_get(&counts, hashes[i])->n > > 1; > >> + } > >> + } > >> + HMAP_FOR_EACH_POP (count, hmap_node, &counts) { > >> + free(count); > >> + } > >> + hmap_destroy(&counts); > >> +} > >> + > >> +static uint16_t > >> +ecmp_id_from_hash(uint32_t hash, unsigned int round) > > > > Same unsigned int nit applies here. (see below) > > > >> +{ > >> + return hash_int(round, hash) % UINT16_MAX + 1; > >> +} > >> + > >> +/* Gives each of the 'n_elements' elements a distinct id in [1, > UINT16_MAX] > >> + * derived from 'hashes', so that the id of an element does not depend > on the > >> + * order in which the elements are passed, nor on the ids of unrelated > >> + * elements. > >> + * > >> + * An element keeps the id derived from its hash unless that id is > already > >> + * taken or another element derives the same id in the same round. In > that > >> + * case all of them retry in the next round, with an id derived from > their > >> + * hash and the round number. The result therefore only depends on > the set > >> + * of hashes. > >> + * > >> + * Elements whose hashes are equal derive the same id in every round, > so they > >> + * can't be told apart. They don't retry and, like elements that still > >> + * collide after ECMP_ID_MAX_ROUNDS rounds, take the next free id in > the > >> + * order they are passed. */ > >> +static void > >> +ecmp_assign_ids(const uint32_t *hashes, uint16_t *ids, size_t > n_elements) > >> +{ > >> + struct hmap slots = HMAP_INITIALIZER(&slots); > >> + /* The slot that each element claims in the current round. */ > >> + struct ecmp_id_slot **claims = xmalloc(n_elements * sizeof > *claims); > >> + /* Whether another element has the same hash. */ > >> + bool *same_hash = xzalloc(n_elements * sizeof *same_hash); > >> + struct ecmp_id_slot *slot; > >> + size_t n_unassigned = n_elements; > >> + size_t n_retrying = n_elements; > >> + > >> + memset(ids, 0, n_elements * sizeof *ids); > >> + for (unsigned int round = 0; n_retrying && round < > ECMP_ID_MAX_ROUNDS; > > > > Nit: Any particular reason why round is an unsigned int instead of > > size_t? size_t is used for all the other loops as usual. > > > >> + round++) { > >> + for (size_t i = 0; i < n_elements; i++) { > >> + if (!ids[i] && !same_hash[i]) { > >> + uint16_t id = ecmp_id_from_hash(hashes[i], round); > >> + claims[i] = ecmp_id_slot_get(&slots, id); > >> + claims[i]->n_claims++; > >> + } > >> + } > >> + for (size_t i = 0; i < n_elements; i++) { > >> + if (!ids[i] && !same_hash[i]) { > >> + slot = claims[i]; > >> + if (!slot->taken && slot->n_claims == 1) { > >> + slot->taken = true; > >> + ids[i] = slot->id; > >> + n_unassigned--; > >> + n_retrying--; > >> + } > >> + } > >> + } > >> + HMAP_FOR_EACH (slot, hmap_node, &slots) { > >> + slot->n_claims = 0; > >> + } > >> + > >> + if (!round && n_unassigned) { > >> + /* Elements with the same hash always claim the same id, > so none > >> + * of them got one in this round. */ > >> + ecmp_find_same_hash(hashes, ids, same_hash, n_elements); > >> + for (size_t i = 0; i < n_elements; i++) { > >> + n_retrying -= same_hash[i]; > >> + } > >> + } > >> + } > >> + free(claims); > >> + free(same_hash); > >> + > >> + for (size_t i = 0; n_unassigned && i < n_elements; i++) { > >> + if (ids[i]) { > >> + continue; > >> + } > >> + uint16_t id = ecmp_id_from_hash(hashes[i], 0); > >> + while ((slot = ecmp_id_slot_get(&slots, id))->taken) { > >> + id = id % UINT16_MAX + 1; > >> + } > >> + slot->taken = true; > >> + ids[i] = id; > >> + n_unassigned--; > >> + } > >> + > >> + HMAP_FOR_EACH_POP (slot, hmap_node, &slots) { > >> + free(slot); > >> + } > >> + hmap_destroy(&slots); > >> +} > >> + > >> +static int > >> +ecmp_route_list_node_id_cmp(const void *a_, const void *b_) > >> +{ > >> + const struct ecmp_route_list_node *a = a_; > >> + const struct ecmp_route_list_node *b = b_; > >> + > >> + return a->id < b->id ? -1 : a->id > b->id; > >> +} > >> + > >> +/* Recomputes everything in 'group' that is derived from its members, > so that > >> + * the result only depends on the set of members. */ > >> +static void > >> +ecmp_group_finalize(struct ecmp_groups_node *group) > >> +{ > >> + size_t n_members = vector_len(&group->route_list); > >> + uint32_t *hashes = xmalloc(n_members * sizeof *hashes); > >> + uint16_t *ids = xmalloc(n_members * sizeof *ids); > >> struct ecmp_route_list_node *er; > >> size_t i = 0; > >> + > >> VECTOR_FOR_EACH_PTR (&group->route_list, er) { > >> - er->id = i++; > >> + hashes[i++] = ecmp_member_hash(er->route); > >> + } > >> + ecmp_assign_ids(hashes, ids, n_members); > >> + i = 0; > >> + VECTOR_FOR_EACH_PTR (&group->route_list, er) { > >> + er->id = ids[i++]; > >> + } > >> + free(hashes); > >> + free(ids); > >> + > >> + /* The members are listed in this order in the select() action. */ > >> + vector_qsort(&group->route_list, ecmp_route_list_node_id_cmp); > >> + > >> + bool first = true; > >> + group->has_discard_route = false; > >> + VECTOR_FOR_EACH_PTR (&group->route_list, er) { > >> + if (er->route->is_discard_route) { > >> + group->has_discard_route = true; > >> + } > >> + if (first) { > >> + sset_destroy(&group->selection_fields); > >> + sset_clone(&group->selection_fields, > >> + &er->route->ecmp_selection_fields); > >> + group->source = er->route->source; > >> + first = false; > >> + } else { > >> + sset_intersect(&group->selection_fields, > >> + &er->route->ecmp_selection_fields); > >> + } > >> } > >> - group->route_count = i; > >> +} > >> + > >> +/* Assigns the group and member ids of all ecmp groups of 'gn'. Must > be > >> + * called after the ecmp groups of 'gn' changed, both on full > recompute and on > >> + * incremental processing, so that both yield the same ids for the > same set > >> + * of routes and the generated logical flows do not change on > recompute. */ > >> +static void > >> +group_ecmp_datapath_finalize(struct group_ecmp_datapath *gn) > >> +{ > >> + size_t n_groups = hmap_count(&gn->ecmp_groups); > >> + if (!n_groups) { > >> + return; > >> + } > >> + > >> + struct ecmp_groups_node **groups = xmalloc(n_groups * sizeof > *groups); > >> + uint32_t *hashes = xmalloc(n_groups * sizeof *hashes); > >> + uint16_t *ids = xmalloc(n_groups * sizeof *ids); > >> + struct ecmp_groups_node *eg; > >> + size_t i = 0; > >> + > >> + HMAP_FOR_EACH (eg, hmap_node, &gn->ecmp_groups) { > >> + ecmp_group_finalize(eg); > >> + groups[i] = eg; > >> + hashes[i++] = ecmp_group_hash(eg); > >> + } > >> + ecmp_assign_ids(hashes, ids, n_groups); > >> + for (i = 0; i < n_groups; i++) { > >> + groups[i]->id = ids[i]; > >> + } > >> + free(groups); > >> + free(hashes); > >> + free(ids); > >> } > >> > >> static struct ecmp_groups_node * > >> @@ -302,7 +562,6 @@ ecmp_groups_add(struct group_ecmp_datapath *gn, > >> struct ecmp_groups_node *eg = xzalloc(sizeof *eg); > >> hmap_insert(&gn->ecmp_groups, &eg->hmap_node, route->hash); > >> > >> - eg->id = hmap_count(&gn->ecmp_groups); > >> eg->prefix = route->prefix; > >> eg->plen = route->plen; > >> eg->is_src_route = route->is_src_route; > >> @@ -321,6 +580,7 @@ ecmp_groups_find(struct group_ecmp_datapath *gn, > >> { > >> struct ecmp_groups_node *eg; > >> HMAP_FOR_EACH_WITH_HASH (eg, hmap_node, route->hash, > &gn->ecmp_groups) { > >> + /* Keep in sync with ecmp_group_hash(). */ > >> if (ipv6_addr_equals(&eg->prefix, &route->prefix) && > >> eg->plen == route->plen && > >> eg->is_src_route == route->is_src_route && > >> @@ -396,6 +656,10 @@ group_ecmp_route(struct group_ecmp_route_data > *data, > >> gn = group_ecmp_datapath_lookup_or_add(data, pr->od); > >> add_route(gn, pr); > >> } > >> + > >> + HMAP_FOR_EACH (gn, hmap_node, &data->datapaths) { > >> + group_ecmp_datapath_finalize(gn); > >> + } > >> } > >> > >> enum engine_node_state > >> @@ -472,9 +736,7 @@ handle_deleted_route(struct group_ecmp_route_data > *data, > >> * unique route. Otherwise it stays an ecmp group with > just one > >> * member. */ > >> ecmp_groups_remove_route(eg, pr); > >> - if (ecmp_group_has_symmetric_reply(eg)) { > >> - ecmp_group_update_ids(eg); > >> - } else { > >> + if (!ecmp_group_has_symmetric_reply(eg)) { > >> const struct ecmp_route_list_node *er = > >> vector_get_ptr(&eg->route_list, 0); > >> unique_routes_add(node, er->route); > >> @@ -482,11 +744,8 @@ handle_deleted_route(struct group_ecmp_route_data > *data, > >> ecmp_groups_node_free(eg); > >> } > >> } else { > >> - /* We can just remove the member from the group. We need > to update > >> - * the indices of all routes so that future insertions > directly > >> - * have a new index. */ > >> + /* We can just remove the member from the group. */ > >> ecmp_groups_remove_route(eg, pr); > >> - ecmp_group_update_ids(eg); > >> } > >> } > >> > >> @@ -537,6 +796,7 @@ > group_ecmp_route_learned_route_change_handler(struct engine_node *eng_node, > >> hmapx_add(&data->trk_data.deleted_datapath_routes, node); > >> hmap_remove(&data->datapaths, &node->hmap_node); > >> } else { > >> + group_ecmp_datapath_finalize(node); > >> hmapx_add(&data->trk_data.crupdated_datapath_routes, node); > >> } > >> } > >> @@ -589,6 +849,7 @@ group_ecmp_route_routes_change_handler(struct > engine_node *eng_node, > >> hmapx_add(&data->trk_data.deleted_datapath_routes, node); > >> hmap_remove(&data->datapaths, &node->hmap_node); > >> } else { > >> + group_ecmp_datapath_finalize(node); > >> hmapx_add(&data->trk_data.crupdated_datapath_routes, node); > >> } > >> } > >> @@ -645,6 +906,7 @@ > group_ecmp_route_dynamic_routes_change_handler(struct engine_node *eng_node, > >> hmapx_add(&gdata->trk_data.deleted_datapath_routes, node); > >> hmap_remove(&gdata->datapaths, &node->hmap_node); > >> } else { > >> + group_ecmp_datapath_finalize(node); > >> hmapx_add(&gdata->trk_data.crupdated_datapath_routes, > node); > >> } > >> } > >> diff --git a/northd/en-group-ecmp-route.h b/northd/en-group-ecmp-route.h > >> index b40935be2..13f84a036 100644 > >> --- a/northd/en-group-ecmp-route.h > >> +++ b/northd/en-group-ecmp-route.h > >> @@ -37,7 +37,6 @@ struct ecmp_groups_node { > >> bool has_discard_route; > >> enum route_source source; > >> uint32_t route_table_id; > >> - uint16_t route_count; > >> struct vector route_list; /* Contains ecmp_route_list_node */ > >> struct sset selection_fields; > >> }; > >> diff --git a/tests/ovn-macros.at b/tests/ovn-macros.at > >> index e0f70bcde..811b285d5 100644 > >> --- a/tests/ovn-macros.at > >> +++ b/tests/ovn-macros.at > >> @@ -1546,6 +1546,40 @@ ovn_strip_lflows() { > >> sed 's/table=[[0-9]]\{1,2\}\s\?/table=??/g' | sort > >> } > >> > >> +# The ECMP group and member ids are derived from hashes of the routes, > replace > >> +# them with <gid> and <mid>. > >> +ovn_strip_ecmp_ids() { > >> + sed -e 's/\(reg8\[[0\.\.15\]] =\{1,2\} \)[[1-9]][[0-9]]*/\1<gid>/' > \ > >> + -e 's/\(reg8\[[16\.\.31\]] =\{1,2\} > \)[[1-9]][[0-9]]*/\1<mid>/' \ > >> + -e ':a' \ > >> + -e 's/\(select[[^)]]*[[(, ]]\)[[0-9]][[0-9]]*/\1<mid>/' \ > >> + -e 'ta' > >> +} > >> + > >> +# ecmp_ids LR PREFIX: prints the group id and the comma separated > member ids > >> +# of the ECMP group of PREFIX on LR, and fails if the members listed > in its > >> +# select() action are not the ones that have a flow in > lr_in_ip_routing_ecmp. > >> +ecmp_ids() { > >> + local flows gid sel mids > >> + flows=$(ovn-sbctl dump-flows $1) > >> + gid=$(echo "$flows" | sed -n "s|.*ip[[46]]\.dst == > $2).*reg8\[[0\.\.15\]] = \([[0-9]]*\);.*|\1|p") > >> + sel=$(echo "$flows" | sed -n "s|.*ip[[46]]\.dst == > $2).*select(\([[^)]]*\));.*|\1|p" | tr -d ' ') > >> + mids=$(echo "$flows" | sed -n "s|.*(reg8\[[0\.\.15\]] == $gid && > reg8\[[16\.\.31\]] == \([[0-9]]*\)).*|\1|p" | sort -n | paste -sd, -) > >> + if test -z "$gid" || test "$sel" != "$mids"; then > >> + echo "$2: group '$gid', select($sel) != members($mids)" >&2 > >> + return 1 > >> + fi > >> + echo "$gid $sel" > >> +} > >> + > >> +# ecmp_member_id LR PREFIX NEXTHOP: prints the member id of NEXTHOP in > the > >> +# ECMP group of PREFIX on LR. > >> +ecmp_member_id() { > >> + local gid > >> + gid=$(ecmp_ids $1 $2 | cut -d' ' -f1) > >> + ovn-sbctl dump-flows $1 | sed -n "s|.*(reg8\[[0\.\.15\]] == $gid > && reg8\[[16\.\.31\]] == \([[0-9]]*\)), action=(reg0 = $3;.*|\1|p" > >> +} > >> + > >> ovn_strip_collector_set() { > >> sed 's/collector_set=[[0-9]]*,\?/collector_set=??,/g' > >> } > >> diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at > >> index d9682f78e..02f86878a 100644 > >> --- a/tests/ovn-northd.at > >> +++ b/tests/ovn-northd.at > >> @@ -7912,18 +7912,18 @@ check_row_count ECMP_Nexthop 1 > >> > >> ovn-sbctl dump-flows lr0 > lr0flows > >> > >> -AT_CHECK([grep -w "lr_in_ip_routing" lr0flows | ovn_strip_lflows], > [0], [dnl > >> +AT_CHECK([grep -w "lr_in_ip_routing" lr0flows | ovn_strip_ecmp_ids | > ovn_strip_lflows], [0], [dnl > >> table=??(lr_in_ip_routing ), priority=0 , match=(1), > action=(drop;) > >> table=??(lr_in_ip_routing ), priority=10300, > match=(ct_mark.ecmp_reply_port == 1 && reg7 == 0 && ip4.dst == 1.0.0.1/32), > action=(ip.ttl--; flags.loopback = 1; eth.src = 00:00:20:20:12:13; reg5 = > 192.168.0.1; outport = "lr0-public"; next;) > >> table=??(lr_in_ip_routing ), priority=10550, match=(nd_rs || > nd_ra), action=(drop;) > >> table=??(lr_in_ip_routing ), priority=1842 , match=(ip4.dst == > 192.168.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; > reg5 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; > flags.loopback = 1; reg9[[9]] = 1; next;) > >> - table=??(lr_in_ip_routing ), priority=1936 , match=(reg7 == 0 && > ip4.dst == 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = 1; reg8[[16..31]] = 1; next;) > >> + table=??(lr_in_ip_routing ), priority=1936 , match=(reg7 == 0 && > ip4.dst == 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = <mid>; next;) > >> table=??(lr_in_ip_routing ), priority=2322 , match=(inport == > "lr0-public" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; > xxreg0 = ip6.dst; xxreg1 = fe80::200:20ff:fe20:1213; eth.src = > 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; reg9[[9]] = > 0; next;) > >> ]) > >> > >> -AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | > ovn_strip_lflows], [0], [dnl > >> +AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | > ovn_strip_ecmp_ids | ovn_strip_lflows], [0], [dnl > >> table=??(lr_in_ip_routing_ecmp), priority=0 , match=(1), > action=(drop;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = > 192.168.0.10; reg5 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = > "lr0-public"; reg9[[9]] = 1; reg9[[10]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 192.168.0.10; reg5 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = > "lr0-public"; reg9[[9]] = 1; reg9[[10]] = 1; next;) > >> table=??(lr_in_ip_routing_ecmp), priority=150 , > match=(reg8[[0..15]] == 0), action=(next;) > >> ]) > >> > >> @@ -7938,18 +7938,18 @@ check_column 192.168.0.10 ECMP_Nexthop nexthop > nexthop=192.168.0.10 > >> check_column 192.168.0.20 ECMP_Nexthop nexthop nexthop=192.168.0.20 > >> > >> ovn-sbctl dump-flows lr0 > lr0flows > >> -AT_CHECK([grep -w "lr_in_ip_routing" lr0flows | ovn_strip_lflows], > [0], [dnl > >> +AT_CHECK([grep -w "lr_in_ip_routing" lr0flows | ovn_strip_ecmp_ids | > ovn_strip_lflows], [0], [dnl > >> table=??(lr_in_ip_routing ), priority=0 , match=(1), > action=(drop;) > >> table=??(lr_in_ip_routing ), priority=10300, > match=(ct_mark.ecmp_reply_port == 1 && reg7 == 0 && ip4.dst == 1.0.0.1/32), > action=(ip.ttl--; flags.loopback = 1; eth.src = 00:00:20:20:12:13; reg5 = > 192.168.0.1; outport = "lr0-public"; next;) > >> table=??(lr_in_ip_routing ), priority=10550, match=(nd_rs || > nd_ra), action=(drop;) > >> table=??(lr_in_ip_routing ), priority=1842 , match=(ip4.dst == > 192.168.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; > reg5 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; > flags.loopback = 1; reg9[[9]] = 1; next;) > >> - table=??(lr_in_ip_routing ), priority=1936 , match=(reg7 == 0 && > ip4.dst == 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > >> + table=??(lr_in_ip_routing ), priority=1936 , match=(reg7 == 0 && > ip4.dst == 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(<mid>, <mid>);) > >> table=??(lr_in_ip_routing ), priority=2322 , match=(inport == > "lr0-public" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; > xxreg0 = ip6.dst; xxreg1 = fe80::200:20ff:fe20:1213; eth.src = > 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; reg9[[9]] = > 0; next;) > >> ]) > >> -AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed > 's/192\.168\.0\..0/192.168.0.??/' | ovn_strip_lflows], [0], [dnl > >> +AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed > 's/192\.168\.0\..0/192.168.0.??/' | ovn_strip_ecmp_ids | > ovn_strip_lflows], [0], [dnl > >> table=??(lr_in_ip_routing_ecmp), priority=0 , match=(1), > action=(drop;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = > 192.168.0.??; reg5 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = > "lr0-public"; reg9[[9]] = 1; reg9[[10]] = 1; next;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 2), action=(reg0 = > 192.168.0.??; reg5 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = > "lr0-public"; reg9[[9]] = 1; reg9[[10]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 192.168.0.??; reg5 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = > "lr0-public"; reg9[[9]] = 1; reg9[[10]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 192.168.0.??; reg5 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = > "lr0-public"; reg9[[9]] = 1; reg9[[10]] = 1; next;) > >> table=??(lr_in_ip_routing_ecmp), priority=150 , > match=(reg8[[0..15]] == 0), action=(next;) > >> ]) > >> > >> @@ -7980,18 +7980,18 @@ check_column 192.168.0.10 ECMP_Nexthop nexthop > nexthop=192.168.0.10 > >> check_column 192.168.0.20 ECMP_Nexthop nexthop nexthop=192.168.0.20 > >> > >> ovn-sbctl dump-flows lr0 > lr0flows > >> -AT_CHECK([grep -w "lr_in_ip_routing" lr0flows | ovn_strip_lflows], > [0], [dnl > >> +AT_CHECK([grep -w "lr_in_ip_routing" lr0flows | ovn_strip_ecmp_ids | > ovn_strip_lflows], [0], [dnl > >> table=??(lr_in_ip_routing ), priority=0 , match=(1), > action=(drop;) > >> table=??(lr_in_ip_routing ), priority=10300, > match=(ct_mark.ecmp_reply_port == 1 && reg7 == 0 && ip4.dst == 1.0.0.1/32), > action=(ip.ttl--; flags.loopback = 1; eth.src = 00:00:20:20:12:13; reg5 = > 192.168.0.1; outport = "lr0-public"; next;) > >> table=??(lr_in_ip_routing ), priority=10550, match=(nd_rs || > nd_ra), action=(drop;) > >> table=??(lr_in_ip_routing ), priority=1842 , match=(ip4.dst == > 192.168.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; > reg5 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; > flags.loopback = 1; reg9[[9]] = 1; next;) > >> - table=??(lr_in_ip_routing ), priority=1936 , match=(reg7 == 0 && > ip4.dst == 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > >> + table=??(lr_in_ip_routing ), priority=1936 , match=(reg7 == 0 && > ip4.dst == 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(<mid>, <mid>);) > >> table=??(lr_in_ip_routing ), priority=2322 , match=(inport == > "lr0-public" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; > xxreg0 = ip6.dst; xxreg1 = fe80::200:20ff:fe20:1213; eth.src = > 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; reg9[[9]] = > 0; next;) > >> ]) > >> -AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed > 's/192\.168\.0\..0/192.168.0.??/' | ovn_strip_lflows], [0], [dnl > >> +AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed > 's/192\.168\.0\..0/192.168.0.??/' | ovn_strip_ecmp_ids | > ovn_strip_lflows], [0], [dnl > >> table=??(lr_in_ip_routing_ecmp), priority=0 , match=(1), > action=(drop;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = > 192.168.0.??; reg5 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = > "lr0-public"; reg9[[9]] = 1; reg9[[10]] = 1; next;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 2), action=(reg0 = > 192.168.0.??; reg5 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = > "lr0-public"; reg9[[9]] = 1; reg9[[10]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 192.168.0.??; reg5 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = > "lr0-public"; reg9[[9]] = 1; reg9[[10]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 192.168.0.??; reg5 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = > "lr0-public"; reg9[[9]] = 1; reg9[[10]] = 1; next;) > >> table=??(lr_in_ip_routing_ecmp), priority=150 , > match=(reg8[[0..15]] == 0), action=(next;) > >> ]) > >> > >> @@ -8029,8 +8029,8 @@ check ovn-nbctl set logical_router_static_route > $route2_uuid selection_fields="i > >> > >> check ovn-nbctl --wait=sb sync > >> ovn-sbctl dump-flows lr0 > lr0flows > >> -AT_CHECK([grep -e "(lr_in_ip_routing ).*3.3.0.0" lr0flows | sed > 's/table=../table=??/' | sort], [0], [dnl > >> - table=??(lr_in_ip_routing ), priority=1744 , match=(reg7 == 0 && > ip4.dst == 3.3.0.0/16), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = 1; reg8[[16..31]] = select(values=(1, 2); > hash_fields="ip_dst,ip_proto,ip_src");) > >> +AT_CHECK([grep -e "(lr_in_ip_routing ).*3.3.0.0" lr0flows | > ovn_strip_ecmp_ids | sed 's/table=../table=??/' | sort], [0], [dnl > >> + table=??(lr_in_ip_routing ), priority=1744 , match=(reg7 == 0 && > ip4.dst == 3.3.0.0/16), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(values=(<mid>, <mid>); > hash_fields="ip_dst,ip_proto,ip_src");) > >> ]) > >> > >> check ovn-nbctl set logical_router_static_route $route1_uuid > selection_fields="ip_src,ip_dst,tp_src,tp_dst" > >> @@ -8038,11 +8038,11 @@ check ovn-nbctl set logical_router_static_route > $route2_uuid selection_fields="i > >> > >> check ovn-nbctl --wait=sb sync > >> ovn-sbctl dump-flows lr0 > lr0flows > >> -AT_CHECK([grep -e "(lr_in_ip_routing ).*3.3.0.0" lr0flows | sed > 's/table=../table=??/' | sort], [0], [dnl > >> - table=??(lr_in_ip_routing ), priority=1744 , match=(reg7 == 0 && > ip4.dst == 3.3.0.0/16), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = 1; reg8[[16..31]] = select(values=(1, 2); > hash_fields="ip_dst,ip_proto,ip_src");) > >> - table=??(lr_in_ip_routing ), priority=1745 , match=(reg7 == 0 && > ip4.dst == 3.3.0.0/16 && sctp), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = 1; reg8[[16..31]] = select(values=(1, 2); > hash_fields="ip_dst,ip_proto,ip_src,sctp_dst,sctp_src");) > >> - table=??(lr_in_ip_routing ), priority=1745 , match=(reg7 == 0 && > ip4.dst == 3.3.0.0/16 && tcp), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = 1; reg8[[16..31]] = select(values=(1, 2); > hash_fields="ip_dst,ip_proto,ip_src,tcp_dst,tcp_src");) > >> - table=??(lr_in_ip_routing ), priority=1745 , match=(reg7 == 0 && > ip4.dst == 3.3.0.0/16 && udp), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = 1; reg8[[16..31]] = select(values=(1, 2); > hash_fields="ip_dst,ip_proto,ip_src,udp_dst,udp_src");) > >> +AT_CHECK([grep -e "(lr_in_ip_routing ).*3.3.0.0" lr0flows | > ovn_strip_ecmp_ids | sed 's/table=../table=??/' | sort], [0], [dnl > >> + table=??(lr_in_ip_routing ), priority=1744 , match=(reg7 == 0 && > ip4.dst == 3.3.0.0/16), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(values=(<mid>, <mid>); > hash_fields="ip_dst,ip_proto,ip_src");) > >> + table=??(lr_in_ip_routing ), priority=1745 , match=(reg7 == 0 && > ip4.dst == 3.3.0.0/16 && sctp), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(values=(<mid>, <mid>); > hash_fields="ip_dst,ip_proto,ip_src,sctp_dst,sctp_src");) > >> + table=??(lr_in_ip_routing ), priority=1745 , match=(reg7 == 0 && > ip4.dst == 3.3.0.0/16 && tcp), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(values=(<mid>, <mid>); > hash_fields="ip_dst,ip_proto,ip_src,tcp_dst,tcp_src");) > >> + table=??(lr_in_ip_routing ), priority=1745 , match=(reg7 == 0 && > ip4.dst == 3.3.0.0/16 && udp), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(values=(<mid>, <mid>); > hash_fields="ip_dst,ip_proto,ip_src,udp_dst,udp_src");) > >> ]) > >> > >> AT_CHECK([grep -e "lr_in_ecmp_stateful_egr" lr0flows | > ovn_strip_lflows], [0], [dnl > >> @@ -17551,19 +17551,19 @@ datapath=$(fetch_column datapath_binding > _uuid external_ids:name=lr0) > >> > >> # Validate the routes. > >> ovn-sbctl dump-flows lr0 > lr0flows > >> -AT_CHECK([grep -w "lr_in_ip_routing" lr0flows | ovn_strip_lflows], > [0], [dnl > >> +AT_CHECK([grep -w "lr_in_ip_routing" lr0flows | ovn_strip_ecmp_ids | > ovn_strip_lflows], [0], [dnl > >> table=??(lr_in_ip_routing ), priority=0 , match=(1), > action=(drop;) > >> table=??(lr_in_ip_routing ), priority=10550, match=(nd_rs || > nd_ra), action=(drop;) > >> - table=??(lr_in_ip_routing ), priority=1840 , match=(reg7 == 0 && > ip4.dst == 192.168.0.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > >> + table=??(lr_in_ip_routing ), priority=1840 , match=(reg7 == 0 && > ip4.dst == 192.168.0.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(<mid>, <mid>);) > >> table=??(lr_in_ip_routing ), priority=1842 , match=(ip4.dst == > 10.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg5 = > 10.0.0.1; eth.src = 00:00:00:00:ff:01; outport = "lr0-sw0"; flags.loopback > = 1; reg9[[9]] = 1; next;) > >> table=??(lr_in_ip_routing ), priority=1842 , match=(ip4.dst == > 10.0.1.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg5 = > 10.0.1.1; eth.src = 00:00:00:00:ff:02; outport = "lr0-sw1"; flags.loopback > = 1; reg9[[9]] = 1; next;) > >> table=??(lr_in_ip_routing ), priority=2322 , match=(inport == > "lr0-sw0" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; > xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:ff01; eth.src = > 00:00:00:00:ff:01; outport = "lr0-sw0"; flags.loopback = 1; reg9[[9]] = 0; > next;) > >> table=??(lr_in_ip_routing ), priority=2322 , match=(inport == > "lr0-sw1" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; > xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:ff02; eth.src = > 00:00:00:00:ff:02; outport = "lr0-sw1"; flags.loopback = 1; reg9[[9]] = 0; > next;) > >> ]) > >> -AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed -e > 's/10\.0\..\./10.0.??./g' -e 's/lr0-sw./lr0-sw??/' -e > 's/00:ff:0./00:ff:0?/' | ovn_strip_lflows], [0], [dnl > >> +AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed -e > 's/10\.0\..\./10.0.??./g' -e 's/lr0-sw./lr0-sw??/' -e > 's/00:ff:0./00:ff:0?/' | ovn_strip_ecmp_ids | ovn_strip_lflows], [0], [dnl > >> table=??(lr_in_ip_routing_ecmp), priority=0 , match=(1), > action=(drop;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = > 10.0.??.10; reg5 = 10.0.??.1; eth.src = 00:00:00:00:ff:0?; outport = > "lr0-sw??"; reg9[[9]] = 1; next;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 2), action=(reg0 = > 10.0.??.10; reg5 = 10.0.??.1; eth.src = 00:00:00:00:ff:0?; outport = > "lr0-sw??"; reg9[[9]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 10.0.??.10; reg5 = 10.0.??.1; eth.src = 00:00:00:00:ff:0?; outport = > "lr0-sw??"; reg9[[9]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 10.0.??.10; reg5 = 10.0.??.1; eth.src = 00:00:00:00:ff:0?; outport = > "lr0-sw??"; reg9[[9]] = 1; next;) > >> table=??(lr_in_ip_routing_ecmp), priority=150 , > match=(reg8[[0..15]] == 0), action=(next;) > >> ]) > >> > >> @@ -17577,20 +17577,20 @@ check ovn-nbctl --wait=sb sync > >> check_row_count Advertised_Route 4 > >> check_row_count Learned_Route 1 > >> ovn-sbctl dump-flows lr0 > lr0flows > >> -AT_CHECK([grep -w "lr_in_ip_routing" lr0flows | ovn_strip_lflows], > [0], [dnl > >> +AT_CHECK([grep -w "lr_in_ip_routing" lr0flows | ovn_strip_ecmp_ids | > ovn_strip_lflows], [0], [dnl > >> table=??(lr_in_ip_routing ), priority=0 , match=(1), > action=(drop;) > >> table=??(lr_in_ip_routing ), priority=10550, match=(nd_rs || > nd_ra), action=(drop;) > >> table=??(lr_in_ip_routing ), priority=1838 , match=(reg7 == 0 && > ip4.dst == 192.168.1.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = > 10.0.0.20; reg5 = 10.0.0.1; eth.src = 00:00:00:00:ff:01; outport = > "lr0-sw0"; flags.loopback = 1; reg9[[9]] = 1; next;) > >> - table=??(lr_in_ip_routing ), priority=1840 , match=(reg7 == 0 && > ip4.dst == 192.168.0.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > >> + table=??(lr_in_ip_routing ), priority=1840 , match=(reg7 == 0 && > ip4.dst == 192.168.0.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(<mid>, <mid>);) > >> table=??(lr_in_ip_routing ), priority=1842 , match=(ip4.dst == > 10.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg5 = > 10.0.0.1; eth.src = 00:00:00:00:ff:01; outport = "lr0-sw0"; flags.loopback > = 1; reg9[[9]] = 1; next;) > >> table=??(lr_in_ip_routing ), priority=1842 , match=(ip4.dst == > 10.0.1.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg5 = > 10.0.1.1; eth.src = 00:00:00:00:ff:02; outport = "lr0-sw1"; flags.loopback > = 1; reg9[[9]] = 1; next;) > >> table=??(lr_in_ip_routing ), priority=2322 , match=(inport == > "lr0-sw0" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; > xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:ff01; eth.src = > 00:00:00:00:ff:01; outport = "lr0-sw0"; flags.loopback = 1; reg9[[9]] = 0; > next;) > >> table=??(lr_in_ip_routing ), priority=2322 , match=(inport == > "lr0-sw1" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; > xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:ff02; eth.src = > 00:00:00:00:ff:02; outport = "lr0-sw1"; flags.loopback = 1; reg9[[9]] = 0; > next;) > >> ]) > >> -AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed -e > 's/10\.0\..\./10.0.??./g' -e 's/lr0-sw./lr0-sw??/' -e > 's/00:ff:0./00:ff:0?/' | ovn_strip_lflows], [0], [dnl > >> +AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed -e > 's/10\.0\..\./10.0.??./g' -e 's/lr0-sw./lr0-sw??/' -e > 's/00:ff:0./00:ff:0?/' | ovn_strip_ecmp_ids | ovn_strip_lflows], [0], [dnl > >> table=??(lr_in_ip_routing_ecmp), priority=0 , match=(1), > action=(drop;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = > 10.0.??.10; reg5 = 10.0.??.1; eth.src = 00:00:00:00:ff:0?; outport = > "lr0-sw??"; reg9[[9]] = 1; next;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 2), action=(reg0 = > 10.0.??.10; reg5 = 10.0.??.1; eth.src = 00:00:00:00:ff:0?; outport = > "lr0-sw??"; reg9[[9]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 10.0.??.10; reg5 = 10.0.??.1; eth.src = 00:00:00:00:ff:0?; outport = > "lr0-sw??"; reg9[[9]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 10.0.??.10; reg5 = 10.0.??.1; eth.src = 00:00:00:00:ff:0?; outport = > "lr0-sw??"; reg9[[9]] = 1; next;) > >> table=??(lr_in_ip_routing_ecmp), priority=150 , > match=(reg8[[0..15]] == 0), action=(next;) > >> ]) > >> > >> @@ -17605,22 +17605,22 @@ check ovn-nbctl --wait=sb sync > >> check_row_count Advertised_Route 4 > >> check_row_count Learned_Route 2 > >> ovn-sbctl dump-flows lr0 > lr0flows > >> -AT_CHECK([grep -w "lr_in_ip_routing" lr0flows | ovn_strip_lflows], > [0], [dnl > >> +AT_CHECK([grep -w "lr_in_ip_routing" lr0flows | ovn_strip_ecmp_ids | > ovn_strip_lflows], [0], [dnl > >> table=??(lr_in_ip_routing ), priority=0 , match=(1), > action=(drop;) > >> table=??(lr_in_ip_routing ), priority=10550, match=(nd_rs || > nd_ra), action=(drop;) > >> - table=??(lr_in_ip_routing ), priority=1838 , match=(reg7 == 0 && > ip4.dst == 192.168.1.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = 2; reg8[[16..31]] = select(1, 2);) > >> - table=??(lr_in_ip_routing ), priority=1840 , match=(reg7 == 0 && > ip4.dst == 192.168.0.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > >> + table=??(lr_in_ip_routing ), priority=1838 , match=(reg7 == 0 && > ip4.dst == 192.168.1.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(<mid>, <mid>);) > >> + table=??(lr_in_ip_routing ), priority=1840 , match=(reg7 == 0 && > ip4.dst == 192.168.0.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(<mid>, <mid>);) > >> table=??(lr_in_ip_routing ), priority=1842 , match=(ip4.dst == > 10.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg5 = > 10.0.0.1; eth.src = 00:00:00:00:ff:01; outport = "lr0-sw0"; flags.loopback > = 1; reg9[[9]] = 1; next;) > >> table=??(lr_in_ip_routing ), priority=1842 , match=(ip4.dst == > 10.0.1.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg5 = > 10.0.1.1; eth.src = 00:00:00:00:ff:02; outport = "lr0-sw1"; flags.loopback > = 1; reg9[[9]] = 1; next;) > >> table=??(lr_in_ip_routing ), priority=2322 , match=(inport == > "lr0-sw0" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; > xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:ff01; eth.src = > 00:00:00:00:ff:01; outport = "lr0-sw0"; flags.loopback = 1; reg9[[9]] = 0; > next;) > >> table=??(lr_in_ip_routing ), priority=2322 , match=(inport == > "lr0-sw1" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; > xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:ff02; eth.src = > 00:00:00:00:ff:02; outport = "lr0-sw1"; flags.loopback = 1; reg9[[9]] = 0; > next;) > >> ]) > >> -AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed -e > 's/10\.0\..\./10.0.??./g' -e 's/lr0-sw./lr0-sw??/' -e > 's/00:ff:0./00:ff:0?/' | ovn_strip_lflows], [0], [dnl > >> +AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed -e > 's/10\.0\..\./10.0.??./g' -e 's/lr0-sw./lr0-sw??/' -e > 's/00:ff:0./00:ff:0?/' | ovn_strip_ecmp_ids | ovn_strip_lflows], [0], [dnl > >> table=??(lr_in_ip_routing_ecmp), priority=0 , match=(1), > action=(drop;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = > 10.0.??.10; reg5 = 10.0.??.1; eth.src = 00:00:00:00:ff:0?; outport = > "lr0-sw??"; reg9[[9]] = 1; next;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 2), action=(reg0 = > 10.0.??.10; reg5 = 10.0.??.1; eth.src = 00:00:00:00:ff:0?; outport = > "lr0-sw??"; reg9[[9]] = 1; next;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 2 && reg8[[16..31]] == 1), action=(reg0 = > 10.0.??.20; reg5 = 10.0.??.1; eth.src = 00:00:00:00:ff:0?; outport = > "lr0-sw??"; reg9[[9]] = 1; next;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 2 && reg8[[16..31]] == 2), action=(reg0 = > 10.0.??.20; reg5 = 10.0.??.1; eth.src = 00:00:00:00:ff:0?; outport = > "lr0-sw??"; reg9[[9]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 10.0.??.10; reg5 = 10.0.??.1; eth.src = 00:00:00:00:ff:0?; outport = > "lr0-sw??"; reg9[[9]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 10.0.??.10; reg5 = 10.0.??.1; eth.src = 00:00:00:00:ff:0?; outport = > "lr0-sw??"; reg9[[9]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 10.0.??.20; reg5 = 10.0.??.1; eth.src = 00:00:00:00:ff:0?; outport = > "lr0-sw??"; reg9[[9]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 10.0.??.20; reg5 = 10.0.??.1; eth.src = 00:00:00:00:ff:0?; outport = > "lr0-sw??"; reg9[[9]] = 1; next;) > >> table=??(lr_in_ip_routing_ecmp), priority=150 , > match=(reg8[[0..15]] == 0), action=(next;) > >> ]) > >> > >> @@ -17647,12 +17647,12 @@ check ovn-nbctl --wait=sb sync > >> check_row_count Advertised_Route 4 > >> check_row_count Learned_Route 4 > >> ovn-sbctl dump-flows lr0 > lr0flows > >> -AT_CHECK([grep -w "lr_in_ip_routing" lr0flows | sed -e > 's/reg8\[[0..15\]] = [[123]]/reg8\[[0..15\]] = ??/' | ovn_strip_lflows], > [0], [dnl > >> +AT_CHECK([grep -w "lr_in_ip_routing" lr0flows | ovn_strip_ecmp_ids | > ovn_strip_lflows], [0], [dnl > >> table=??(lr_in_ip_routing ), priority=0 , match=(1), > action=(drop;) > >> table=??(lr_in_ip_routing ), priority=10550, match=(nd_rs || > nd_ra), action=(drop;) > >> - table=??(lr_in_ip_routing ), priority=1838 , match=(reg7 == 0 && > ip4.dst == 192.168.0.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = ??; reg8[[16..31]] = select(1, 2);) > >> - table=??(lr_in_ip_routing ), priority=1838 , match=(reg7 == 0 && > ip4.dst == 192.168.1.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = ??; reg8[[16..31]] = select(1, 2);) > >> - table=??(lr_in_ip_routing ), priority=1840 , match=(reg7 == 0 && > ip4.dst == 192.168.0.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = ??; reg8[[16..31]] = select(1, 2);) > >> + table=??(lr_in_ip_routing ), priority=1838 , match=(reg7 == 0 && > ip4.dst == 192.168.0.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(<mid>, <mid>);) > >> + table=??(lr_in_ip_routing ), priority=1838 , match=(reg7 == 0 && > ip4.dst == 192.168.1.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(<mid>, <mid>);) > >> + table=??(lr_in_ip_routing ), priority=1840 , match=(reg7 == 0 && > ip4.dst == 192.168.0.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(<mid>, <mid>);) > >> table=??(lr_in_ip_routing ), priority=1842 , match=(ip4.dst == > 10.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg5 = > 10.0.0.1; eth.src = 00:00:00:00:ff:01; outport = "lr0-sw0"; flags.loopback > = 1; reg9[[9]] = 1; next;) > >> table=??(lr_in_ip_routing ), priority=1842 , match=(ip4.dst == > 10.0.1.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg5 = > 10.0.1.1; eth.src = 00:00:00:00:ff:02; outport = "lr0-sw1"; flags.loopback > = 1; reg9[[9]] = 1; next;) > >> table=??(lr_in_ip_routing ), priority=2322 , match=(inport == > "lr0-sw0" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; > xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:ff01; eth.src = > 00:00:00:00:ff:01; outport = "lr0-sw0"; flags.loopback = 1; reg9[[9]] = 0; > next;) > >> @@ -17697,13 +17697,13 @@ check_uuid ovn-sbctl create Learned_Route \ > >> check ovn-nbctl --wait=sb sync > >> > >> ovn-sbctl dump-flows lr0 > lr0flows > >> -AT_CHECK([grep -w "lr_in_ip_routing" lr0flows | grep -F "42.42.42.42" > | ovn_strip_lflows], [0], [dnl > >> - table=??(lr_in_ip_routing ), priority=1934 , match=(reg7 == 0 && > ip4.dst == 42.42.42.42/32), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > >> +AT_CHECK([grep -w "lr_in_ip_routing" lr0flows | grep -F "42.42.42.42" > | ovn_strip_ecmp_ids | ovn_strip_lflows], [0], [dnl > >> + table=??(lr_in_ip_routing ), priority=1934 , match=(reg7 == 0 && > ip4.dst == 42.42.42.42/32), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(<mid>, <mid>);) > >> ]) > >> > >> -AT_CHECK([grep "lr_in_ip_routing_ecmp" lr0flows | grep "42" | sed -e > 's/reg8\[[16..31\]] == [[12]]/reg8\[[16..31\]] == ??/g' | > ovn_strip_lflows], [0], [dnl > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == ??), action=(reg0 = > 192.168.2.42; eth.src = 00:00:00:00:ff:02; outport = "lrp2"; reg9[[9]] = 1; > next;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == ??), action=(reg0 = > 192.168.3.42; eth.src = 00:00:00:00:ff:03; outport = "lrp3"; reg9[[9]] = 1; > next;) > >> +AT_CHECK([grep "lr_in_ip_routing_ecmp" lr0flows | grep "42" | > ovn_strip_ecmp_ids | ovn_strip_lflows], [0], [dnl > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 192.168.2.42; eth.src = 00:00:00:00:ff:02; outport = "lrp2"; reg9[[9]] = 1; > next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 192.168.3.42; eth.src = 00:00:00:00:ff:03; outport = "lrp3"; reg9[[9]] = 1; > next;) > >> ]) > >> > >> OVN_CLEANUP_NORTHD > >> @@ -19461,8 +19461,8 @@ check_row_count Advertised_Route 1 \ > >> # One ECMP selection flow covers the prefix and two member flows carry > the > >> # respective neighbor nexthops. > >> ovn-sbctl lflow-list lr0 > lr0_flows > >> -AT_CHECK([grep 'lr_in_ip_routing.*172.16.1.10/32' lr0_flows | \ > >> - grep -c 'select(1, 2)'], [0], [1 > >> +AT_CHECK([grep 'lr_in_ip_routing.*172.16.1.10/32' lr0_flows | > ovn_strip_ecmp_ids | \ > >> + grep -c 'select(<mid>, <mid>)'], [0], [1 > >> ]) > >> AT_CHECK([grep 'lr_in_ip_routing_ecmp' lr0_flows | \ > >> grep -E -c 'reg0 = 10\.0\.0\.[[12]];'], [0], [2 > >> @@ -19481,6 +19481,85 @@ OVN_CLEANUP_NORTHD > >> AT_CLEANUP > >> ]) > >> > >> +OVN_FOR_EACH_NORTHD_NO_HV([ > >> +AT_SETUP([dynamic-routing - NAT and LB reachable over several paths > use ECMP]) > >> +AT_KEYWORDS([dynamic-routing ecmp]) > >> +ovn_start > >> + > >> +# lr0 reaches lr1 over the switches "a" and "b" and redistributes NAT > and LB > >> +# addresses on both ports. The NAT and the LB of lr1 then produce one > >> +# forwarding route per path on lr0. The two routes of each ECMP group > share > >> +# their source row and only differ in the nexthop and output port, > they must > >> +# still get distinct ids that don't depend on the processing order. > >> + > >> +check ovn-nbctl lr-add lr0 -- set Logical_Router lr0 \ > >> + options:dynamic-routing=true options:chassis=hv1 > >> +check ovn-nbctl lr-add lr1 -- set Logical_Router lr1 > options:chassis=hv1 > >> +i=0 > >> +for s in a b; do > >> + i=$((i + 1)) > >> + check ovn-nbctl ls-add $s > >> + check ovn-nbctl lrp-add lr0 lr0-$s 00:00:00:00:0$i:01 10.0.$i.1/24 > >> + check ovn-nbctl lrp-set-options lr0-$s > dynamic-routing-redistribute=lb,nat > >> + check ovn-nbctl lsp-add-router-port $s $s-lr0 lr0-$s > >> + check ovn-nbctl lrp-add lr1 lr1-$s 00:00:00:00:0$i:02 10.0.$i.2/24 > >> + check ovn-nbctl lsp-add-router-port $s $s-lr1 lr1-$s > >> +done > >> +check ovn-nbctl lr-nat-add lr1 dnat_and_snat 172.16.1.10 192.168.1.10 > >> +check ovn-nbctl lb-add lb1 172.16.1.20:80 192.168.1.20:80 > >> +check ovn-nbctl --wait=sb lr-lb-add lr1 lb1 > >> + > >> +ovn-sbctl dump-flows lr0 > lr0flows > >> +AT_CHECK([grep -w "lr_in_ip_routing" lr0flows | grep "172\.16\.1\." | > ovn_strip_ecmp_ids | ovn_strip_lflows], [0], [dnl > >> + table=??(lr_in_ip_routing ), priority=1935 , match=(ip4.dst == > 172.16.1.10/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = > <gid>; reg8[[16..31]] = select(<mid>, <mid>);) > >> + table=??(lr_in_ip_routing ), priority=1935 , match=(ip4.dst == > 172.16.1.20/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = > <gid>; reg8[[16..31]] = select(<mid>, <mid>);) > >> +]) > >> +AT_CHECK([grep "lr_in_ip_routing_ecmp" lr0flows | grep "reg0 = > 10\.0\." | ovn_strip_ecmp_ids | ovn_strip_lflows], [0], [dnl > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 10.0.1.2; reg5 = 10.0.1.1; eth.src = 00:00:00:00:01:01; outport = "lr0-a"; > reg9[[9]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 10.0.1.2; reg5 = 10.0.1.1; eth.src = 00:00:00:00:01:01; outport = "lr0-a"; > reg9[[9]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 10.0.2.2; reg5 = 10.0.2.1; eth.src = 00:00:00:00:02:01; outport = "lr0-b"; > reg9[[9]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 10.0.2.2; reg5 = 10.0.2.1; eth.src = 00:00:00:00:02:01; outport = "lr0-b"; > reg9[[9]] = 1; next;) > >> +]) > >> + > >> +dnl Both members of each group come from the same NAT or LB row. > >> +nat=$(fetch_column nb:NAT _uuid external_ip=172.16.1.10) > >> +lb=$(fetch_column nb:Load_Balancer _uuid name=lb1) > >> +check_row_count Logical_Flow 2 > external_ids:stage-name=lr_in_ip_routing_ecmp \ > >> + external_ids:stage-hint=${nat:0:8} > >> +check_row_count Logical_Flow 2 > external_ids:stage-name=lr_in_ip_routing_ecmp \ > >> + external_ids:stage-hint=${lb:0:8} > >> + > >> +dnl ecmp_ids also checks that the members have distinct ids. > >> +nat_ids=$(ecmp_ids lr0 172.16.1.10/32) > >> +lb_ids=$(ecmp_ids lr0 172.16.1.20/32) > >> +AT_CHECK([test -n "$nat_ids" && test -n "$lb_ids"]) > >> +CHECK_NO_CHANGE_AFTER_RECOMPUTE > >> + > >> +dnl A third path adds a member to each group without changing the ids > of the > >> +dnl existing members. > >> +check ovn-nbctl ls-add c > >> +check ovn-nbctl lrp-add lr0 lr0-c 00:00:00:00:03:01 10.0.3.1/24 > >> +check ovn-nbctl lrp-set-options lr0-c > dynamic-routing-redistribute=lb,nat > >> +check ovn-nbctl lsp-add-router-port c c-lr0 lr0-c > >> +check ovn-nbctl lrp-add lr1 lr1-c 00:00:00:00:03:02 10.0.3.2/24 > >> +check ovn-nbctl --wait=sb lsp-add-router-port c c-lr1 lr1-c > >> +CHECK_NO_CHANGE_AFTER_RECOMPUTE > >> + > >> +for old in "$nat_ids 172.16.1.10/32" "$lb_ids 172.16.1.20/32"; do > >> + set -- $old > >> + new=$(ecmp_ids lr0 $3) > >> + AT_CHECK([test "${new%% *}" = "$1"]) > >> + AT_CHECK([echo "${new#* }" | tr , '\n' | wc -l], [0], [3 > >> +]) > >> + for m in $(echo $2 | tr , ' '); do > >> + AT_CHECK([echo "${new#* }" | tr , '\n' | grep -qx $m]) > >> + done > >> +done > >> + > >> +OVN_CLEANUP_NORTHD > >> +AT_CLEANUP > >> +]) > >> + > >> OVN_FOR_EACH_NORTHD_NO_HV([ > >> AT_SETUP([dynamic-routing - LB redistribute uses backend LSP as > tracked_port]) > >> AT_KEYWORDS([dynamic-routing]) > >> @@ -24203,6 +24282,87 @@ done > >> OVN_CLEANUP_NORTHD > >> AT_CLEANUP > >> > >> +OVN_FOR_EACH_NORTHD_NO_HV([ > >> +AT_SETUP([Static routes - ECMP ids stable across recompute]) > >> +AT_KEYWORDS([ecmp]) > >> +ovn_start > >> + > >> +check ovn-nbctl lr-add lr0 > >> +for i in 1 2 3 4; do > >> + check ovn-nbctl lrp-add lr0 lr0-sw$i 00:00:00:00:0$i:01 > 10.0.$i.1/24 > >> +done > >> +check ovn-nbctl --wait=sb sync > >> + > >> +dnl Add the routes in "reverse" order, the ids must not depend on it. > >> +for p in 192.168.30.0/24 192.168.20.0/24 192.168.10.0/24; do > >> + for i in 3 2 1; do > >> + check ovn-nbctl --wait=sb --ecmp lr-route-add lr0 $p 10.0.$i.10 > >> + done > >> +done > >> +CHECK_NO_CHANGE_AFTER_RECOMPUTE > >> + > >> +AT_CHECK([ovn-sbctl dump-flows lr0 | grep -w lr_in_ip_routing | grep > select | ovn_strip_ecmp_ids | ovn_strip_lflows], [0], [dnl > >> + table=??(lr_in_ip_routing ), priority=1840 , match=(reg7 == 0 && > ip4.dst == 192.168.10.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(<mid>, <mid>, <mid>);) > >> + table=??(lr_in_ip_routing ), priority=1840 , match=(reg7 == 0 && > ip4.dst == 192.168.20.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(<mid>, <mid>, <mid>);) > >> + table=??(lr_in_ip_routing ), priority=1840 , match=(reg7 == 0 && > ip4.dst == 192.168.30.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(<mid>, <mid>, <mid>);) > >> +]) > >> +g10=$(ecmp_ids lr0 192.168.10.0/24) > >> +g20=$(ecmp_ids lr0 192.168.20.0/24) > >> +g30=$(ecmp_ids lr0 192.168.30.0/24) > >> +AT_CHECK([test -n "$g10" && test -n "$g20" && test -n "$g30"]) > >> +AT_CHECK([printf '%s\n' "$g10" "$g20" "$g30" | cut -d' ' -f1 | sort -u > | wc -l], [0], [3 > >> +]) > >> + > >> +dnl Remove a member from the middle of a group with more than 2 > members, the > >> +dnl ids of the other members and groups must not change. > >> +m2=$(ecmp_member_id lr0 192.168.10.0/24 10.0.2.10) > >> +check as northd ovn-appctl -t ovn-northd inc-engine/clear-stats > >> +check ovn-nbctl --wait=sb lr-route-del lr0 192.168.10.0/24 10.0.2.10 > >> +check_engine_compute group_ecmp_route incremental > >> +CHECK_NO_CHANGE_AFTER_RECOMPUTE > >> +g10_ids=$(echo "${g10#* }" | tr , '\n' | grep -vx "$m2" | paste -sd, -) > >> +AT_CHECK_UNQUOTED([ecmp_ids lr0 192.168.10.0/24], [0], [${g10%% *} > $g10_ids > >> +]) > >> +AT_CHECK_UNQUOTED([ecmp_ids lr0 192.168.20.0/24], [0], [$g20 > >> +]) > >> +AT_CHECK_UNQUOTED([ecmp_ids lr0 192.168.30.0/24], [0], [$g30 > >> +]) > >> +g10=$(ecmp_ids lr0 192.168.10.0/24) > >> + > >> +dnl Remove a whole group and then add a new one, the ids of the other > groups > >> +dnl must not change and the group ids must stay unique. > >> +check as northd ovn-appctl -t ovn-northd inc-engine/clear-stats > >> +check ovn-nbctl --wait=sb lr-route-del lr0 192.168.20.0/24 > >> +check ovn-nbctl --wait=sb --ecmp lr-route-add lr0 192.168.40.0/24 > 10.0.1.10 > >> +check ovn-nbctl --wait=sb --ecmp lr-route-add lr0 192.168.40.0/24 > 10.0.4.10 > >> +check_engine_compute group_ecmp_route incremental > >> +CHECK_NO_CHANGE_AFTER_RECOMPUTE > >> +AT_CHECK_UNQUOTED([ecmp_ids lr0 192.168.10.0/24], [0], [$g10 > >> +]) > >> +AT_CHECK_UNQUOTED([ecmp_ids lr0 192.168.30.0/24], [0], [$g30 > >> +]) > >> +g40=$(ecmp_ids lr0 192.168.40.0/24) > >> +AT_CHECK([test -n "$g40"]) > >> +AT_CHECK([printf '%s\n' "$g10" "$g30" "$g40" | cut -d' ' -f1 | sort -u > | wc -l], [0], [3 > >> +]) > >> + > >> +dnl Once no member is a discard route anymore the group must stop > dropping. > >> +route=$(ovn-nbctl --bare --columns _uuid find > Logical_Router_Static_Route \ > >> + ip_prefix="192.168.30.0/24" nexthop="10.0.3.10") > >> +check ovn-nbctl --wait=sb set Logical_Router_Static_Route $route > nexthop=discard > >> +AT_CHECK([ovn-sbctl dump-flows lr0 | grep lr_in_ip_routing_ecmp | grep > -c "drop;"], [0], [4 > >> +]) > >> +check as northd ovn-appctl -t ovn-northd inc-engine/clear-stats > >> +check ovn-nbctl --wait=sb set Logical_Router_Static_Route $route > nexthop=10.0.3.10 > >> +check_engine_compute group_ecmp_route incremental > >> +AT_CHECK([ovn-sbctl dump-flows lr0 | grep lr_in_ip_routing_ecmp | grep > -c "drop;"], [0], [1 > >> +]) > >> +CHECK_NO_CHANGE_AFTER_RECOMPUTE > >> + > >> +OVN_CLEANUP_NORTHD > >> +AT_CLEANUP > >> +]) > >> + > >> OVN_FOR_EACH_NORTHD_NO_HV([ > >> AT_SETUP([Static routes - ECMP with discard]) > >> ovn_start > >> @@ -24217,14 +24377,14 @@ check ovn-nbctl > \ > >> check ovn-nbctl --wait=sb sync > >> > >> ovn-sbctl dump-flows lr0 > lr0flows > >> -AT_CHECK([grep 'lr_in_ip_routing' lr0flows | grep 'select' | > ovn_strip_lflows], [0], [dnl > >> - table=??(lr_in_ip_routing ), priority=1840 , match=(reg7 == 0 && > ip4.dst == 192.168.10.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > >> +AT_CHECK([grep 'lr_in_ip_routing' lr0flows | grep 'select' | > ovn_strip_ecmp_ids | ovn_strip_lflows], [0], [dnl > >> + table=??(lr_in_ip_routing ), priority=1840 , match=(reg7 == 0 && > ip4.dst == 192.168.10.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(<mid>, <mid>);) > >> ]) > >> > >> -AT_CHECK([grep 'lr_in_ip_routing_ecmp' lr0flows | sed -E 's/== > [[1-9]]+\)/== \?)/' | ovn_strip_lflows], [0], [dnl > >> +AT_CHECK([grep 'lr_in_ip_routing_ecmp' lr0flows | ovn_strip_ecmp_ids | > ovn_strip_lflows], [0], [dnl > >> table=??(lr_in_ip_routing_ecmp), priority=0 , match=(1), > action=(drop;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == ?), action=(reg0 = > 10.0.10.10; reg5 = 10.0.10.1; eth.src = 00:00:00:00:10:01; outport = > "lr0-sw1"; reg9[[9]] = 1; next;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == ?), action=(reg0 = > 10.0.20.10; reg5 = 10.0.20.1; eth.src = 00:00:00:00:20:01; outport = > "lr0-sw2"; reg9[[9]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 10.0.10.10; reg5 = 10.0.10.1; eth.src = 00:00:00:00:10:01; outport = > "lr0-sw1"; reg9[[9]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 10.0.20.10; reg5 = 10.0.20.1; eth.src = 00:00:00:00:20:01; outport = > "lr0-sw2"; reg9[[9]] = 1; next;) > >> table=??(lr_in_ip_routing_ecmp), priority=150 , > match=(reg8[[0..15]] == 0), action=(next;) > >> ]) > >> > >> @@ -24235,14 +24395,14 @@ ecmp2=$(fetch_column > nb:Logical_Router_Static_Route _uuid nexthop="10.0.20.10") > >> check ovn-nbctl --wait=sb set Logical_Router_Static_Route $ecmp1 > nexthop="discard" > >> > >> ovn-sbctl dump-flows lr0 > lr0flows > >> -AT_CHECK([grep 'lr_in_ip_routing' lr0flows | grep 'select' | > ovn_strip_lflows], [0], [dnl > >> - table=??(lr_in_ip_routing ), priority=1840 , match=(reg7 == 0 && > ip4.dst == 192.168.10.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > >> +AT_CHECK([grep 'lr_in_ip_routing' lr0flows | grep 'select' | > ovn_strip_ecmp_ids | ovn_strip_lflows], [0], [dnl > >> + table=??(lr_in_ip_routing ), priority=1840 , match=(reg7 == 0 && > ip4.dst == 192.168.10.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(<mid>, <mid>);) > >> ]) > >> > >> -AT_CHECK([grep 'lr_in_ip_routing_ecmp' lr0flows | sed -E 's/== > [[1-9]]+\)/== \?)/' | ovn_strip_lflows], [0], [dnl > >> +AT_CHECK([grep 'lr_in_ip_routing_ecmp' lr0flows | ovn_strip_ecmp_ids | > ovn_strip_lflows], [0], [dnl > >> table=??(lr_in_ip_routing_ecmp), priority=0 , match=(1), > action=(drop;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == ?), action=(drop;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == ?), action=(drop;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(drop;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(drop;) > >> table=??(lr_in_ip_routing_ecmp), priority=150 , > match=(reg8[[0..15]] == 0), action=(next;) > >> ]) > >> > >> @@ -24250,14 +24410,14 @@ AT_CHECK([grep 'lr_in_ip_routing_ecmp' > lr0flows | sed -E 's/== [[1-9]]+\)/== \?) > >> check ovn-nbctl --wait=sb set Logical_Router_Static_Route $ecmp2 > nexthop="discard" > >> > >> ovn-sbctl dump-flows lr0 > lr0flows > >> -AT_CHECK([grep 'lr_in_ip_routing' lr0flows | grep 'select' | > ovn_strip_lflows], [0], [dnl > >> - table=??(lr_in_ip_routing ), priority=1840 , match=(reg7 == 0 && > ip4.dst == 192.168.10.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > >> +AT_CHECK([grep 'lr_in_ip_routing' lr0flows | grep 'select' | > ovn_strip_ecmp_ids | ovn_strip_lflows], [0], [dnl > >> + table=??(lr_in_ip_routing ), priority=1840 , match=(reg7 == 0 && > ip4.dst == 192.168.10.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(<mid>, <mid>);) > >> ]) > >> > >> -AT_CHECK([grep 'lr_in_ip_routing_ecmp' lr0flows | sed -E 's/== > [[1-9]]+\)/== \?)/' | ovn_strip_lflows], [0], [dnl > >> +AT_CHECK([grep 'lr_in_ip_routing_ecmp' lr0flows | ovn_strip_ecmp_ids | > ovn_strip_lflows], [0], [dnl > >> table=??(lr_in_ip_routing_ecmp), priority=0 , match=(1), > action=(drop;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == ?), action=(drop;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == ?), action=(drop;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(drop;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(drop;) > >> table=??(lr_in_ip_routing_ecmp), priority=150 , > match=(reg8[[0..15]] == 0), action=(next;) > >> ]) > >> > >> @@ -24266,14 +24426,14 @@ check ovn-nbctl set > Logical_Router_Static_Route $ecmp1 nexthop="10.0.10.10" > >> check ovn-nbctl --wait=sb set Logical_Router_Static_Route $ecmp2 > nexthop="10.0.20.10" > >> > >> ovn-sbctl dump-flows lr0 > lr0flows > >> -AT_CHECK([grep 'lr_in_ip_routing' lr0flows | grep 'select' | > ovn_strip_lflows], [0], [dnl > >> - table=??(lr_in_ip_routing ), priority=1840 , match=(reg7 == 0 && > ip4.dst == 192.168.10.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > >> +AT_CHECK([grep 'lr_in_ip_routing' lr0flows | grep 'select' | > ovn_strip_ecmp_ids | ovn_strip_lflows], [0], [dnl > >> + table=??(lr_in_ip_routing ), priority=1840 , match=(reg7 == 0 && > ip4.dst == 192.168.10.0/24), action=(ip.ttl--; flags.loopback = 1; > reg8[[0..15]] = <gid>; reg8[[16..31]] = select(<mid>, <mid>);) > >> ]) > >> > >> -AT_CHECK([grep 'lr_in_ip_routing_ecmp' lr0flows | sed -E 's/== > [[1-9]]+\)/== \?)/' | ovn_strip_lflows], [0], [dnl > >> +AT_CHECK([grep 'lr_in_ip_routing_ecmp' lr0flows | ovn_strip_ecmp_ids | > ovn_strip_lflows], [0], [dnl > >> table=??(lr_in_ip_routing_ecmp), priority=0 , match=(1), > action=(drop;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == ?), action=(reg0 = > 10.0.10.10; reg5 = 10.0.10.1; eth.src = 00:00:00:00:10:01; outport = > "lr0-sw1"; reg9[[9]] = 1; next;) > >> - table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == ?), action=(reg0 = > 10.0.20.10; reg5 = 10.0.20.1; eth.src = 00:00:00:00:20:01; outport = > "lr0-sw2"; reg9[[9]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 10.0.10.10; reg5 = 10.0.10.1; eth.src = 00:00:00:00:10:01; outport = > "lr0-sw1"; reg9[[9]] = 1; next;) > >> + table=??(lr_in_ip_routing_ecmp), priority=100 , > match=(reg8[[0..15]] == <gid> && reg8[[16..31]] == <mid>), action=(reg0 = > 10.0.20.10; reg5 = 10.0.20.1; eth.src = 00:00:00:00:20:01; outport = > "lr0-sw2"; reg9[[9]] = 1; next;) > >> table=??(lr_in_ip_routing_ecmp), priority=150 , > match=(reg8[[0..15]] == 0), action=(next;) > >> ]) > >> > > > > _______________________________________________ > > dev mailing list > > [email protected] > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > -- _‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’._ * **‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.* _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
