On Thu, Jul 29, 2021 at 4:03 AM Han Zhou <[email protected]> wrote:
>
> From: Ankur Sharma <[email protected]>
>
> By default, OVN support only one DGP (distributed gateway port) per
> logical router. While a single DGP port suffices for most of the North
> South connectivity, there are requirements where a logical router could
> be connected to multiple external networks and based on routing decision
> packet could go to different ones.
>
> This patch adds flexibility of having multiple DGPs per logical router.
>
> Changes can classified as following:
> a. Data structure changes to allow multiple DGPs per ovn_datapath.
>
> b. Consumption of new data structure in logical flows for
> individual features.
>
> c. Features that require changes are:
> i. Regular NS traffic flow.
> ii. Network Address Translation.
> iii. Load Balancer
> iv. Gateway_mtu.
> v. reside-on-redirect-chassis
> vi. Misc code sections that assumed a single DGP.
>
> d. Except for reside-on-redirect-chassis all the other features
> could be extended to multiple DGPs. Reside on redirect
> chassis with its current specification could not be extended
> and hence should be used only with the logical router that
> has a single DGP.
>
> This patch doesn't support NAT & load-balancer features for multiple
> DGPs yet, but added validations that disables NAT/load-balancer
> features when there are more than one DGP configured per router.
>
> Signed-off-by: Ankur Sharma <[email protected]>
> Co-authored-by: Dhathri Purohith <[email protected]>
> Signed-off-by: Dhathri Purohith <[email protected]>
> Co-authored-by: Abhiram Sangana <[email protected]>
> Signed-off-by: Abhiram Sangana <[email protected]>
> Co-authored-by: Han Zhou <[email protected]>
> Signed-off-by: Han Zhou <[email protected]>
Hi Han, Ankur et all.
Thanks for adding this feature.
Please see below for few comments.
Also this patch needs a rebase.
Thanks
Numan
> ---
> NEWS | 3 +
> northd/lrouter.dl | 103 +++-----
> northd/ovn-northd.8.xml | 6 +-
> northd/ovn-northd.c | 535 ++++++++++++++++++++++------------------
> northd/ovn_northd.dl | 178 +++++++------
> ovn-architecture.7.xml | 19 +-
> ovn-nb.xml | 27 +-
> ovs | 2 +-
> tests/ovn-northd.at | 92 +++++++
> tests/ovn.at | 307 +++++++++++++++++++++++
> 10 files changed, 870 insertions(+), 402 deletions(-)
>
> diff --git a/NEWS b/NEWS
> index eefdae9bc..2a1c56b2d 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -33,6 +33,9 @@ OVN v21.06.0 - 18 Jun 2021
> "ovn-trim-limit-lflow-cache" and "ovn-trim-wmark-perc-lflow-cache", to
> allow enforcing a lflow cache size limit and high watermark percentage
> for which automatic memory trimming is performed.
> + - Support multiple distributed gateway ports on a single logical router.
> + (NAT and load-balancer are not supported yet when there are multiple
> + distributed gateway ports).
>
> OVN v21.03.0 - 12 Mar 2021
> -------------------------
> diff --git a/northd/lrouter.dl b/northd/lrouter.dl
> index 4a24f3f61..d37350ab8 100644
> --- a/northd/lrouter.dl
> +++ b/northd/lrouter.dl
> @@ -138,14 +138,14 @@ Warning[message] :-
> var message = "Bad configuration: distributed gateway port configured on
> "
> "port ${lrp.name} on L3 gateway router".
>
> -/* DistributedGatewayPortCandidate.
> +/* Distributed gateway ports.
> *
> - * Each row pairs a logical router with its distributed gateway port,
> - * but without checking that there is at most one DGP per LR.
> + * Each row means 'lrp' is a distributed gateway port on 'lr_uuid'.
> *
> - * (Use DistributedGatewayPort instead, since it guarantees uniqueness.) */
> -relation DistributedGatewayPortCandidate(lr_uuid: uuid, lrp_uuid: uuid)
> -DistributedGatewayPortCandidate(lr_uuid, lrp_uuid) :-
> + * A logical router can have multiple distributed gateway ports. */
> +relation DistributedGatewayPort(lrp: Intern<nb::Logical_Router_Port>,
> + lr_uuid: uuid)
> +DistributedGatewayPort(lrp, lr_uuid) :-
> lr in nb::Logical_Router(._uuid = lr_uuid),
> LogicalRouterPort(lrp_uuid, lr._uuid),
> lrp in &nb::Logical_Router_Port(._uuid = lrp_uuid),
> @@ -153,30 +153,10 @@ DistributedGatewayPortCandidate(lr_uuid, lrp_uuid) :-
> var has_hcg = lrp.ha_chassis_group.is_some(),
> var has_gc = not lrp.gateway_chassis.is_empty(),
> has_hcg or has_gc.
> -Warning[message] :-
> - DistributedGatewayPortCandidate(lr_uuid, lrp_uuid),
> - var lrps = lrp_uuid.group_by(lr_uuid).to_set(),
> - lrps.size() > 1,
> - lr in nb::Logical_Router(._uuid = lr_uuid),
> - var message = "Bad configuration: multiple distributed gateway ports on "
> - "logical router ${lr.name}; ignoring all of them".
> -
> -/* Distributed gateway ports.
> - *
> - * Each row means 'lrp' is the distributed gateway port on 'lr_uuid'.
> - *
> - * There is at most one distributed gateway port per logical router. */
> -relation DistributedGatewayPort(lrp: Intern<nb::Logical_Router_Port>,
> lr_uuid: uuid)
> -DistributedGatewayPort(lrp, lr_uuid) :-
> - DistributedGatewayPortCandidate(lr_uuid, lrp_uuid),
> - var lrps = lrp_uuid.group_by(lr_uuid).to_set(),
> - lrps.size() == 1,
> - Some{var lrp_uuid} = lrps.nth(0),
> - lrp in &nb::Logical_Router_Port(._uuid = lrp_uuid).
>
> /* HAChassis is an abstraction over nb::Gateway_Chassis and nb::HA_Chassis,
> which
> * are different ways to represent the same configuration. Each row is
> - * effectively one HA_Chassis record. (Usually, we could associated each
> + * effectively one HA_Chassis record. (Usually, we could associate each
> * row with a particular 'lr_uuid', but it's permissible for more than one
> * logical router to use a HA chassis group, so we omit it so that multiple
> * references get merged.)
> @@ -236,18 +216,20 @@ HAChassisGroup(ha_chassis_group_uuid(hac_group_uuid),
> .name = name,
> .external_ids = external_ids).
>
> -/* Each row maps from a logical router to the name of its HAChassisGroup.
> - * This level of indirection is needed because multiple logical routers
> - * are allowed to reference a given HAChassisGroup. */
> -relation LogicalRouterHAChassisGroup(lr_uuid: uuid,
> - hacg_uuid: uuid)
> -LogicalRouterHAChassisGroup(lr_uuid, ha_chassis_group_uuid(lrp._uuid)) :-
> - DistributedGatewayPort(lrp, lr_uuid),
> +/* Each row maps from a distributed gateway logical router port to the name
> of
> + * its HAChassisGroup.
> + * This level of indirection is needed because multiple distributed gateway
> + * logical router ports are allowed to reference a given HAChassisGroup. */
> +relation DistributedGatewayPortHAChassisGroup(
> + lrp: Intern<nb::Logical_Router_Port>,
> + hacg_uuid: uuid)
> +DistributedGatewayPortHAChassisGroup(lrp, ha_chassis_group_uuid(lrp._uuid))
> :-
> + DistributedGatewayPort(.lrp = lrp),
> lrp.ha_chassis_group == None,
> lrp.gateway_chassis.size() > 0.
> -LogicalRouterHAChassisGroup(lr_uuid,
> - ha_chassis_group_uuid(hac_group_uuid)) :-
> - DistributedGatewayPort(lrp, lr_uuid),
> +DistributedGatewayPortHAChassisGroup(lrp,
> + ha_chassis_group_uuid(hac_group_uuid))
> :-
> + DistributedGatewayPort(.lrp = lrp),
> Some{var hac_group_uuid} = lrp.ha_chassis_group,
> nb::HA_Chassis_Group(._uuid = hac_group_uuid).
>
> @@ -259,14 +241,19 @@ RouterPortIsRedirect(lrp, false) :-
> &nb::Logical_Router_Port(._uuid = lrp),
> not DistributedGatewayPort(&nb::Logical_Router_Port{._uuid = lrp}, _).
>
> -relation LogicalRouterRedirectPort(lr: uuid, has_redirect_port:
> Option<Intern<nb::Logical_Router_Port>>)
> -
> -LogicalRouterRedirectPort(lr, Some{lrp}) :-
> - DistributedGatewayPort(lrp, lr).
> -
> -LogicalRouterRedirectPort(lr, None) :-
> - nb::Logical_Router(._uuid = lr),
> - not DistributedGatewayPort(_, lr).
> +/*
> + * LogicalRouterDGWPorts maps from each logical router UUID
> + * to the logical router's set of distributed gateway (or redirect) ports. */
> +relation LogicalRouterDGWPorts(
> + lr_uuid: uuid,
> + l3dgw_ports: Vec<Intern<nb::Logical_Router_Port>>)
> +LogicalRouterDGWPorts(lr_uuid, l3dgw_ports) :-
> + DistributedGatewayPort(lrp, lr_uuid),
> + var l3dgw_ports = lrp.group_by(lr_uuid).to_vec().
> +LogicalRouterDGWPorts(lr_uuid, vec_empty()) :-
> + lr in nb::Logical_Router(),
> + var lr_uuid = lr._uuid,
> + not DistributedGatewayPort(_, lr_uuid).
>
> typedef ExceptionalExtIps = AllowedExtIps{ips: Intern<nb::Address_Set>}
> | ExemptedExtIps{ips: Intern<nb::Address_Set>}
> @@ -450,9 +437,7 @@ LogicalRouterCopp0(lr, meters) :-
>
> /* Router relation collects all attributes of a logical router.
> *
> - * `l3dgw_port` - optional redirect port (see `DistributedGatewayPort`)
> - * `redirect_port_name` - derived redirect port name (or empty string if
> - * router does not have a redirect port)
> + * `l3dgw_ports` - optional redirect ports (see `DistributedGatewayPort`)
> * `is_gateway` - true iff the router is a gateway router. Together with
> * `l3dgw_port`, this flag affects the generation of various flows
> * related to NAT and load balancing.
> @@ -474,8 +459,7 @@ typedef Router = Router {
> external_ids: Map<string,string>,
>
> /* Additional computed fields. */
> - l3dgw_port: Option<Intern<nb::Logical_Router_Port>>,
> - redirect_port_name: string,
> + l3dgw_ports: Vec<Intern<nb::Logical_Router_Port>>,
> is_gateway: bool,
> nats: Vec<NAT>,
> snat_ips: Map<v46_ip, Set<NAT>>,
> @@ -498,23 +482,18 @@ Router[Router{
> .options = lr.options,
> .external_ids = lr.external_ids,
>
> - .l3dgw_port = l3dgw_port,
> - .redirect_port_name =
> - match (l3dgw_port) {
> - Some{rport} ->
> json_string_escape(chassis_redirect_name(rport.name)),
> - _ -> ""
> - },
> - .is_gateway = lr.options.contains_key("chassis"),
> - .nats = nats,
> - .snat_ips = snat_ips,
> - .lbs = lbs,
> - .mcast_cfg = mcast_cfg,
> + .l3dgw_ports = l3dgw_ports,
> + .is_gateway = lr.options.contains_key("chassis"),
> + .nats = nats,
> + .snat_ips = snat_ips,
> + .lbs = lbs,
> + .mcast_cfg = mcast_cfg,
> .learn_from_arp_request = learn_from_arp_request,
> .force_lb_snat = force_lb_snat,
> .copp = copp}.intern()] :-
> lr in nb::Logical_Router(),
> lr.is_enabled(),
> - LogicalRouterRedirectPort(lr._uuid, l3dgw_port),
> + LogicalRouterDGWPorts(lr._uuid, l3dgw_ports),
> LogicalRouterNATs(lr._uuid, nats),
> LogicalRouterLBs(lr._uuid, lbs),
> LogicalRouterSnatIPs(lr._uuid, snat_ips),
> diff --git a/northd/ovn-northd.8.xml b/northd/ovn-northd.8.xml
> index 99a19f853..8a368ef61 100644
> --- a/northd/ovn-northd.8.xml
> +++ b/northd/ovn-northd.8.xml
> @@ -3764,10 +3764,10 @@ icmp6 {
> <h3>Ingress Table 17: Gateway Redirect</h3>
>
> <p>
> - For distributed logical routers where one of the logical router
> + For distributed logical routers where one or more of the logical router
> ports specifies a gateway chassis, this table redirects
> - certain packets to the distributed gateway port instance on the
> - gateway chassis. This table has the following flows:
> + certain packets to the distributed gateway port instances on the
> + gateway chassises. This table has the following flows:
> </p>
>
> <ul>
> diff --git a/northd/ovn-northd.c b/northd/ovn-northd.c
> index ebe12cace..87c4478fa 100644
> --- a/northd/ovn-northd.c
> +++ b/northd/ovn-northd.c
> @@ -655,13 +655,12 @@ struct ovn_datapath {
> bool is_gw_router;
>
> /* OVN northd only needs to know about the logical router gateway port
> for
> - * NAT on a distributed router. This "distributed gateway port" is
> - * populated only when there is a gateway chassis specified for one of
> - * the ports on the logical router. Otherwise this will be NULL. */
> - struct ovn_port *l3dgw_port;
> - /* The "derived" OVN port representing the instance of l3dgw_port on
> - * the gateway chassis. */
> - struct ovn_port *l3redirect_port;
> + * NAT on a distributed router. The "distributed gateway ports" are
> + * populated only when there is a gateway chassis or ha chassis group
> + * specified for some of the ports on the logical router. Otherwise this
> + * will be NULL. */
> + struct ovn_port **l3dgw_ports;
> + size_t n_l3dgw_ports;
>
> /* NAT entries configured on the router. */
> struct ovn_nat *nat_entries;
> @@ -802,6 +801,16 @@ init_nat_entries(struct ovn_datapath *od)
> return;
> }
>
> + if (od->n_l3dgw_ports > 1) {
> + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1);
> + VLOG_WARN_RL(&rl, "NAT is configured on logical router %s, which has
> %"
> + PRIuSIZE" distributed gateway ports. NAT is not
> supported"
> + " yet when there is more than one distributed gateway "
> + "port on the router.",
> + od->nbr->name, od->n_l3dgw_ports);
> + return;
> + }
> +
> od->nat_entries = xmalloc(od->nbr->n_nat * sizeof *od->nat_entries);
>
> for (size_t i = 0; i < od->nbr->n_nat; i++) {
> @@ -941,6 +950,7 @@ ovn_datapath_destroy(struct hmap *datapaths, struct
> ovn_datapath *od)
> destroy_lb_ips(od);
> free(od->nat_entries);
> free(od->localnet_ports);
> + free(od->l3dgw_ports);
> ovn_ls_port_group_destroy(&od->nb_pgs);
> destroy_mcast_info_for_datapath(od);
>
> @@ -1489,9 +1499,18 @@ struct ovn_port {
> /* Logical port multicast data. */
> struct mcast_port_info mcast_info;
>
> - /* This is ordinarily false. It is true if and only if this ovn_port is
> - * derived from a chassis-redirect port. */
> - bool derived;
> + /* At most one of l3dgw_port and cr_port can be not NULL. */
> +
> + /* This is set to a distributed gateway port if and only if this ovn_port
> + * is "derived" from it. Otherwise this is set to NULL. The derived
> + * ovn_port represents the instance of distributed gateway port on the
> + * gateway chassis.*/
> + struct ovn_port *l3dgw_port;
> +
> + /* This is set to the "derived" chassis-redirect port of this port if and
> + * only if this port is a distributed gateway port. Otherwise this is set
> + * to NULL. */
> + struct ovn_port *cr_port;
In my opinion, the above 2 variables - 'l3dgw_port' and 'cr_port' are
confusing.
For the code - if(op->l3dgw_port), It is not immediately obvious to
me that "op" is a derived
"chassisredirect" port.
I'd suggest to have the bool - derived to be preserved.
How about this ?
bool derived;
union {
struct ovn_port *cr_port;
struct ovn_port *l3gw_port;
};
If 'derived' is true then 'l3gw_port' would be not NULL. Otherwise it
would be NULL.
Instance of 'struct ovn_port' for distributed router port would have
"cr_port" set.
>
> bool has_unknown; /* If the addresses have 'unknown' defined. */
>
> @@ -1578,7 +1597,7 @@ ovn_port_create(struct hmap *ports, const char *key,
> op->key = xstrdup(key);
> op->sb = sb;
> ovn_port_set_nb(op, nbsp, nbrp);
> - op->derived = false;
> + op->l3dgw_port = op->cr_port = NULL;
> hmap_insert(ports, &op->key_node, hash_string(op->key, 0));
> return op;
> }
> @@ -1682,7 +1701,7 @@ lrport_is_enabled(const struct
> nbrec_logical_router_port *lrport)
> static struct ovn_port *
> ovn_port_get_peer(struct hmap *ports, struct ovn_port *op)
> {
> - if (!op->nbsp || !lsp_is_router(op->nbsp) || op->derived) {
> + if (!op->nbsp || !lsp_is_router(op->nbsp) || op->l3dgw_port) {
> return NULL;
> }
>
> @@ -2426,6 +2445,7 @@ join_logical_ports(struct northd_context *ctx,
> tag_alloc_add_existing_tags(tag_alloc_table, nbsp);
> }
> } else {
> + size_t n_allocated_l3dgw_ports = 0;
> for (size_t i = 0; i < od->nbr->n_ports; i++) {
> const struct nbrec_logical_router_port *nbrp
> = od->nbr->ports[i];
> @@ -2481,36 +2501,32 @@ join_logical_ports(struct northd_context *ctx,
> "on L3 gateway router", nbrp->name);
> continue;
> }
> - if (od->l3dgw_port || od->l3redirect_port) {
> - static struct vlog_rate_limit rl
> - = VLOG_RATE_LIMIT_INIT(1, 1);
> - VLOG_WARN_RL(&rl, "Bad configuration: multiple "
> - "distributed gateway ports on logical "
> - "router %s", od->nbr->name);
> - continue;
> - }
>
> char *redirect_name =
> ovn_chassis_redirect_name(nbrp->name);
> struct ovn_port *crp = ovn_port_find(ports,
> redirect_name);
> if (crp && crp->sb && crp->sb->datapath == od->sb) {
> - crp->derived = true;
> ovn_port_set_nb(crp, NULL, nbrp);
> ovs_list_remove(&crp->list);
> ovs_list_push_back(both, &crp->list);
> } else {
> crp = ovn_port_create(ports, redirect_name,
> NULL, nbrp, NULL);
> - crp->derived = true;
> ovs_list_push_back(nb_only, &crp->list);
> }
> + crp->l3dgw_port = op;
> + op->cr_port = crp;
> crp->od = od;
> free(redirect_name);
>
> - /* Set l3dgw_port and l3redirect_port in od, for later
> - * use during flow creation. */
> - od->l3dgw_port = op;
> - od->l3redirect_port = crp;
> + /* Add to l3dgw_ports in od, for later use during flow
> + * creation. */
> + if (od->n_l3dgw_ports == n_allocated_l3dgw_ports) {
> + od->l3dgw_ports = x2nrealloc(od->l3dgw_ports,
> +
> &n_allocated_l3dgw_ports,
> + sizeof
> *od->l3dgw_ports);
> + }
> + od->l3dgw_ports[od->n_l3dgw_ports++] = op;
>
> assign_routable_addresses(op);
> }
> @@ -2522,7 +2538,7 @@ join_logical_ports(struct northd_context *ctx,
> * to their peers. */
> struct ovn_port *op;
> HMAP_FOR_EACH (op, key_node, ports) {
> - if (op->nbsp && lsp_is_router(op->nbsp) && !op->derived) {
> + if (op->nbsp && lsp_is_router(op->nbsp) && !op->l3dgw_port) {
> struct ovn_port *peer = ovn_port_get_peer(ports, op);
> if (!peer || !peer->nbrp) {
> continue;
> @@ -2553,7 +2569,7 @@ join_logical_ports(struct northd_context *ctx,
> if (peer->od && peer->od->mcast_info.rtr.relay) {
> op->od->mcast_info.sw.flood_relay = true;
> }
> - } else if (op->nbrp && op->nbrp->peer && !op->derived) {
> + } else if (op->nbrp && op->nbrp->peer && !op->l3dgw_port) {
> struct ovn_port *peer = ovn_port_find(ports, op->nbrp->peer);
> if (peer) {
> if (peer->nbrp) {
> @@ -2598,7 +2614,8 @@ get_nat_addresses(const struct ovn_port *op, size_t *n,
> bool routable_only)
> struct eth_addr mac;
> if (!op || !op->nbrp || !op->od || !op->od->nbr
> || (!op->od->nbr->n_nat && !op->od->nbr->n_load_balancer)
> - || !eth_addr_from_string(op->nbrp->mac, &mac)) {
> + || !eth_addr_from_string(op->nbrp->mac, &mac)
> + || op->od->n_l3dgw_ports > 1) {
> *n = n_nats;
> return NULL;
> }
> @@ -2629,7 +2646,7 @@ get_nat_addresses(const struct ovn_port *op, size_t *n,
> bool routable_only)
>
> /* Determine whether this NAT rule satisfies the conditions for
> * distributed NAT processing. */
> - if (op->od->l3redirect_port && !strcmp(nat->type, "dnat_and_snat")
> + if (op->od->l3dgw_ports && !strcmp(nat->type, "dnat_and_snat")
small nit: I'd suggest to use - 'op->od->n_l3dgw_ports' in the above if.
And also suggest to use in other places.
> && nat->logical_port && nat->external_mac) {
> /* Distributed NAT rule. */
> if (eth_addr_from_string(nat->external_mac, &mac)) {
> @@ -2695,9 +2712,9 @@ get_nat_addresses(const struct ovn_port *op, size_t *n,
> bool routable_only)
> if (central_ip_address) {
> /* Gratuitous ARP for centralized NAT rules on distributed gateway
> * ports should be restricted to the gateway chassis. */
> - if (op->od->l3redirect_port) {
> + if (op->od->l3dgw_ports) {
> ds_put_format(&c_addresses, " is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + op->od->l3dgw_ports[0]->cr_port->json_key);
> }
>
> addresses[n_nats++] = ds_steal_cstr(&c_addresses);
> @@ -3010,7 +3027,7 @@ ovn_port_update_sbrec(struct northd_context *ctx,
> /* If the router is for l3 gateway, it resides on a chassis
> * and its port type is "l3gateway". */
> const char *chassis_name = smap_get(&op->od->nbr->options,
> "chassis");
> - if (op->derived) {
> + if (op->l3dgw_port) {
> sbrec_port_binding_set_type(op->sb, "chassisredirect");
> } else if (chassis_name) {
> sbrec_port_binding_set_type(op->sb, "l3gateway");
> @@ -3020,7 +3037,7 @@ ovn_port_update_sbrec(struct northd_context *ctx,
>
> struct smap new;
> smap_init(&new);
> - if (op->derived) {
> + if (op->l3dgw_port) {
> const char *redirect_type = smap_get(&op->nbrp->options,
> "redirect-type");
>
> @@ -3200,7 +3217,7 @@ ovn_port_update_sbrec(struct northd_context *ctx,
> char **nats = NULL;
> if (nat_addresses && !strcmp(nat_addresses, "router")) {
> if (op->peer && op->peer->od
> - && (chassis || op->peer->od->l3redirect_port)) {
> + && (chassis || op->peer->od->l3dgw_ports)) {
> nats = get_nat_addresses(op->peer, &n_nats, false);
> }
> /* Only accept manual specification of ethernet address
> @@ -3236,12 +3253,26 @@ ovn_port_update_sbrec(struct northd_context *ctx,
> * sending the GARPs for the router port IPs.
> * */
> bool add_router_port_garp = false;
> - if (op->peer && op->peer->nbrp && op->peer->od->l3dgw_port &&
> - op->peer->od->l3redirect_port &&
> - (smap_get_bool(&op->peer->nbrp->options,
> - "reside-on-redirect-chassis", false) ||
> - op->peer == op->peer->od->l3dgw_port)) {
> - add_router_port_garp = true;
> + if (op->peer && op->peer->nbrp && op->peer->od->l3dgw_ports) {
> + if (op->peer->cr_port) {
> + add_router_port_garp = true;
> + } else if (smap_get_bool(&op->peer->nbrp->options,
> + "reside-on-redirect-chassis", false)) {
> + if (op->peer->od->n_l3dgw_ports == 1) {
> + add_router_port_garp = true;
> + } else {
> + static struct vlog_rate_limit rl =
> + VLOG_RATE_LIMIT_INIT(1, 1);
> + VLOG_WARN_RL(&rl, "\"reside-on-redirect-chassis\" is
> "
> + "set on logical router port %s, which "
> + "is on logical router %s, which has %"
> + PRIuSIZE" distributed gateway ports.
> This"
> + "option can only be used when there is "
> + "a single distributed gateway port.",
> + op->peer->key, op->peer->od->nbr->name,
> + op->peer->od->n_l3dgw_ports);
> + }
> + }
> } else if (chassis && op->od->n_localnet_ports) {
> add_router_port_garp = true;
> }
> @@ -3256,9 +3287,10 @@ ovn_port_update_sbrec(struct northd_context *ctx,
>
> op->peer->lrp_networks.ipv4_addrs[i].addr_s);
> }
>
> - if (op->peer->od->l3redirect_port) {
> + if (op->peer->od->l3dgw_ports) {
> ds_put_format(&garp_info, " is_chassis_resident(%s)",
> - op->peer->od->l3redirect_port->json_key);
> + op->peer->od->l3dgw_ports[0]
> + ->cr_port->json_key);
> }
>
> n_nats++;
> @@ -3531,7 +3563,17 @@ build_ovn_lr_lbs(struct hmap *datapaths, struct hmap
> *lbs)
> if (!od->nbr) {
> continue;
> }
> - if (!smap_get(&od->nbr->options, "chassis") && !od->l3dgw_port) {
> + if (!smap_get(&od->nbr->options, "chassis")
> + && od->n_l3dgw_ports != 1) {
> + if (od->n_l3dgw_ports > 1 && od->nbr->n_load_balancer) {
> + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1,
> 1);
> + VLOG_WARN_RL(&rl, "Load-balancers are configured on logical "
> + "router %s, which has %"PRIuSIZE" distributed "
> + "gateway ports. Load-balancer is not supported "
> + "yet when there is more than one distributed "
> + "gateway port on the router.",
> + od->nbr->name, od->n_l3dgw_ports);
> + }
> continue;
> }
>
> @@ -6440,13 +6482,16 @@ build_lrouter_groups__(struct hmap *ports, struct
> ovn_datapath *od)
> {
> ovs_assert((od && od->nbr && od->lr_group));
>
> - if (od->l3dgw_port && od->l3redirect_port) {
> - /* It's a logical router with gateway port. If it
> + if (od->l3dgw_ports) {
This "if" check can be dropped as its not necessary.
> + /* It's a logical router with gateway ports. If it
> * has HA_Chassis_Group associated to it in SB DB, then store the
> * ha chassis group name. */
> - if (od->l3redirect_port->sb->ha_chassis_group) {
> - sset_add(&od->lr_group->ha_chassis_groups,
> - od->l3redirect_port->sb->ha_chassis_group->name);
> + for (size_t i = 0; i < od->n_l3dgw_ports; i++) {
> + struct ovn_port *crp = od->l3dgw_ports[i]->cr_port;
> + if (crp->sb->ha_chassis_group) {
> + sset_add(&od->lr_group->ha_chassis_groups,
> + crp->sb->ha_chassis_group->name);
> + }
> }
> }
>
> @@ -7833,16 +7878,17 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op,
> ds_clear(match);
> ds_put_format(match, "eth.dst == "ETH_ADDR_FMT,
> ETH_ADDR_ARGS(mac));
> - if (op->peer->od->l3dgw_port
> - && op->peer->od->l3redirect_port
> + if (op->peer->od->l3dgw_ports
> && op->od->n_localnet_ports) {
> bool add_chassis_resident_check = false;
> - if (op->peer == op->peer->od->l3dgw_port) {
> + const char *json_key;
> + if (op->peer->cr_port) {
> /* The peer of this port represents a distributed
> * gateway port. The destination lookup flow for the
> * router's distributed gateway port MAC address
> should
> * only be programmed on the gateway chassis. */
> add_chassis_resident_check = true;
> + json_key = op->peer->cr_port->json_key;
> } else {
> /* Check if the option 'reside-on-redirect-chassis'
> * is set to true on the peer port. If set to true
> @@ -7853,12 +7899,15 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op,
> */
> add_chassis_resident_check = smap_get_bool(
> &op->peer->nbrp->options,
> - "reside-on-redirect-chassis", false);
> + "reside-on-redirect-chassis", false) &&
> + op->peer->od->n_l3dgw_ports == 1;
> + json_key =
> + op->peer->od->l3dgw_ports[0]->cr_port->json_key;
> }
>
> if (add_chassis_resident_check) {
> ds_put_format(match, " && is_chassis_resident(%s)",
> -
> op->peer->od->l3redirect_port->json_key);
> + json_key);
> }
> }
>
> @@ -7871,8 +7920,7 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op,
>
> /* Add ethernet addresses specified in NAT rules on
> * distributed logical routers. */
> - if (op->peer->od->l3dgw_port
> - && op->peer == op->peer->od->l3dgw_port) {
> + if (op->peer->cr_port) {
> for (int j = 0; j < op->peer->od->nbr->n_nat; j++) {
> const struct nbrec_nat *nat
> =
> op->peer->od->nbr->nat[j];
> @@ -9139,14 +9187,14 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip
> *lb_vip,
> &lb->nlb->header_);
> }
>
> - if (od->l3redirect_port &&
> + if (od->l3dgw_ports &&
> (lb_vip->n_backends || !lb_vip->empty_backend_rej)) {
> new_match_p = xasprintf("%s && is_chassis_resident(%s)",
> new_match,
> - od->l3redirect_port->json_key);
> + od->l3dgw_ports[0]->cr_port->json_key);
> est_match_p = xasprintf("%s && is_chassis_resident(%s)",
> est_match,
> - od->l3redirect_port->json_key);
> + od->l3dgw_ports[0]->cr_port->json_key);
> }
>
> if (snat_type == NO_FORCE_SNAT &&
> @@ -9191,15 +9239,15 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip
> *lb_vip,
> free(est_match_p);
> }
>
> - if (!od->l3dgw_port || !od->l3redirect_port || !lb_vip->n_backends) {
> + if (!od->l3dgw_ports || !lb_vip->n_backends) {
> goto next;
> }
>
> - char *undnat_match_p = xasprintf("%s) && outport == %s && "
> - "is_chassis_resident(%s)",
> - ds_cstr(&undnat_match),
> - od->l3dgw_port->json_key,
> - od->l3redirect_port->json_key);
> + char *undnat_match_p = xasprintf(
> + "%s) && outport == %s && is_chassis_resident(%s)",
> + ds_cstr(&undnat_match),
> + od->l3dgw_ports[0]->json_key,
> + od->l3dgw_ports[0]->cr_port->json_key);
> if (snat_type == SKIP_SNAT) {
> ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> undnat_match_p, skip_snat_est_action,
> @@ -9695,9 +9743,9 @@ build_lrouter_port_nat_arp_nd_flow(struct ovn_port *op,
> * upstream MAC learning points to the gateway chassis.
> * Also need to avoid generation of multiple ARP responses
> * from different chassis. */
> - if (op->od->l3redirect_port) {
> + if (op->od->l3dgw_ports) {
> ds_put_format(&match, "is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + op->od->l3dgw_ports[0]->cr_port->json_key);
> }
> }
>
> @@ -9968,7 +10016,7 @@ build_adm_ctrl_flows_for_lrouter_port(
> return;
> }
>
> - if (op->derived) {
> + if (op->l3dgw_port) {
> /* No ingress packets should be received on a chassisredirect
> * port. */
> return;
> @@ -9991,12 +10039,11 @@ build_adm_ctrl_flows_for_lrouter_port(
> ds_clear(match);
> ds_put_format(match, "eth.dst == %s && inport == %s",
> op->lrp_networks.ea_s, op->json_key);
> - if (op->od->l3dgw_port && op == op->od->l3dgw_port
> - && op->od->l3redirect_port) {
> + if (op->cr_port) {
> /* Traffic with eth.dst = l3dgw_port->lrp_networks.ea_s
> * should only be received on the gateway chassis. */
> ds_put_format(match, " && is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + op->cr_port->json_key);
> }
> ovn_lflow_add_with_hint(lflows, op->od, S_ROUTER_IN_ADMISSION, 50,
> ds_cstr(match), ds_cstr(actions),
> @@ -10135,10 +10182,9 @@ build_neigh_learning_flows_for_lrouter_port(
> op->lrp_networks.ipv4_addrs[i].network_s,
> op->lrp_networks.ipv4_addrs[i].plen,
> op->lrp_networks.ipv4_addrs[i].addr_s);
> - if (op->od->l3dgw_port && op == op->od->l3dgw_port
> - && op->od->l3redirect_port) {
> + if (op->cr_port) {
> ds_put_format(match, " && is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + op->cr_port->json_key);
> }
> const char *actions_s = REGBIT_LOOKUP_NEIGHBOR_RESULT
> " = lookup_arp(inport, arp.spa, arp.sha); "
> @@ -10155,10 +10201,9 @@ build_neigh_learning_flows_for_lrouter_port(
> op->json_key,
> op->lrp_networks.ipv4_addrs[i].network_s,
> op->lrp_networks.ipv4_addrs[i].plen);
> - if (op->od->l3dgw_port && op == op->od->l3dgw_port
> - && op->od->l3redirect_port) {
> + if (op->cr_port) {
> ds_put_format(match, " && is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + op->cr_port->json_key);
> }
> ds_clear(actions);
> ds_put_format(actions, REGBIT_LOOKUP_NEIGHBOR_RESULT
> @@ -10643,7 +10688,7 @@ build_arp_resolve_flows_for_lrouter_port(
> }
> }
>
> - if (!op->derived && op->od->l3redirect_port) {
> + if (op->cr_port) {
> const char *redirect_type = smap_get(&op->nbrp->options,
> "redirect-type");
> if (redirect_type && !strcasecmp(redirect_type, "bridged")) {
> @@ -10656,7 +10701,7 @@ build_arp_resolve_flows_for_lrouter_port(
> ds_clear(match);
> ds_put_format(match, "outport == %s && "
> "!is_chassis_resident(%s)", op->json_key,
> - op->od->l3redirect_port->json_key);
> + op->cr_port->json_key);
> ds_clear(actions);
> ds_put_format(actions, "eth.dst = %s; next;",
> op->lrp_networks.ea_s);
> @@ -10904,8 +10949,8 @@ build_arp_resolve_flows_for_lrouter_port(
> &op->nbsp->header_);
> }
>
> - if (smap_get(&peer->od->nbr->options, "chassis") ||
> - (peer->od->l3dgw_port && peer == peer->od->l3dgw_port)) {
> + if (smap_get(&peer->od->nbr->options, "chassis")
> + || peer->cr_port) {
> routable_addresses_to_lflows(lflows, router_port, peer,
> match, actions);
> }
> @@ -10934,110 +10979,111 @@ build_check_pkt_len_flows_for_lrouter(
> struct ds *match, struct ds *actions,
> struct shash *meter_groups)
> {
> - if (od->nbr) {
> -
> - /* Packets are allowed by default. */
> - ovn_lflow_add(lflows, od, S_ROUTER_IN_CHK_PKT_LEN, 0, "1",
> - "next;");
> - ovn_lflow_add(lflows, od, S_ROUTER_IN_LARGER_PKTS, 0, "1",
> - "next;");
> + if (!od->nbr) {
> + return;
> + }
>
> - if (od->l3dgw_port && od->l3redirect_port) {
> - int gw_mtu = 0;
> - if (od->l3dgw_port->nbrp) {
> - gw_mtu = smap_get_int(&od->l3dgw_port->nbrp->options,
> - "gateway_mtu", 0);
> - }
> - /* Add the flows only if gateway_mtu is configured. */
> - if (gw_mtu <= 0) {
> - return;
> - }
> + /* Packets are allowed by default. */
> + ovn_lflow_add(lflows, od, S_ROUTER_IN_CHK_PKT_LEN, 0, "1",
> + "next;");
> + ovn_lflow_add(lflows, od, S_ROUTER_IN_LARGER_PKTS, 0, "1",
> + "next;");
> + for (size_t dgp = 0; dgp < od->n_l3dgw_ports; dgp++) {
> + int gw_mtu = 0;
> + if (od->l3dgw_ports[dgp]->nbrp) {
> + gw_mtu = smap_get_int(&od->l3dgw_ports[dgp]->nbrp->options,
> + "gateway_mtu", 0);
> + }
> + /* Add the flows only if gateway_mtu is configured. */
> + if (gw_mtu <= 0) {
> + continue;
> + }
>
> - ds_clear(match);
> - ds_put_format(match, "outport == %s", od->l3dgw_port->json_key);
> + ds_clear(match);
> + ds_put_format(match, "outport == %s",
> + od->l3dgw_ports[dgp]->json_key);
>
> - ds_clear(actions);
> - ds_put_format(actions,
> - REGBIT_PKT_LARGER" = check_pkt_larger(%d);"
> - " next;", gw_mtu + VLAN_ETH_HEADER_LEN);
> - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_CHK_PKT_LEN, 50,
> - ds_cstr(match), ds_cstr(actions),
> - &od->l3dgw_port->nbrp->header_);
> + ds_clear(actions);
> + ds_put_format(actions,
> + REGBIT_PKT_LARGER" = check_pkt_larger(%d);"
> + " next;", gw_mtu + VLAN_ETH_HEADER_LEN);
> + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_CHK_PKT_LEN, 50,
> + ds_cstr(match), ds_cstr(actions),
> + &od->l3dgw_ports[dgp]->nbrp->header_);
>
> - for (size_t i = 0; i < od->nbr->n_ports; i++) {
> - struct ovn_port *rp = ovn_port_find(ports,
> - od->nbr->ports[i]->name);
> - if (!rp || rp == od->l3dgw_port) {
> - continue;
> - }
> + for (size_t i = 0; i < od->nbr->n_ports; i++) {
> + struct ovn_port *rp = ovn_port_find(ports,
> + od->nbr->ports[i]->name);
> + if (!rp || rp->cr_port) {
> + continue;
> + }
>
> - if (rp->lrp_networks.ipv4_addrs) {
> - ds_clear(match);
> - ds_put_format(match, "inport == %s && outport == %s"
> - " && ip4 && "REGBIT_PKT_LARGER,
> - rp->json_key, od->l3dgw_port->json_key);
> + if (rp->lrp_networks.ipv4_addrs) {
> + ds_clear(match);
> + ds_put_format(match, "inport == %s && outport == %s"
> + " && ip4 && "REGBIT_PKT_LARGER,
> + rp->json_key, od->l3dgw_ports[dgp]->json_key);
>
> - ds_clear(actions);
> - /* Set icmp4.frag_mtu to gw_mtu */
> - ds_put_format(actions,
> - "icmp4_error {"
> - REGBIT_EGRESS_LOOPBACK" = 1; "
> - "eth.dst = %s; "
> - "ip4.dst = ip4.src; "
> - "ip4.src = %s; "
> - "ip.ttl = 255; "
> - "icmp4.type = 3; /* Destination Unreachable. */ "
> - "icmp4.code = 4; /* Frag Needed and DF was Set. */ "
> - "icmp4.frag_mtu = %d; "
> - "next(pipeline=ingress, table=%d); };",
> - rp->lrp_networks.ea_s,
> - rp->lrp_networks.ipv4_addrs[0].addr_s,
> - gw_mtu,
> - ovn_stage_get_table(S_ROUTER_IN_ADMISSION));
> - ovn_lflow_add_with_hint__(lflows, od,
> - S_ROUTER_IN_LARGER_PKTS, 50,
> - ds_cstr(match),
> ds_cstr(actions),
> - NULL,
> - copp_meter_get(
> - COPP_ICMP4_ERR,
> - rp->od->nbr->copp,
> - meter_groups),
> - &rp->nbrp->header_);
> - }
> + ds_clear(actions);
> + /* Set icmp4.frag_mtu to gw_mtu */
> + ds_put_format(actions,
> + "icmp4_error {"
> + REGBIT_EGRESS_LOOPBACK" = 1; "
> + "eth.dst = %s; "
> + "ip4.dst = ip4.src; "
> + "ip4.src = %s; "
> + "ip.ttl = 255; "
> + "icmp4.type = 3; /* Destination Unreachable. */ "
> + "icmp4.code = 4; /* Frag Needed and DF was Set. */ "
> + "icmp4.frag_mtu = %d; "
> + "next(pipeline=ingress, table=%d); };",
> + rp->lrp_networks.ea_s,
> + rp->lrp_networks.ipv4_addrs[0].addr_s,
> + gw_mtu,
> + ovn_stage_get_table(S_ROUTER_IN_ADMISSION));
> + ovn_lflow_add_with_hint__(lflows, od,
> + S_ROUTER_IN_LARGER_PKTS, 50,
> + ds_cstr(match), ds_cstr(actions),
> + NULL,
> + copp_meter_get(
> + COPP_ICMP4_ERR,
> + rp->od->nbr->copp,
> + meter_groups),
> + &rp->nbrp->header_);
> + }
>
> - if (rp->lrp_networks.ipv6_addrs) {
> - ds_clear(match);
> - ds_put_format(match, "inport == %s && outport == %s"
> - " && ip6 && "REGBIT_PKT_LARGER,
> - rp->json_key, od->l3dgw_port->json_key);
> + if (rp->lrp_networks.ipv6_addrs) {
> + ds_clear(match);
> + ds_put_format(match, "inport == %s && outport == %s"
> + " && ip6 && "REGBIT_PKT_LARGER,
> + rp->json_key, od->l3dgw_ports[dgp]->json_key);
>
> - ds_clear(actions);
> - /* Set icmp6.frag_mtu to gw_mtu */
> - ds_put_format(actions,
> - "icmp6_error {"
> - REGBIT_EGRESS_LOOPBACK" = 1; "
> - "eth.dst = %s; "
> - "ip6.dst = ip6.src; "
> - "ip6.src = %s; "
> - "ip.ttl = 255; "
> - "icmp6.type = 2; /* Packet Too Big. */ "
> - "icmp6.code = 0; "
> - "icmp6.frag_mtu = %d; "
> - "next(pipeline=ingress, table=%d); };",
> - rp->lrp_networks.ea_s,
> - rp->lrp_networks.ipv6_addrs[0].addr_s,
> - gw_mtu,
> - ovn_stage_get_table(S_ROUTER_IN_ADMISSION));
> - ovn_lflow_add_with_hint__(lflows, od,
> - S_ROUTER_IN_LARGER_PKTS, 50,
> - ds_cstr(match),
> ds_cstr(actions),
> - NULL,
> - copp_meter_get(
> - COPP_ICMP6_ERR,
> - rp->od->nbr->copp,
> - meter_groups),
> - &rp->nbrp->header_);
> - }
> + ds_clear(actions);
> + /* Set icmp6.frag_mtu to gw_mtu */
> + ds_put_format(actions,
> + "icmp6_error {"
> + REGBIT_EGRESS_LOOPBACK" = 1; "
> + "eth.dst = %s; "
> + "ip6.dst = ip6.src; "
> + "ip6.src = %s; "
> + "ip.ttl = 255; "
> + "icmp6.type = 2; /* Packet Too Big. */ "
> + "icmp6.code = 0; "
> + "icmp6.frag_mtu = %d; "
> + "next(pipeline=ingress, table=%d); };",
> + rp->lrp_networks.ea_s,
> + rp->lrp_networks.ipv6_addrs[0].addr_s,
> + gw_mtu,
> + ovn_stage_get_table(S_ROUTER_IN_ADMISSION));
> + ovn_lflow_add_with_hint__(lflows, od,
> + S_ROUTER_IN_LARGER_PKTS, 50,
> + ds_cstr(match), ds_cstr(actions),
> + NULL,
> + copp_meter_get(
> + COPP_ICMP6_ERR,
> + rp->od->nbr->copp,
> + meter_groups),
> + &rp->nbrp->header_);
> }
> }
> }
> @@ -11055,32 +11101,32 @@ build_gateway_redirect_flows_for_lrouter(
> struct ovn_datapath *od, struct hmap *lflows,
> struct ds *match, struct ds *actions)
> {
> - if (od->nbr) {
> - if (od->l3dgw_port && od->l3redirect_port) {
> - const struct ovsdb_idl_row *stage_hint = NULL;
> -
> - if (od->l3dgw_port->nbrp) {
> - stage_hint = &od->l3dgw_port->nbrp->header_;
> - }
> + if (!od->nbr) {
> + return;
> + }
> + for (size_t i = 0; i < od->n_l3dgw_ports; i++) {
> + const struct ovsdb_idl_row *stage_hint = NULL;
>
> - /* For traffic with outport == l3dgw_port, if the
> - * packet did not match any higher priority redirect
> - * rule, then the traffic is redirected to the central
> - * instance of the l3dgw_port. */
> - ds_clear(match);
> - ds_put_format(match, "outport == %s",
> - od->l3dgw_port->json_key);
> - ds_clear(actions);
> - ds_put_format(actions, "outport = %s; next;",
> - od->l3redirect_port->json_key);
> - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_GW_REDIRECT, 50,
> - ds_cstr(match), ds_cstr(actions),
> - stage_hint);
> + if (od->l3dgw_ports[i]->nbrp) {
> + stage_hint = &od->l3dgw_ports[i]->nbrp->header_;
> }
>
> - /* Packets are allowed by default. */
> - ovn_lflow_add(lflows, od, S_ROUTER_IN_GW_REDIRECT, 0, "1", "next;");
> + /* For traffic with outport == l3dgw_port, if the
> + * packet did not match any higher priority redirect
> + * rule, then the traffic is redirected to the central
> + * instance of the l3dgw_port. */
> + ds_clear(match);
> + ds_put_format(match, "outport == %s",
> + od->l3dgw_ports[i]->json_key);
> + ds_clear(actions);
> + ds_put_format(actions, "outport = %s; next;",
> + od->l3dgw_ports[i]->cr_port->json_key);
> + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_GW_REDIRECT, 50,
> + ds_cstr(match), ds_cstr(actions),
> + stage_hint);
> }
> + /* Packets are allowed by default. */
> + ovn_lflow_add(lflows, od, S_ROUTER_IN_GW_REDIRECT, 0, "1", "next;");
> }
>
> /* Local router ingress table ARP_REQUEST: ARP request.
> @@ -11179,7 +11225,7 @@ build_egress_delivery_flows_for_lrouter_port(
> return;
> }
>
> - if (op->derived) {
> + if (op->l3dgw_port) {
> /* No egress packets should be processed in the context of
> * a chassisredirect port. The chassisredirect port should
> * be replaced by the l3dgw port in the local output
> @@ -11269,7 +11315,7 @@ build_dhcpv6_reply_flows_for_lrouter_port(
> struct ovn_port *op, struct hmap *lflows,
> struct ds *match)
> {
> - if (op->nbrp && (!op->derived)) {
> + if (op->nbrp && (!op->l3dgw_port)) {
> for (size_t i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) {
> ds_clear(match);
> ds_put_format(match, "ip6.dst == %s && udp.src == 547 &&"
> @@ -11289,7 +11335,7 @@ build_ipv6_input_flows_for_lrouter_port(
> struct ds *match, struct ds *actions,
> struct shash *meter_groups)
> {
> - if (op->nbrp && (!op->derived)) {
> + if (op->nbrp && (!op->l3dgw_port)) {
> /* No ingress packets are accepted on a chassisredirect
> * port, so no need to program flows for that port. */
> if (op->lrp_networks.n_ipv6_addrs) {
> @@ -11315,15 +11361,14 @@ build_ipv6_input_flows_for_lrouter_port(
> * router's own IP address. */
> for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) {
> ds_clear(match);
> - if (op->od->l3dgw_port && op == op->od->l3dgw_port
> - && op->od->l3redirect_port) {
> + if (op->cr_port) {
> /* Traffic with eth.src = l3dgw_port->lrp_networks.ea_s
> * should only be sent from the gateway chassi, so that
> * upstream MAC learning points to the gateway chassis.
> * Also need to avoid generation of multiple ND replies
> * from different chassis. */
> ds_put_format(match, "is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + op->cr_port->json_key);
> }
>
> build_lrouter_nd_flow(op->od, op, "nd_na_router",
> @@ -11334,7 +11379,7 @@ build_ipv6_input_flows_for_lrouter_port(
> }
>
> /* UDP/TCP/SCTP port unreachable */
> - if (!op->od->is_gw_router && !op->od->l3dgw_port) {
> + if (!op->od->is_gw_router && !op->od->l3dgw_ports) {
> for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) {
> ds_clear(match);
> ds_put_format(match,
> @@ -11504,7 +11549,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> {
> /* No ingress packets are accepted on a chassisredirect
> * port, so no need to program flows for that port. */
> - if (op->nbrp && (!op->derived)) {
> + if (op->nbrp && (!op->l3dgw_port)) {
> if (op->lrp_networks.n_ipv4_addrs) {
> /* L3 admission control: drop packets that originate from an
> * IPv4 address owned by the router or a broadcast address
> @@ -11574,16 +11619,18 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> op->lrp_networks.ipv4_addrs[i].network_s,
> op->lrp_networks.ipv4_addrs[i].plen);
>
> - if (op->od->l3dgw_port && op->od->l3redirect_port && op->peer
> + if (op->od->l3dgw_ports && op->peer
> && op->peer->od->n_localnet_ports) {
> bool add_chassis_resident_check = false;
> - if (op == op->od->l3dgw_port) {
> + const char *json_key;
> + if (op->cr_port) {
> /* Traffic with eth.src = l3dgw_port->lrp_networks.ea_s
> * should only be sent from the gateway chassis, so that
> * upstream MAC learning points to the gateway chassis.
> * Also need to avoid generation of multiple ARP
> responses
> * from different chassis. */
> add_chassis_resident_check = true;
> + json_key = op->cr_port->json_key;
> } else {
> /* Check if the option 'reside-on-redirect-chassis'
> * is set to true on the router port. If set to true
> @@ -11595,12 +11642,14 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> */
> add_chassis_resident_check = smap_get_bool(
> &op->nbrp->options,
> - "reside-on-redirect-chassis", false);
> + "reside-on-redirect-chassis", false) &&
> + op->od->n_l3dgw_ports == 1;
> + json_key = op->od->l3dgw_ports[0]->cr_port->json_key;
> }
>
> if (add_chassis_resident_check) {
> ds_put_format(match, " && is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + json_key);
> }
> }
>
> @@ -11613,9 +11662,9 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> const char *ip_address;
> if (sset_count(&op->od->lb_ips_v4)) {
> ds_clear(match);
> - if (op == op->od->l3dgw_port) {
> + if (op->cr_port) {
> ds_put_format(match, "is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + op->cr_port->json_key);
> }
>
> struct ds load_balancer_ips_v4 = DS_EMPTY_INITIALIZER;
> @@ -11633,9 +11682,9 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
>
> SSET_FOR_EACH (ip_address, &op->od->lb_ips_v6) {
> ds_clear(match);
> - if (op == op->od->l3dgw_port) {
> + if (op->cr_port) {
> ds_put_format(match, "is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + op->cr_port->json_key);
> }
>
> build_lrouter_nd_flow(op->od, op, "nd_na",
> @@ -11644,7 +11693,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> lflows, meter_groups);
> }
>
> - if (!op->od->is_gw_router && !op->od->l3dgw_port) {
> + if (!op->od->is_gw_router && !op->od->l3dgw_ports) {
> /* UDP/TCP/SCTP port unreachable. */
> for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) {
> ds_clear(match);
> @@ -11741,7 +11790,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> * exception is on the l3dgw_port where we might need to use a
> * different ETH address.
> */
> - if (op != op->od->l3dgw_port) {
> + if (!op->cr_port) {
> return;
> }
>
> @@ -11823,12 +11872,12 @@ build_lrouter_in_unsnat_flow(struct hmap *lflows,
> struct ovn_datapath *od,
> ds_clear(actions);
> ds_put_format(match, "ip && ip%s.dst == %s && inport == %s",
> is_v6 ? "6" : "4", nat->external_ip,
> - od->l3dgw_port->json_key);
> - if (!distributed && od->l3redirect_port) {
> + od->l3dgw_ports[0]->json_key);
> + if (!distributed && od->l3dgw_ports) {
> /* Flows for NAT rules that are centralized are only
> * programmed on the gateway chassis. */
> ds_put_format(match, " && is_chassis_resident(%s)",
> - od->l3redirect_port->json_key);
> + od->l3dgw_ports[0]->cr_port->json_key);
> }
>
> if (!strcmp(nat->type, "dnat_and_snat") && stateless) {
> @@ -11900,12 +11949,12 @@ build_lrouter_in_dnat_flow(struct hmap *lflows,
> struct ovn_datapath *od,
> ds_clear(match);
> ds_put_format(match, "ip && ip%s.dst == %s && inport == %s",
> is_v6 ? "6" : "4", nat->external_ip,
> - od->l3dgw_port->json_key);
> - if (!distributed && od->l3redirect_port) {
> + od->l3dgw_ports[0]->json_key);
> + if (!distributed && od->l3dgw_ports) {
> /* Flows for NAT rules that are centralized are only
> * programmed on the gateway chassis. */
> ds_put_format(match, " && is_chassis_resident(%s)",
> - od->l3redirect_port->json_key);
> + od->l3dgw_ports[0]->cr_port->json_key);
> }
> ds_clear(actions);
> if (nat->allowed_ext_ips || nat->exempted_ext_ips) {
> @@ -11944,7 +11993,7 @@ build_lrouter_out_undnat_flow(struct hmap *lflows,
> struct ovn_datapath *od,
> *
> * Note that this only applies for NAT on a distributed router.
> */
> - if (!od->l3dgw_port ||
> + if (!od->l3dgw_ports ||
> (strcmp(nat->type, "dnat") && strcmp(nat->type, "dnat_and_snat"))) {
> return;
> }
> @@ -11952,12 +12001,12 @@ build_lrouter_out_undnat_flow(struct hmap *lflows,
> struct ovn_datapath *od,
> ds_clear(match);
> ds_put_format(match, "ip && ip%s.src == %s && outport == %s",
> is_v6 ? "6" : "4", nat->logical_ip,
> - od->l3dgw_port->json_key);
> - if (!distributed && od->l3redirect_port) {
> + od->l3dgw_ports[0]->json_key);
> + if (!distributed && od->l3dgw_ports) {
> /* Flows for NAT rules that are centralized are only
> * programmed on the gateway chassis. */
> ds_put_format(match, " && is_chassis_resident(%s)",
> - od->l3redirect_port->json_key);
> + od->l3dgw_ports[0]->cr_port->json_key);
> }
> ds_clear(actions);
> if (distributed) {
> @@ -12030,13 +12079,13 @@ build_lrouter_out_snat_flow(struct hmap *lflows,
> struct ovn_datapath *od,
> ds_clear(match);
> ds_put_format(match, "ip && ip%s.src == %s && outport == %s",
> is_v6 ? "6" : "4", nat->logical_ip,
> - od->l3dgw_port->json_key);
> - if (!distributed && od->l3redirect_port) {
> + od->l3dgw_ports[0]->json_key);
> + if (!distributed && od->l3dgw_ports) {
> /* Flows for NAT rules that are centralized are only
> * programmed on the gateway chassis. */
> priority += 128;
> ds_put_format(match, " && is_chassis_resident(%s)",
> - od->l3redirect_port->json_key);
> + od->l3dgw_ports[0]->cr_port->json_key);
> }
> ds_clear(actions);
>
> @@ -12077,11 +12126,11 @@ build_lrouter_ingress_flow(struct hmap *lflows,
> struct ovn_datapath *od,
> struct ds *actions, struct eth_addr mac,
> bool distributed, bool is_v6)
> {
> - if (od->l3dgw_port && !strcmp(nat->type, "snat")) {
> + if (od->l3dgw_ports && !strcmp(nat->type, "snat")) {
> ds_clear(match);
> ds_put_format(
> match, "inport == %s && %s == %s",
> - od->l3dgw_port->json_key,
> + od->l3dgw_ports[0]->json_key,
> is_v6 ? "ip6.src" : "ip4.src", nat->external_ip);
> ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_INPUT,
> 120, ds_cstr(match), "next;",
> @@ -12099,14 +12148,14 @@ build_lrouter_ingress_flow(struct hmap *lflows,
> struct ovn_datapath *od,
> */
> ds_clear(actions);
> ds_put_format(actions, REG_INPORT_ETH_ADDR " = %s; next;",
> - od->l3dgw_port->lrp_networks.ea_s);
> + od->l3dgw_ports[0]->lrp_networks.ea_s);
>
> ds_clear(match);
> ds_put_format(match,
> "eth.dst == "ETH_ADDR_FMT" && inport == %s"
> " && is_chassis_resident(\"%s\")",
> ETH_ADDR_ARGS(mac),
> - od->l3dgw_port->json_key,
> + od->l3dgw_ports[0]->json_key,
> nat->logical_port);
> ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_ADMISSION, 50,
> ds_cstr(match), ds_cstr(actions),
> @@ -12188,7 +12237,7 @@ lrouter_check_nat_entry(struct ovn_datapath *od,
> const struct nbrec_nat *nat,
> /* For distributed router NAT, determine whether this NAT rule
> * satisfies the conditions for distributed NAT processing. */
> *distributed = false;
> - if (od->l3dgw_port && !strcmp(nat->type, "dnat_and_snat") &&
> + if (od->l3dgw_ports && !strcmp(nat->type, "dnat_and_snat") &&
> nat->logical_port && nat->external_mac) {
> if (eth_addr_from_string(nat->external_mac, mac)) {
> *distributed = true;
> @@ -12233,7 +12282,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath
> *od, struct hmap *lflows,
> * not committed, it would produce ongoing datapath flows with the ct.new
> * flag set. Some NICs are unable to offload these flows.
> */
> - if ((od->is_gw_router || od->l3dgw_port) &&
> + if ((od->is_gw_router || od->l3dgw_ports) &&
> (od->nbr->n_nat || od->nbr->n_load_balancer)) {
> ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 50,
> "ip", "flags.loopback = 1; ct_dnat;");
> @@ -12249,7 +12298,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath
> *od, struct hmap *lflows,
> /* NAT rules are only valid on Gateway routers and routers with
> * l3dgw_port (router has a port with gateway chassis
> * specified). */
> - if (!od->is_gw_router && !od->l3dgw_port) {
> + if (!od->is_gw_router && !od->l3dgw_ports) {
> return;
> }
>
> @@ -12290,14 +12339,14 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath
> *od, struct hmap *lflows,
> ds_clear(match);
> ds_put_format(
> match, "outport == %s && %s == %s",
> - od->l3dgw_port->json_key,
> + od->l3dgw_ports[0]->json_key,
> is_v6 ? REG_NEXT_HOP_IPV6 : REG_NEXT_HOP_IPV4,
> nat->external_ip);
> ds_clear(actions);
> ds_put_format(
> actions, "eth.dst = %s; next;",
> distributed ? nat->external_mac :
> - od->l3dgw_port->lrp_networks.ea_s);
> + od->l3dgw_ports[0]->lrp_networks.ea_s);
> ovn_lflow_add_with_hint(lflows, od,
> S_ROUTER_IN_ARP_RESOLVE,
> 100, ds_cstr(match),
> @@ -12333,7 +12382,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath
> *od, struct hmap *lflows,
> ds_put_format(match,
> "ip%s.src == %s && outport == %s",
> is_v6 ? "6" : "4", nat->logical_ip,
> - od->l3dgw_port->json_key);
> + od->l3dgw_ports[0]->json_key);
> /* Add a rule to drop traffic from a distributed NAT if
> * the virtual port has not claimed yet becaused otherwise
> * the traffic will be centralized misconfiguring the TOR switch.
> @@ -12360,16 +12409,16 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath
> *od, struct hmap *lflows,
> * gateway port have ip.dst matching a NAT external IP, then
> * loop a clone of the packet back to the beginning of the
> * ingress pipeline with inport = outport. */
> - if (od->l3dgw_port) {
> + if (od->l3dgw_ports) {
> /* Distributed router. */
> ds_clear(match);
> ds_put_format(match, "ip%s.dst == %s && outport == %s",
> is_v6 ? "6" : "4",
> nat->external_ip,
> - od->l3dgw_port->json_key);
> + od->l3dgw_ports[0]->json_key);
> if (!distributed) {
> ds_put_format(match, " && is_chassis_resident(%s)",
> - od->l3redirect_port->json_key);
> + od->l3dgw_ports[0]->cr_port->json_key);
> } else {
> ds_put_format(match, " && is_chassis_resident(\"%s\")",
> nat->logical_port);
> diff --git a/northd/ovn_northd.dl b/northd/ovn_northd.dl
> index 7bfaae992..c6ebecb81 100644
> --- a/northd/ovn_northd.dl
> +++ b/northd/ovn_northd.dl
> @@ -183,7 +183,7 @@ OutProxy_Port_Binding(._uuid = lsp._uuid,
> },
> Some{var router_port} = lsp.options.get("router-port"),
> var opt_chassis = peer.and_then(|p| p.router.options.get("chassis")),
> - var l3dgw_port = peer.and_then(|p| p.router.l3dgw_port),
> + var l3dgw_port = peer.and_then(|p| p.router.l3dgw_ports.nth(0)),
> (var __type, var options) = {
> var options = ["peer" -> router_port];
> match (opt_chassis) {
> @@ -239,7 +239,7 @@ OutProxy_Port_Binding(._uuid = lsp._uuid,
> Some{rport} -> match (
> (rport.lrp.options.get_bool_def("reside-on-redirect-chassis",
> false)
> and l3dgw_port.is_some()) or
> - Some{rport.lrp} == l3dgw_port or
> + rport.is_redirect or
> (rport.router.options.contains_key("chassis") and
> not sw.localnet_ports.is_empty())) {
> false -> set_empty(),
> @@ -333,7 +333,7 @@ function get_router_load_balancer_ips(router:
> Intern<Router>,
> function get_nat_addresses(rport: Intern<RouterPort>, routable_only: bool):
> Set<string> =
> {
> var addresses = set_empty();
> - var has_redirect = rport.router.l3dgw_port.is_some();
> + var has_redirect = not rport.router.l3dgw_ports.is_empty();
I didn't understand this. Why is the new code using "is_empty()"
where as the deleted code has "is_some()" ?
> match (eth_addr_from_string(rport.lrp.mac)) {
> None -> addresses,
> Some{mac} -> {
> @@ -400,7 +400,10 @@ function get_nat_addresses(rport: Intern<RouterPort>,
> routable_only: bool): Set<
> /* Gratuitous ARP for centralized NAT rules on distributed
> gateway
> * ports should be restricted to the gateway chassis. */
> if (has_redirect) {
> - c_addresses = c_addresses ++ "
> is_chassis_resident(${rport.router.redirect_port_name})"
> + c_addresses = c_addresses ++ match
> (rport.router.l3dgw_ports.nth(0)) {
> + None -> "",
> + Some {var gw_port} -> "
> is_chassis_resident(${json_string_escape(chassis_redirect_name(gw_port.name))})"
> + }
> } else ();
>
> addresses.insert(c_addresses)
> @@ -415,8 +418,10 @@ function get_garp_nat_addresses(rport:
> Intern<RouterPort>): string = {
> for (ipv4_addr in rport.networks.ipv4_addrs) {
> garp_info.push("${ipv4_addr.addr}")
> };
> - if (rport.router.redirect_port_name != "") {
> -
> garp_info.push("is_chassis_resident(${rport.router.redirect_port_name})")
> + match (rport.router.l3dgw_ports.nth(0)) {
> + None -> (),
> + Some {var gw_port} -> garp_info.push(
> +
> "is_chassis_resident(${json_string_escape(chassis_redirect_name(gw_port.name))})")
> };
> garp_info.join(" ")
> }
> @@ -453,7 +458,7 @@ OutProxy_Port_Binding(// lrp._uuid is already in use;
> generate a new UUID by
> .nat_addresses = set_empty(),
> .external_ids = lrp.external_ids) :-
> DistributedGatewayPort(lrp, lr_uuid),
> - LogicalRouterHAChassisGroup(lr_uuid, hacg_uuid),
> + DistributedGatewayPortHAChassisGroup(lrp, hacg_uuid),
> var redirect_type = match (lrp.options.get("redirect-type")) {
> Some{var value} -> ["redirect-type" -> value],
> _ -> map_empty()
> @@ -509,7 +514,8 @@ sb::Out_Port_Binding(._uuid = pbinding._uuid,
> * chassis. RefChassisSet has a row for every logical router. */
> relation RefChassis(lr_uuid: uuid, chassis_uuid: uuid)
> RefChassis(lr_uuid, chassis_uuid) :-
> - LogicalRouterHAChassisGroup(lr_uuid, _),
> + DistributedGatewayPortHAChassisGroup(lrp, _),
> + DistributedGatewayPort(lrp, lr_uuid),
> ConnectedLogicalRouter[(lr_uuid, set_uuid)],
> ConnectedLogicalRouter[(lr2_uuid, set_uuid)],
> FirstHopLogicalRouter(lr2_uuid, ls_uuid),
> @@ -536,7 +542,8 @@ RefChassisSet(lr_uuid, set_empty()) :-
> relation HAChassisGroupRefChassisSet(hacg_uuid: uuid,
> chassis_uuids: Set<uuid>)
> HAChassisGroupRefChassisSet(hacg_uuid, chassis_uuids) :-
> - LogicalRouterHAChassisGroup(lr_uuid, hacg_uuid),
> + DistributedGatewayPortHAChassisGroup(lrp, hacg_uuid),
> + DistributedGatewayPort(lrp, lr_uuid),
> RefChassisSet(lr_uuid, chassis_uuids),
> var chassis_uuids = chassis_uuids.group_by(hacg_uuid).union().
>
> @@ -4453,7 +4460,7 @@ for (&SwitchPort(.lsp = lsp,
> .peer = Some{&RouterPort{.lrp = lrp,
> .is_redirect = is_redirect,
> .router = &Router{._uuid = lr_uuid,
> -
> .redirect_port_name = redirect_port_name}}})
> + .l3dgw_ports =
> l3dgw_ports}}})
> if (lsp.addresses.contains("router") and lsp.__type != "external"))
> {
> Some{var mac} = scan_eth_addr(lrp.mac) in {
> @@ -4473,6 +4480,14 @@ for (&SwitchPort(.lsp = lsp,
> */
> lrp.options.get_bool_def("reside-on-redirect-chassis", false))
> in
> var __match = if (add_chassis_resident_check) {
> + var redirect_port_name = if (is_redirect) {
> + json_string_escape(chassis_redirect_name(lrp.name))
> + } else {
> + match (l3dgw_ports.nth(0)) {
> + Some {var gw_port} ->
> json_string_escape(chassis_redirect_name(gw_port.name)),
> + None -> ""
> + }
> + };
> /* The destination lookup flow for the router's
> * distributed gateway port MAC address should only be
> * programmed on the "redirect-chassis". */
> @@ -4872,13 +4887,8 @@ var rLNIR = rEGBIT_LOOKUP_NEIGHBOR_IP_RESULT() in
>
> /* Check if we need to learn mac-binding from ARP requests. */
> for (RouterPortNetworksIPv4Addr(rp@&RouterPort{.router = router}, addr)) {
> - var is_l3dgw_port = match (router.l3dgw_port) {
> - Some{l3dgw_lrp} -> l3dgw_lrp._uuid == rp.lrp._uuid,
> - None -> false
> - } in
> - var has_redirect_port = router.redirect_port_name != "" in
> - var chassis_residence = match (is_l3dgw_port and has_redirect_port) {
> - true -> " && is_chassis_resident(${router.redirect_port_name})",
> + var chassis_residence = match (rp.is_redirect) {
> + true -> " &&
> is_chassis_resident(${json_string_escape(chassis_redirect_name(rp.lrp.name))})",
> false -> ""
> } in
> var rLNR = rEGBIT_LOOKUP_NEIGHBOR_RESULT() in
> @@ -5038,7 +5048,7 @@ relation AddChassisResidentCheck_(lrp: uuid, add_check:
> bool)
> AddChassisResidentCheck_(lrp._uuid, res) :-
> &SwitchPort(.peer = Some{&RouterPort{.lrp = lrp, .router = router,
> .is_redirect = is_redirect}},
> .sw = sw),
> - router.l3dgw_port.is_some(),
> + not router.l3dgw_ports.is_empty(),
> not sw.localnet_ports.is_empty(),
> var res = if (is_redirect) {
> /* Traffic with eth.src = l3dgw_port->lrp_networks.ea
> @@ -5143,7 +5153,8 @@ LogicalRouterArpNdFlow(router, nat, None,
> rEG_INPORT_ETH_ADDR(), None, false, 90
> * different ETH address.
> */
> LogicalRouterPortNatArpNdFlow(router, nat, l3dgw_port) :-
> - router in &Router(._uuid = lr_uuid, .l3dgw_port = Some{l3dgw_port}),
> + router in &Router(._uuid = lr_uuid, .l3dgw_ports = l3dgw_ports),
> + Some {var l3dgw_port} = l3dgw_ports.nth(0),
> LogicalRouterNAT(lr_uuid, nat),
> /* Skip SNAT entries for now, we handle unique SNAT IPs separately
> * below.
> @@ -5151,7 +5162,8 @@ LogicalRouterPortNatArpNdFlow(router, nat, l3dgw_port)
> :-
> nat.nat.__type != "snat".
> /* Now handle SNAT entries too, one per unique SNAT IP. */
> LogicalRouterPortNatArpNdFlow(router, nat, l3dgw_port) :-
> - router in &Router(.l3dgw_port = Some{l3dgw_port}, .snat_ips = snat_ips),
> + router in &Router(.l3dgw_ports = l3dgw_ports, .snat_ips = snat_ips),
> + Some {var l3dgw_port} = l3dgw_ports.nth(0),
> var snat_ip = FlatMap(snat_ips),
> (var ip, var nats) = snat_ip,
> Some{var nat} = nats.nth(0).
> @@ -5181,9 +5193,9 @@ LogicalRouterArpNdFlow(router, nat, Some{lrp}, mac,
> None, true, 91) :-
> * upstream MAC learning points to the gateway chassis.
> * Also need to avoid generation of multiple ARP responses
> * from different chassis. */
> - match (router.redirect_port_name) {
> - "" -> "",
> - s -> "is_chassis_resident(${s})"
> + match (router.l3dgw_ports.nth(0)) {
> + None -> "",
> + Some {var gw_port} ->
> "is_chassis_resident(${json_string_escape(chassis_redirect_name(gw_port.name))})"
> }
> )
> }.
> @@ -5328,7 +5340,15 @@ for (RouterPortNetworksIPv4Addr(.port =
> &RouterPort{.lrp = lrp,
> var __match =
> "arp.spa == ${addr.match_network()}" ++
> if (add_chassis_resident_check) {
> - " && is_chassis_resident(${router.redirect_port_name})"
> + var redirect_port_name = if (is_redirect) {
> + json_string_escape(chassis_redirect_name(lrp.name))
> + } else {
> + match (router.l3dgw_ports.nth(0)) {
> + None -> "",
> + Some {var gw_port} ->
> json_string_escape(chassis_redirect_name(gw_port.name))
> + }
> + };
> + " && is_chassis_resident(${redirect_port_name})"
> } else "" in
> LogicalRouterArpFlow(.lr = router,
> .lrp = Some{lrp},
> @@ -5347,7 +5367,7 @@ for (&RouterPort(.lrp = lrp,
> .networks = networks,
> .is_redirect = is_redirect))
> var residence_check = match (is_redirect) {
> - true -> Some{"is_chassis_resident(${router.redirect_port_name})"},
> + true ->
> Some{"is_chassis_resident(${json_string_escape(chassis_redirect_name(lrp.name))})"},
> false -> None
> } in {
> (var all_ips_v4, _) = get_router_load_balancer_ips(router, false) in {
> @@ -5417,7 +5437,7 @@ Flow(.logical_datapath = lr_uuid,
> for (RouterPortNetworksIPv4Addr(
> .port = &RouterPort{
> .router = &Router{._uuid = lr_uuid,
> - .l3dgw_port = None,
> + .l3dgw_ports = vec_empty(),
> .is_gateway = false,
> .copp = copp},
> .lrp = lrp},
> @@ -5553,7 +5573,7 @@ for (RouterPortNetworksIPv6Addr(.port =
> &RouterPort{.lrp = lrp,
> /* UDP/TCP/SCTP port unreachable */
> for (RouterPortNetworksIPv6Addr(
> .port = &RouterPort{.router = &Router{._uuid = lr_uuid,
> - .l3dgw_port = None,
> + .l3dgw_ports = vec_empty(),
> .is_gateway = false,
> .copp = copp},
> .lrp = lrp,
> @@ -5681,11 +5701,11 @@ for (r in &Router(._uuid = lr_uuid)) {
> }
>
> for (r in &Router(._uuid = lr_uuid,
> - .l3dgw_port = l3dgw_port,
> + .l3dgw_ports = l3dgw_ports,
> .is_gateway = is_gateway,
> .nat = nat,
> .load_balancer = load_balancer)
> - if (l3dgw_port.is_some() or is_gateway) and (not is_empty(nat) or not
> is_empty(load_balancer))) {
> + if (l3dgw_ports.len() > 0 or is_gateway) and (not is_empty(nat) or not
> is_empty(load_balancer))) {
> /* If the router has load balancer or DNAT rules, re-circulate every
> packet
> * through the DNAT zone so that packets that need to be unDNATed in the
> * reverse direction get unDNATed.
> @@ -5768,7 +5788,7 @@ function lrouter_nat_add_ext_ip_match(
> },
> false -> {
> /* S_ROUTER_OUT_SNAT uses priority (mask + 1 + 128 + 1)
> */
> - var is_gw_router = router.l3dgw_port == None;
> + var is_gw_router = router.l3dgw_ports.is_empty();
> var mask_1bits = mask.cidr_bits().unwrap_or(8'd0) as
> integer;
> mask_1bits + 2 + { if (not is_gw_router) 128 else 0 }
> }
> @@ -5873,10 +5893,9 @@ VirtualLogicalPort(Some{logical_port}) :-
> * l3dgw_port (router has a port with "redirect-chassis"
> * specified). */
> for (r in &Router(._uuid = lr_uuid,
> - .l3dgw_port = l3dgw_port,
> - .redirect_port_name = redirect_port_name,
> + .l3dgw_ports = l3dgw_ports,
> .is_gateway = is_gateway)
> - if l3dgw_port.is_some() or is_gateway)
> + if not l3dgw_ports.is_empty() or is_gateway)
> {
> for (LogicalRouterNAT(.lr = lr_uuid, .nat = nat)) {
> var ipX = nat.external_ip.ipX() in
> @@ -5894,7 +5913,7 @@ for (r in &Router(._uuid = lr_uuid,
> } in
> /* For distributed router NAT, determine whether this NAT rule
> * satisfies the conditions for distributed NAT processing. */
> - var mac = match ((l3dgw_port.is_some() and nat.nat.__type ==
> "dnat_and_snat",
> + var mac = match ((not l3dgw_ports.is_empty() and nat.nat.__type ==
> "dnat_and_snat",
> nat.nat.logical_port, nat.external_mac)) {
> (true, Some{_}, Some{mac}) -> Some{mac},
> _ -> None
> @@ -5912,7 +5931,7 @@ for (r in &Router(._uuid = lr_uuid,
> * not know about the possibility of eventual additional SNAT in
> * egress pipeline. */
> if (nat.nat.__type == "snat" or nat.nat.__type ==
> "dnat_and_snat") {
> - if (l3dgw_port == None) {
> + if (l3dgw_ports.is_empty()) {
> /* Gateway router. */
> var actions = if (stateless) {
> "${ipX}.dst=${nat.nat.logical_ip}; next;"
> @@ -5926,7 +5945,7 @@ for (r in &Router(._uuid = lr_uuid,
> .actions = actions,
> .external_ids = stage_hint(nat.nat._uuid))
> };
> - Some{var gwport} = l3dgw_port in {
> + Some {var gwport} = l3dgw_ports.nth(0) in {
> /* Distributed router. */
>
> /* Traffic received on l3dgw_port is subject to NAT. */
> @@ -5936,7 +5955,7 @@ for (r in &Router(._uuid = lr_uuid,
> if (mac == None) {
> /* Flows for NAT rules that are centralized are
> only
> * programmed on the "redirect-chassis". */
> - " && is_chassis_resident(${redirect_port_name})"
> + " &&
> is_chassis_resident(${json_string_escape(chassis_redirect_name(gwport.name))})"
> } else { "" } in
> var actions = if (stateless) {
> "${ipX}.dst=${nat.nat.logical_ip}; next;"
> @@ -5962,7 +5981,7 @@ for (r in &Router(._uuid = lr_uuid,
> ""
> } in
> if (nat.nat.__type == "dnat" or nat.nat.__type ==
> "dnat_and_snat") {
> - None = l3dgw_port in
> + l3dgw_ports.is_empty() in
> var __match = "ip && ${ipX}.dst == ${nat.nat.external_ip}" in
> (var ext_ip_match, var ext_flow) =
> lrouter_nat_add_ext_ip_match(
> r, nat, __match, ipX, true, mask) in
> @@ -5994,14 +6013,14 @@ for (r in &Router(._uuid = lr_uuid,
> .external_ids = stage_hint(nat.nat._uuid))
> };
>
> - Some{var gwport} = l3dgw_port in
> + Some {var gwport} = l3dgw_ports.nth(0) in
> var __match =
> "ip && ${ipX}.dst == ${nat.nat.external_ip}"
> " && inport == ${json_string_escape(gwport.name)}" ++
> if (mac == None) {
> /* Flows for NAT rules that are centralized are only
> * programmed on the "redirect-chassis". */
> - " && is_chassis_resident(${redirect_port_name})"
> + " &&
> is_chassis_resident(${json_string_escape(chassis_redirect_name(gwport.name))})"
> } else { "" } in
> (var ext_ip_match, var ext_flow) =
> lrouter_nat_add_ext_ip_match(
> r, nat, __match, ipX, true, mask) in
> @@ -6025,7 +6044,7 @@ for (r in &Router(._uuid = lr_uuid,
> };
>
> /* ARP resolve for NAT IPs. */
> - Some{var gwport} = l3dgw_port in {
> + Some {var gwport} = l3dgw_ports.nth(0) in {
> var gwport_name = json_string_escape(gwport.name) in {
> if (nat.nat.__type == "snat") {
> var __match = "inport == ${gwport_name} && "
> @@ -6062,14 +6081,14 @@ for (r in &Router(._uuid = lr_uuid,
> * Note that this only applies for NAT on a distributed router.
> */
> if ((nat.nat.__type == "dnat" or nat.nat.__type ==
> "dnat_and_snat")) {
> - Some{var gwport} = l3dgw_port in
> + Some {var gwport} = l3dgw_ports.nth(0) in
> var __match =
> "ip && ${ipX}.src == ${nat.nat.logical_ip}"
> " && outport == ${json_string_escape(gwport.name)}" ++
> if (mac == None) {
> /* Flows for NAT rules that are centralized are only
> * programmed on the "redirect-chassis". */
> - " && is_chassis_resident(${redirect_port_name})"
> + " &&
> is_chassis_resident(${json_string_escape(chassis_redirect_name(gwport.name))})"
> } else { "" } in
> var actions =
> match (mac) {
> @@ -6099,7 +6118,7 @@ for (r in &Router(._uuid = lr_uuid,
> ""
> } in
> if (nat.nat.__type == "snat" or nat.nat.__type ==
> "dnat_and_snat") {
> - None = l3dgw_port in
> + l3dgw_ports.is_empty() in
> var __match = "ip && ${ipX}.src == ${nat.nat.logical_ip}" in
> (var ext_ip_match, var ext_flow) =
> lrouter_nat_add_ext_ip_match(
> r, nat, __match, ipX, false, mask) in
> @@ -6124,14 +6143,14 @@ for (r in &Router(._uuid = lr_uuid,
> .external_ids = stage_hint(nat.nat._uuid))
> };
>
> - Some{var gwport} = l3dgw_port in
> + Some {var gwport} = l3dgw_ports.nth(0) in
> var __match =
> "ip && ${ipX}.src == ${nat.nat.logical_ip}"
> " && outport == ${json_string_escape(gwport.name)}" ++
> if (mac == None) {
> /* Flows for NAT rules that are centralized are only
> * programmed on the "redirect-chassis". */
> - " && is_chassis_resident(${redirect_port_name})"
> + " &&
> is_chassis_resident(${json_string_escape(chassis_redirect_name(gwport.name))})"
> } else { "" } in
> (var ext_ip_match, var ext_flow) =
> lrouter_nat_add_ext_ip_match(
> r, nat, __match, ipX, false, mask) in
> @@ -6169,7 +6188,7 @@ for (r in &Router(._uuid = lr_uuid,
> * on the l3dgw_port instance where nat->logical_port is
> * resident. */
> Some{var mac_addr} = mac in
> - Some{var gwport} = l3dgw_port in
> + Some{var gwport} = l3dgw_ports.nth(0) in
> Some{var logical_port} = nat.nat.logical_port in
> var __match =
> "eth.dst == ${mac_addr} && inport ==
> ${json_string_escape(gwport.name)}"
> @@ -6195,7 +6214,7 @@ for (r in &Router(._uuid = lr_uuid,
> * stage is sent out with proper IP/MAC src addresses
> */
> Some{var mac_addr} = mac in
> - Some{var gwport} = l3dgw_port in
> + Some{var gwport} = l3dgw_ports.nth(0) in
> Some{var logical_port} = nat.nat.logical_port in
> Some{var external_mac} = nat.nat.external_mac in
> var __match =
> @@ -6214,7 +6233,7 @@ for (r in &Router(._uuid = lr_uuid,
> .external_ids = stage_hint(nat.nat._uuid));
>
> for (VirtualLogicalPort(nat.nat.logical_port)) {
> - Some{var gwport} = l3dgw_port in
> + Some{var gwport} = l3dgw_ports.nth(0) in
> Flow(.logical_datapath = lr_uuid,
> .stage = s_ROUTER_IN_GW_REDIRECT(),
> .priority = 80,
> @@ -6229,14 +6248,14 @@ for (r in &Router(._uuid = lr_uuid,
> * gateway port have ip.dst matching a NAT external IP, then
> * loop a clone of the packet back to the beginning of the
> * ingress pipeline with inport = outport. */
> - Some{var gwport} = l3dgw_port in
> + Some{var gwport} = l3dgw_ports.nth(0) in
> /* Distributed router. */
> Some{var port} = match (mac) {
> Some{_} -> match (nat.nat.logical_port) {
> Some{name} -> Some{json_string_escape(name)},
> None -> None: Option<string>
> },
> - None -> Some{redirect_port_name}
> + None ->
> Some{json_string_escape(chassis_redirect_name(gwport.name))}
> } in
> var __match = "${ipX}.dst == ${nat.nat.external_ip} && outport
> == ${json_string_escape(gwport.name)} && is_chassis_resident(${port})" in
> var regs = {
> @@ -6264,7 +6283,7 @@ for (r in &Router(._uuid = lr_uuid,
> };
>
> /* Handle force SNAT options set in the gateway router. */
> - if (l3dgw_port == None) {
> + if (l3dgw_ports.is_empty()) {
> var dnat_force_snat_ips = get_force_snat_ip(r.options, "dnat") in
> if (not dnat_force_snat_ips.is_empty())
> LogicalRouterForceSnatFlows(.logical_router = lr_uuid,
> @@ -6292,14 +6311,13 @@ function nats_contain_vip(nats: Vec<NAT>, vip:
> v46_ip): bool {
> * Gateway routers or router with gateway port. */
> for (RouterLBVIP(
> .router = r@&Router{._uuid = lr_uuid,
> - .l3dgw_port = l3dgw_port,
> - .redirect_port_name = redirect_port_name,
> + .l3dgw_ports = l3dgw_ports,
> .is_gateway = is_gateway,
> .nats = nats},
> .lb = lb,
> .vip = vip,
> .backends = backends)
> - if l3dgw_port.is_some() or is_gateway)
> + if not l3dgw_ports.is_empty() or is_gateway)
> {
> if (backends == "" and not lb.options.get_bool_def("reject", false)) {
> for (LoadBalancerEmptyEvents(lb)) {
> @@ -6368,8 +6386,8 @@ for (RouterLBVIP(
> (110, "")
> } in
> var __match = match1 ++ match2 ++
> - match ((l3dgw_port, backends != "" or
> lb.options.get_bool_def("reject", false))) {
> - (Some{gwport}, true) -> " &&
> is_chassis_resident(${redirect_port_name})",
> + match ((l3dgw_ports.nth(0), backends != "" or
> lb.options.get_bool_def("reject", false))) {
> + (Some{gw_port}, true) -> " &&
> is_chassis_resident(${json_string_escape(chassis_redirect_name(gw_port.name))})",
> _ -> ""
> } in
> var snat_for_lb = snat_for_lb(r.options, lb) in
> @@ -6381,8 +6399,8 @@ for (RouterLBVIP(
> } else {
> ""
> } ++
> - match ((l3dgw_port, backends != "" or
> lb.options.get_bool_def("reject", false))) {
> - (Some{gwport}, true) -> " &&
> is_chassis_resident(${redirect_port_name})",
> + match ((l3dgw_ports.nth(0), backends != "" or
> lb.options.get_bool_def("reject", false))) {
> + (Some {var gw_port}, true) -> " &&
> is_chassis_resident(${json_string_escape(chassis_redirect_name(gw_port.name))})",
> _ -> ""
> } in
> var actions =
> @@ -6421,7 +6439,7 @@ for (RouterLBVIP(
> .external_ids = stage_hint(lb._uuid))
> };
>
> - Some{var gwport} = l3dgw_port in
> + Some{var gwport} = l3dgw_ports.nth(0) in
> /* Add logical flows to UNDNAT the load balanced reverse traffic
> in
> * the router egress pipleine stage - S_ROUTER_OUT_UNDNAT if the
> logical
> * router has a gateway router port associated.
> @@ -6446,7 +6464,7 @@ for (RouterLBVIP(
> var undnat_match =
> "${ip_address.ipX()} && (" ++ conds.join(" || ") ++
> ") && outport == ${json_string_escape(gwport.name)} && "
> - "is_chassis_resident(${redirect_port_name})" in
> +
> "is_chassis_resident(${json_string_escape(chassis_redirect_name(gwport.name))})"
> in
> var action =
> match (snat_for_lb) {
> SkipSNAT -> "flags.skip_snat_for_lb = 1; ct_dnat;",
> @@ -6477,14 +6495,14 @@ MeteredFlow(.logical_datapath = r._uuid,
> .controller_meter = meter,
> .external_ids = stage_hint(lb._uuid)) :-
> r in &Router(),
> - r.l3dgw_port.is_some() or r.is_gateway,
> + r.l3dgw_ports.len() > 0 or r.is_gateway,
> LBVIPWithStatus[lbvip@&LBVIPWithStatus{.lb = lb}],
> r.load_balancer.contains(lb._uuid),
> var __match
> = "ct.new && " ++
> get_match_for_lb_key(lbvip.vip_addr, lbvip.vip_port, lb.protocol,
> true, true) ++
> - match (r.l3dgw_port) {
> - Some{gwport} -> " &&
> is_chassis_resident(${r.redirect_port_name})",
> + match (r.l3dgw_ports.nth(0)) {
> + Some{gw_port} -> " &&
> is_chassis_resident(${json_string_escape(chassis_redirect_name(gw_port.name))})",
> _ -> ""
> },
> var priority = if (lbvip.vip_port != 0) 120 else 110,
> @@ -7290,11 +7308,11 @@ Flow(.logical_datapath = router._uuid,
> .stage = s_ROUTER_IN_ARP_RESOLVE(),
> .priority = 50,
> .__match = "outport == ${rp.json_name} && "
> -
> "!is_chassis_resident(${router.redirect_port_name})",
> +
> "!is_chassis_resident(${json_string_escape(chassis_redirect_name(l3dgw_port.name))})",
> .actions = "eth.dst = ${rp.networks.ea}; next;",
> .external_ids = stage_hint(lrp._uuid)) :-
> rp in &RouterPort(.lrp = lrp, .router = router),
> - router.redirect_port_name != "",
> + Some{var l3dgw_port} = router.l3dgw_ports.nth(0),
> Some{"bridged"} = lrp.options.get("redirect-type").
>
>
> @@ -7537,9 +7555,8 @@ Flow(.logical_datapath = lr_uuid,
> "next;",
> .external_ids = stage_hint(l3dgw_port._uuid)) :-
> r in &Router(._uuid = lr_uuid),
> - Some{var l3dgw_port} = r.l3dgw_port,
> + Some{var l3dgw_port} = r.l3dgw_ports.nth(0),
> var l3dgw_port_json_name = json_string_escape(l3dgw_port.name),
> - r.redirect_port_name != "",
> var gw_mtu = l3dgw_port.options.get_int_def("gateway_mtu", 0),
> gw_mtu > 0,
> var mtu = gw_mtu + vLAN_ETH_HEADER_LEN().
> @@ -7564,9 +7581,8 @@ MeteredFlow(.logical_datapath = lr_uuid,
> .controller_meter = r.copp.get(cOPP_ICMP4_ERR()),
> .external_ids = stage_hint(rp.lrp._uuid)) :-
> r in &Router(._uuid = lr_uuid),
> - Some{var l3dgw_port} = r.l3dgw_port,
> + Some{var l3dgw_port} = r.l3dgw_ports.nth(0),
> var l3dgw_port_json_name = json_string_escape(l3dgw_port.name),
> - r.redirect_port_name != "",
> var gw_mtu = l3dgw_port.options.get_int_def("gateway_mtu", 0),
> gw_mtu > 0,
> rp in &RouterPort(.router = r),
> @@ -7593,9 +7609,8 @@ MeteredFlow(.logical_datapath = lr_uuid,
> .controller_meter = r.copp.get(cOPP_ICMP6_ERR()),
> .external_ids = stage_hint(rp.lrp._uuid)) :-
> r in &Router(._uuid = lr_uuid),
> - Some{var l3dgw_port} = r.l3dgw_port,
> + Some{var l3dgw_port} = r.l3dgw_ports.nth(0),
> var l3dgw_port_json_name = json_string_escape(l3dgw_port.name),
> - r.redirect_port_name != "",
> var gw_mtu = l3dgw_port.options.get_int_def("gateway_mtu", 0),
> gw_mtu > 0,
> rp in &RouterPort(.router = r),
> @@ -7609,21 +7624,20 @@ MeteredFlow(.logical_datapath = lr_uuid,
> * of the traffic to the l3redirect_port which represents
> * the central instance of the l3dgw_port.
> */
> -for (&Router(._uuid = lr_uuid,
> - .l3dgw_port = l3dgw_port,
> - .redirect_port_name = redirect_port_name))
> +for (&Router(._uuid = lr_uuid))
> {
> /* For traffic with outport == l3dgw_port, if the
> * packet did not match any higher priority redirect
> * rule, then the traffic is redirected to the central
> * instance of the l3dgw_port. */
> - Some{var gwport} = l3dgw_port in
> - Flow(.logical_datapath = lr_uuid,
> - .stage = s_ROUTER_IN_GW_REDIRECT(),
> - .priority = 50,
> - .__match = "outport == ${json_string_escape(gwport.name)}",
> - .actions = "outport = ${redirect_port_name}; next;",
> - .external_ids = stage_hint(gwport._uuid));
> + for (DistributedGatewayPort(lrp, lr_uuid)) {
> + Flow(.logical_datapath = lr_uuid,
> + .stage = s_ROUTER_IN_GW_REDIRECT(),
> + .priority = 50,
> + .__match = "outport ==
> ${json_string_escape(lrp.name)}",
> + .actions = "outport =
> ${json_string_escape(chassis_redirect_name(lrp.name))}; next;",
> + .external_ids = stage_hint(lrp._uuid))
> + };
>
> /* Packets are allowed by default. */
> Flow(.logical_datapath = lr_uuid,
> diff --git a/ovn-architecture.7.xml b/ovn-architecture.7.xml
> index 0eef9b739..3598b5073 100644
> --- a/ovn-architecture.7.xml
> +++ b/ovn-architecture.7.xml
> @@ -731,6 +731,13 @@
> highest-priority gateway that is online.
> </p>
>
> + <p>
> + A logical router can have multiple distributed gateway ports, each
> + connecting different external networks. However, some features, such as
> NAT
> + and load balancers, are not supported yet for logical routers with more
> + than one distributed gateway port configured.
> + </p>
> +
> <h4>Physical VLAN MTU Issues</h4>
>
> <p>
> @@ -1968,8 +1975,9 @@
>
> <p>
> If the logical router doesn't have a distributed gateway port connecting
> - to the localnet logical switch which provides external connectivity,
> - then this option is ignored by <code>OVN</code>.
> + to the localnet logical switch which provides external connectivity, or
> + if it has more than one distributed gateway ports, then this option is
> + ignored by <code>OVN</code>.
> </p>
>
> <p>
> @@ -2086,6 +2094,13 @@
> a tunnel.
> </p>
>
> + <p>
> + If the logical router doesn't have a distributed gateway port connecting
> + to the localnet logical switch which provides external connectivity, or
> + if it has more than one distributed gateway ports, then this option is
> + ignored by <code>OVN</code>.
> + </p>
> +
> <p>
> Following happens for bridged redirection:
> </p>
> diff --git a/ovn-nb.xml b/ovn-nb.xml
> index c1176e81f..ec51b5608 100644
> --- a/ovn-nb.xml
> +++ b/ovn-nb.xml
> @@ -2032,13 +2032,14 @@
>
> <column name="nat">
> One or more NAT rules for the router. NAT rules only work on
> - Gateway routers, and on distributed routers with logical gateway ports.
> + Gateway routers, and on distributed routers with one and only one
> + distributed gateway port.
> </column>
>
> <column name="load_balancer">
> Load balance a virtual ip address to a set of logical port ip
> addresses. Load balancer rules only work on the Gateway routers or
> - routers with distributed gateway ports.
> + routers with one and only one distributed gateway port.
> </column>
>
> <group title="Naming">
> @@ -2453,8 +2454,7 @@
> If either of these are set, this logical router port represents a
> distributed gateway port that connects this router to a
> logical switch with a <code>localnet</code> port or a
> - connection to another OVN deployment. There may be at most
> - one such logical router port on each logical router.
> + connection to another OVN deployment.
> </p>
>
> <p>
> @@ -2476,8 +2476,16 @@
> </p>
>
> <p>
> - When more than one gateway chassis is specified, OVN only uses
> - one at a time. OVN can rely on OVS BFD implementation to monitor
> + There can be more than one distributed gateway ports configured
> + on each logical router, each connecting to different L2 segments.
> + However, features such as NAT and load-balancer are not supported
> + on logical routers with more than one distributed gateway ports.
> + </p>
> +
> + <p>
> + For each distributed gateway port, it may have more than one gateway
> + chassises. When more than one gateway chassis is specified, OVN only
> + uses one at a time. OVN can rely on OVS BFD implementation to
> monitor
> gateway connectivity, preferring the highest-priority gateway
> that is online. Priorities are specified in the
> <code>priority</code>
> column of <ref table="Gateway_Chassis"/> or <ref
> table="HA_Chassis"/>.
> @@ -2563,8 +2571,8 @@
> </p>
>
> <p>
> - OVN honors this option only if the logical router has a
> distributed
> - gateway port and if the LRP's peer switch has a
> + OVN honors this option only if the logical router has one and
> only
> + one distributed gateway port and if the LRP's peer switch has a
> <code>localnet</code> port.
> </p>
> </column>
> @@ -2588,7 +2596,8 @@
> <p>
> Setting this option to <code>overlay</code> or leaving it unset
> has
> no effect. This option may usefully be set only on a distributed
> - gateway port. It is otherwise ignored.
> + gateway port when there is one and only one distributed gateway
> + port on the logical router. It is otherwise ignored.
> </p>
> </column>
> </group>
> diff --git a/ovs b/ovs
> index daf627f45..e6ad4d8d9 160000
> --- a/ovs
> +++ b/ovs
> @@ -1 +1 @@
> -Subproject commit daf627f459ffbc7171d42a2c01f80754bfd54edc
> +Subproject commit e6ad4d8d9c9273f226ec9a993b64fccfb50bdf4c
> diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at
> index 94405349e..dfb5b70f9 100644
> --- a/tests/ovn-northd.at
> +++ b/tests/ovn-northd.at
> @@ -4815,3 +4815,95 @@ AT_CHECK([ovn-sbctl --columns=tags list logical_flow |
> grep lsp0 -c], [0], [dnl
>
> AT_CLEANUP
> ])
> +
> +OVN_FOR_EACH_NORTHD([
> +AT_SETUP([ovn-northd -- lr multiple gw ports])
> +AT_KEYWORDS([multiple-l3dgw-ports])
> +ovn_start
> +
> +# Logical network:
> +# 1 Logical Router, 3 bridged Logical Switches,
> +# 1 gateway chassis attached to each corresponding LRP.
> +#
> +# | S1 (gw1)
> +# |
> +# ls ---- DR -- S3 (gw3)
> +# (20.0.0.0/24) |
> +# | S2 (gw2)
> +#
> +# Validate basic LR logical flows.
> +
> +check ovn-sbctl chassis-add gw1 geneve 127.0.0.1
> +check ovn-sbctl chassis-add gw2 geneve 128.0.0.1
> +check ovn-sbctl chassis-add gw3 geneve 129.0.0.1
> +
> +check ovn-nbctl lr-add DR
> +check ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24
> +check ovn-nbctl lrp-add DR DR-S2 03:ac:10:01:00:01 10.0.0.1/24
> +check ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.1/24
> +check ovn-nbctl lrp-add DR DR-ls 05:ac:10:01:00:01 20.0.0.1/24
> +
> +check ovn-nbctl ls-add S1
> +check ovn-nbctl lsp-add S1 S1-DR
> +check ovn-nbctl lsp-set-type S1-DR router
> +check ovn-nbctl lsp-set-addresses S1-DR router
> +check ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1
> +
> +check ovn-nbctl ls-add S2
> +check ovn-nbctl lsp-add S2 S2-DR
> +check ovn-nbctl lsp-set-type S2-DR router
> +check ovn-nbctl lsp-set-addresses S2-DR router
> +check ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2
> +
> +check ovn-nbctl ls-add S3
> +check ovn-nbctl lsp-add S3 S3-DR
> +check ovn-nbctl lsp-set-type S3-DR router
> +check ovn-nbctl lsp-set-addresses S3-DR router
> +check ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3
> +
> +check ovn-nbctl ls-add ls
> +check ovn-nbctl lsp-add ls ls-DR
> +check ovn-nbctl lsp-set-type ls-DR router
> +check ovn-nbctl lsp-set-addresses ls-DR router
> +check ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls
> +
> +check ovn-nbctl lrp-set-gateway-chassis DR-S1 gw1
> +check ovn-nbctl lrp-set-gateway-chassis DR-S2 gw2
> +check ovn-nbctl lrp-set-gateway-chassis DR-S3 gw3
> +
> +check ovn-nbctl --wait=sb sync
> +
> +ovn-sbctl dump-flows DR > lrflows
> +AT_CAPTURE_FILE([lrflows])
> +
> +# Check the flows in lr_in_admission stage
> +AT_CHECK([grep lr_in_admission lrflows | grep cr-DR | wc -l], [0], [3
> +])
> +AT_CHECK([grep lr_in_admission lrflows | grep cr-DR-S1 | wc -l], [0], [1
> +])
> +AT_CHECK([grep lr_in_admission lrflows | grep cr-DR-S2 | wc -l], [0], [1
> +])
> +AT_CHECK([grep lr_in_admission lrflows | grep cr-DR-S3 | wc -l], [0], [1
> +])
> +
I'd suggest to check for the exact flow match actions rather than using "wc -l".
This would ensure that both northd-c and northd-ddlog generate the
exact same flows.
> +# Check the flows in lr_in_lookup_neighbor stage
> +AT_CHECK([grep lr_in_lookup_neighbor lrflows | grep cr-DR | wc -l], [0], [3
> +])
> +AT_CHECK([grep lr_in_lookup_neighbor lrflows | grep cr-DR-S1 | wc -l], [0],
> [1
> +])
> +AT_CHECK([grep lr_in_lookup_neighbor lrflows | grep cr-DR-S2 | wc -l], [0],
> [1
> +])
> +AT_CHECK([grep lr_in_lookup_neighbor lrflows | grep cr-DR-S3 | wc -l], [0],
> [1
> +])
> +
> +# Check the flows in lr_in_gw_redirect stage
> +AT_CHECK([grep lr_in_gw_redirect lrflows | grep cr-DR | wc -l], [0], [3
> +])
> +AT_CHECK([grep lr_in_gw_redirect lrflows | grep cr-DR-S1 | wc -l], [0], [1
> +])
> +AT_CHECK([grep lr_in_gw_redirect lrflows | grep cr-DR-S2 | wc -l], [0], [1
> +])
> +AT_CHECK([grep lr_in_gw_redirect lrflows | grep cr-DR-S3 | wc -l], [0], [1
> +])
> +AT_CLEANUP
> +])
> diff --git a/tests/ovn.at b/tests/ovn.at
> index 13d860f10..13f2a4990 100644
> --- a/tests/ovn.at
> +++ b/tests/ovn.at
> @@ -27351,3 +27351,310 @@ AT_CHECK([ovs-ofctl dump-flows br-int table=44 |
> grep 10.0.0.144], [0], [ignore]
> OVN_CLEANUP([hv1])
> AT_CLEANUP
> ])
> +
> +OVN_FOR_EACH_NORTHD([
> +AT_SETUP([ovn -- lr multiple gw ports])
> +AT_KEYWORDS([multiple-l3dgw-ports])
> +ovn_start
> +
> +# Logical network:
> +# 1 LR, 3 Logical Switches,
> +# 1 gateway chassis attached to each corresponding LRP.
> +#
> +# | S1 (gw1)
> +# |
> +# ls ---- DR -- S3 (gw3)
> +# (20.0.0.0/24) |
> +# | S2 (gw2)
> +#
> +# S1 - VLAN 1000
> +# S2 - VLAN 2000
> +# S3 - VLAN 3000
> +#
> +# 5 chassis(s), HV1----HV5
> +#
> +# HV1 - VIF11
> +# HV2 - Gateway chassis gw1
> +# HV3 - Gateway chassis gw2
> +# HV4 - Gateway chassis gw3
> +# HV5 - North endpoint
> +
> +ovn-nbctl lr-add DR
> +ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24
> +ovn-nbctl lrp-add DR DR-S2 08:ac:10:01:00:01 10.0.0.1/24
> +ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.1/24
> +ovn-nbctl lrp-add DR DR-ls 06:ac:10:01:00:01 20.0.0.1/24
> +
> +ovn-nbctl ls-add S1
> +ovn-nbctl lsp-add S1 S1-DR
> +ovn-nbctl lsp-set-type S1-DR router
> +ovn-nbctl lsp-set-addresses S1-DR router
> +ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1
> +ovn-nbctl lsp-add S1 ln1 "" 1000
> +ovn-nbctl lsp-set-addresses ln1 unknown
> +ovn-nbctl lsp-set-type ln1 localnet
> +ovn-nbctl lsp-set-options ln1 network_name=phys
> +
> +ovn-nbctl ls-add S2
> +ovn-nbctl lsp-add S2 S2-DR
> +ovn-nbctl lsp-set-type S2-DR router
> +ovn-nbctl lsp-set-addresses S2-DR router
> +ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2
> +ovn-nbctl lsp-add S2 ln2 "" 2000
> +ovn-nbctl lsp-set-addresses ln2 unknown
> +ovn-nbctl lsp-set-type ln2 localnet
> +ovn-nbctl lsp-set-options ln2 network_name=phys
> +
> +ovn-nbctl ls-add S3
> +ovn-nbctl lsp-add S3 S3-DR
> +ovn-nbctl lsp-set-type S3-DR router
> +ovn-nbctl lsp-set-addresses S3-DR router
> +ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3
> +ovn-nbctl lsp-add S3 ln3 "" 3000
> +ovn-nbctl lsp-set-addresses ln3 unknown
> +ovn-nbctl lsp-set-type ln3 localnet
> +ovn-nbctl lsp-set-options ln3 network_name=phys
> +
> +ovn-nbctl ls-add ls
> +ovn-nbctl lsp-add ls ls-DR
> +ovn-nbctl lsp-set-type ls-DR router
> +ovn-nbctl lsp-set-addresses ls-DR router
> +ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls
> +
> +# Add the lsp lp11 to ls. This will map to VIF11.
> +ovn-nbctl lsp-add ls lp11
> +ovn-nbctl lsp-set-addresses lp11 "f0:00:00:00:00:10 20.0.0.10"
> +ovn-nbctl lsp-set-port-security lp11 f0:00:00:00:00:10
> +
> +# Add the Northbound endpoint, lp-north1
> +ovn-nbctl ls-add ls-north1
> +ovn-nbctl lsp-add ls-north1 ln4 "" 1000
> +ovn-nbctl lsp-set-addresses ln4 unknown
> +ovn-nbctl lsp-set-type ln4 localnet
> +ovn-nbctl lsp-set-options ln4 network_name=phys
> +
> +ovn-nbctl lsp-add ls-north1 lp-north1
> +ovn-nbctl lsp-set-addresses lp-north1 "f0:f0:00:00:00:11 172.16.1.10"
> +ovn-nbctl lsp-set-port-security lp-north1 f0:f0:00:00:00:11
> +
> +# Add the Northbound endpoint, lp-north2
> +ovn-nbctl ls-add ls-north2
> +ovn-nbctl lsp-add ls-north2 ln5 "" 2000
> +ovn-nbctl lsp-set-addresses ln5 unknown
> +ovn-nbctl lsp-set-type ln5 localnet
> +ovn-nbctl lsp-set-options ln5 network_name=phys
> +
> +ovn-nbctl lsp-add ls-north2 lp-north2
> +ovn-nbctl lsp-set-addresses lp-north2 "f0:f0:00:00:00:22 10.0.0.10"
> +ovn-nbctl lsp-set-port-security lp-north2 f0:f0:00:00:00:22
> +
> +# Add the Northbound endpoint, lp-north3
> +ovn-nbctl ls-add ls-north3
> +ovn-nbctl lsp-add ls-north3 ln6 "" 3000
> +ovn-nbctl lsp-set-addresses ln6 unknown
> +ovn-nbctl lsp-set-type ln6 localnet
> +ovn-nbctl lsp-set-options ln6 network_name=phys
> +
> +ovn-nbctl lsp-add ls-north3 lp-north3
> +ovn-nbctl lsp-set-addresses lp-north3 "f0:f0:00:00:00:33 192.168.0.10"
> +ovn-nbctl lsp-set-port-security lp-north3 f0:f0:00:00:00:33
> +
> +# Add 5 chassis
> +net_add n1
> +for i in 1 2 3 4 5; do
> + sim_add hv$i
> + as hv$i
> + ovs-vsctl add-br br-phys
> + ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys
> + ovn_attach n1 br-phys 192.168.0.$i 24 $encap
> +done
> +
> +# Add a vif on HV1
> +as hv1 ovs-vsctl add-port br-int vif11 -- \
> + set Interface vif11 external-ids:iface-id=lp11 \
> + options:tx_pcap=hv1/vif11-tx.pcap \
> + options:rxq_pcap=hv1/vif11-rx.pcap \
> + ofport-request=11
> +OVS_WAIT_UNTIL([test x`ovn-nbctl lsp-get-up lp11` = xup])
> +
> +as hv5 ovs-vsctl add-port br-int vif-north1 -- \
> + set Interface vif-north1 external-ids:iface-id=lp-north1 \
> + options:tx_pcap=hv5/vif-north1-tx.pcap \
> + options:rxq_pcap=hv5/vif-north1-rx.pcap \
> + ofport-request=44
> +
> +as hv5 ovs-vsctl add-port br-int vif-north2 -- \
> + set Interface vif-north2 external-ids:iface-id=lp-north2 \
> + options:tx_pcap=hv5/vif-north2-tx.pcap \
> + options:rxq_pcap=hv5/vif-north2-rx.pcap \
> + ofport-request=45
> +
> +as hv5 ovs-vsctl add-port br-int vif-north3 -- \
> + set Interface vif-north3 external-ids:iface-id=lp-north3 \
> + options:tx_pcap=hv5/vif-north3-tx.pcap \
> + options:rxq_pcap=hv5/vif-north3-rx.pcap \
> + ofport-request=46
> +
> +ovn-nbctl lrp-set-gateway-chassis DR-S1 hv2
> +ovn-nbctl lrp-set-gateway-chassis DR-S2 hv3
> +ovn-nbctl lrp-set-gateway-chassis DR-S3 hv4
> +
> +ovn-nbctl --wait=sb sync
> +OVN_POPULATE_ARP
> +
> +vif_to_ls () {
> + case ${1} in dnl (
> + vif?[[11]]) echo ls ;; dnl (
> + vif-north1) echo ls-north1 ;; dnl (
> + vif-north2) echo ls-north2 ;; dnl (
> + vif-north3) echo ls-north3 ;; dnl (
> + *) AT_FAIL_IF([:]) ;;
> + esac
> +}
> +
> +vif_to_hv () {
> + case ${1} in dnl (
> + vif[[1]]?) echo hv1 ;; dnl (
> + vif-north1) echo hv5 ;; dnl (
> + vif-north2) echo hv5 ;; dnl (
> + vif-north3) echo hv5 ;; dnl (
> + *) AT_FAIL_IF([:]) ;;
> + esac
> +}
> +
> +vif_to_lrp () {
> + case ${1} in dnl (
> + vif?[[11]]) echo DR-ls ;; dnl (
> + *) AT_FAIL_IF([:]) ;;
> + esac
> +
> +}
> +
> +ip_to_hex() {
> + printf "%02x%02x%02x%02x" "${@}"
> +}
> +
> +# test_arp INPORT SHA SPA TPA
> +#
> +# Causes a packet to be received on INPORT. The packet is an ARP
> +# request with SHA, SPA, and TPA as specified.
> +test_arp() {
> + local inport=$1 sha=$2 spa=$3 tpa=$4
> + local
> request=ffffffffffff${sha}08060001080006040001${sha}${spa}ffffffffffff${tpa}
> + hv=`vif_to_hv $inport`
> + as $hv ovs-appctl netdev-dummy/receive $inport $request
> +}
> +
> +
> +test_ip() {
> + # This packet has bad checksums but logical L3 routing doesn't check.
> + local inport=${1} src_mac=${2} dst_mac=${3} src_ip=${4} dst_ip=${5}
> outport=${6}
> + local
> packet=${dst_mac}${src_mac}08004500001c0000000040110000${src_ip}${dst_ip}0035111100080000
> + shift; shift; shift; shift; shift
> + hv=`vif_to_hv $inport`
> + as $hv ovs-appctl netdev-dummy/receive $inport $packet
> + in_ls=`vif_to_ls $inport`
> + for outport; do
> + out_ls=`vif_to_ls $outport`
> + if test $in_ls = $out_ls; then
> + # Ports on the same logical switch receive exactly the same
> packet.
> + echo $packet
> + else
> + # Routing decrements TTL and updates source and dest MAC
> + # (and checksum).
> + # For North-South, packet will come via gateway chassis, i.e
> hv3
> + if test $inport = vif-north1; then
> + echo
> f0000000001006ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000
> >> $outport.expected
> + fi
> + if test $outport = vif-north1; then
> + echo
> f0f00000001102ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000
> >> $outport.expected
> + fi
> + if test $outport = vif-north2; then
> + echo
> f0f00000002208ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000
> >> $outport.expected
> + fi
> + if test $outport = vif-north3; then
> + echo
> f0f00000003304ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000
> >> $outport.expected
> + fi
> + fi >> $outport.expected
> + done
> +}
> +
> +echo "------ OVN dump ------"
> +ovn-nbctl show
> +ovn-sbctl show
> +ovn-sbctl list port_binding
> +ovn-sbctl list mac_binding
> +ovn-sbctl list datapath_binding
> +
> +ovn-sbctl dump-flows DR
> +ovn-sbctl dump-flows S1
> +ovn-sbctl dump-flows ls
> +
> +echo "------ hv1 dump ------"
> +as hv1 ovs-vsctl show
> +as hv1 ovs-vsctl list Open_Vswitch
> +as hv1 ovs-ofctl dump-flows br-int
> +
> +echo "------ hv2 dump ------"
> +as hv2 ovs-vsctl show
> +as hv2 ovs-vsctl list Open_Vswitch
> +as hv2 ovs-ofctl dump-flows br-int
> +
> +echo "------ hv3 dump ------"
> +as hv3 ovs-vsctl show
> +as hv3 ovs-vsctl list Open_Vswitch
> +as hv3 ovs-ofctl dump-flows br-int
> +
> +echo "------ hv4 dump ------"
> +as hv4 ovs-vsctl show
> +as hv4 ovs-vsctl list Open_Vswitch
> +as hv5 ovs-ofctl dump-flows br-int
> +
> +# N-S with lp-north1
> +echo "Send Dummy ARP"
> +sip=`ip_to_hex 172 16 1 10`
> +tip=`ip_to_hex 172 16 1 50`
> +test_arp vif-north1 f0f000000011 $sip $tip
> +
> +echo "Send traffic North to South"
> +sip=`ip_to_hex 172 16 1 10`
> +dip=`ip_to_hex 20 0 0 10`
> +test_ip vif-north1 f0f000000011 02ac10010001 $sip $dip vif11
> +# Confirm that North to south traffic works fine.
> +OVN_CHECK_PACKETS_REMOVE_BROADCAST([hv1/vif11-tx.pcap], [vif11.expected])
> +
> +echo "Send traffic South to North1"
> +sip=`ip_to_hex 20 0 0 10`
> +dip=`ip_to_hex 172 16 1 10`
> +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north1
> +# Confirm that South to North traffic works fine.
> +OVN_CHECK_PACKETS_REMOVE_BROADCAST([hv5/vif-north1-tx.pcap],
> [vif-north1.expected])
> +
> +# N-S with lp-north2
> +echo "Send Dummy ARP"
> +sip=`ip_to_hex 10 0 0 10`
> +tip=`ip_to_hex 10 0 0 50`
> +test_arp vif-north2 f0f000000022 $sip $tip
> +
> +echo "Send traffic South to North2"
> +sip=`ip_to_hex 20 0 0 10`
> +dip=`ip_to_hex 10 0 0 10`
> +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north2
> +# Confirm that South to North traffic works fine.
> +OVN_CHECK_PACKETS_REMOVE_BROADCAST([hv5/vif-north2-tx.pcap],
> [vif-north2.expected])
> +
> +# N-S with lp-north3
> +echo "Send Dummy ARP"
> +sip=`ip_to_hex 192 168 0 10`
> +tip=`ip_to_hex 192 168 0 50`
> +test_arp vif-north3 f0f000000033 $sip $tip
> +
> +echo "Send traffic South to North3"
> +sip=`ip_to_hex 20 0 0 10`
> +dip=`ip_to_hex 192 168 0 10`
> +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north3
> +# Confirm that South to North traffic works fine.
> +OVN_CHECK_PACKETS_REMOVE_BROADCAST([hv5/vif-north3-tx.pcap],
> [vif-north3.expected])
> +
> +AT_CLEANUP
> +])
> --
> 2.30.2
>
> _______________________________________________
> dev mailing list
> [email protected]
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
