On Tue, Aug 3, 2021 at 1:55 PM Han Zhou <[email protected]> wrote:
>
> On Tue, Aug 3, 2021 at 10:14 AM Numan Siddique <[email protected]> wrote:
> >
> > On Thu, Jul 29, 2021 at 4:03 AM Han Zhou <[email protected]> wrote:
> > >
> > > From: Ankur Sharma <[email protected]>
> > >
> > > By default, OVN support only one DGP (distributed gateway port) per
> > > logical router. While a single DGP port suffices for most of the North
> > > South connectivity, there are requirements where a logical router could
> > > be connected to multiple external networks and based on routing decision
> > > packet could go to different ones.
> > >
> > > This patch adds flexibility of having multiple DGPs per logical router.
> > >
> > > Changes can classified as following:
> > > a. Data structure changes to allow multiple DGPs per ovn_datapath.
> > >
> > > b. Consumption of new data structure in logical flows for
> > > individual features.
> > >
> > > c. Features that require changes are:
> > > i. Regular NS traffic flow.
> > > ii. Network Address Translation.
> > > iii. Load Balancer
> > > iv. Gateway_mtu.
> > > v. reside-on-redirect-chassis
> > > vi. Misc code sections that assumed a single DGP.
> > >
> > > d. Except for reside-on-redirect-chassis all the other features
> > > could be extended to multiple DGPs. Reside on redirect
> > > chassis with its current specification could not be extended
> > > and hence should be used only with the logical router that
> > > has a single DGP.
> > >
> > > This patch doesn't support NAT & load-balancer features for multiple
> > > DGPs yet, but added validations that disables NAT/load-balancer
> > > features when there are more than one DGP configured per router.
> > >
> > > Signed-off-by: Ankur Sharma <[email protected]>
> > > Co-authored-by: Dhathri Purohith <[email protected]>
> > > Signed-off-by: Dhathri Purohith <[email protected]>
> > > Co-authored-by: Abhiram Sangana <[email protected]>
> > > Signed-off-by: Abhiram Sangana <[email protected]>
> > > Co-authored-by: Han Zhou <[email protected]>
> > > Signed-off-by: Han Zhou <[email protected]>
> >
> > Hi Han, Ankur et all.
> >
> > Thanks for adding this feature.
> >
> > Please see below for few comments.
> >
> > Also this patch needs a rebase.
> >
> > Thanks
> > Numan
> >
>
> Thanks Numan for the review!
>
> > > ---
> > > NEWS | 3 +
> > > northd/lrouter.dl | 103 +++-----
> > > northd/ovn-northd.8.xml | 6 +-
> > > northd/ovn-northd.c | 535 ++++++++++++++++++++++------------------
> > > northd/ovn_northd.dl | 178 +++++++------
> > > ovn-architecture.7.xml | 19 +-
> > > ovn-nb.xml | 27 +-
> > > ovs | 2 +-
> > > tests/ovn-northd.at | 92 +++++++
> > > tests/ovn.at | 307 +++++++++++++++++++++++
> > > 10 files changed, 870 insertions(+), 402 deletions(-)
> > >
> > > diff --git a/NEWS b/NEWS
> > > index eefdae9bc..2a1c56b2d 100644
> > > --- a/NEWS
> > > +++ b/NEWS
> > > @@ -33,6 +33,9 @@ OVN v21.06.0 - 18 Jun 2021
> > > "ovn-trim-limit-lflow-cache" and
> "ovn-trim-wmark-perc-lflow-cache", to
> > > allow enforcing a lflow cache size limit and high watermark
> percentage
> > > for which automatic memory trimming is performed.
> > > + - Support multiple distributed gateway ports on a single logical
> router.
> > > + (NAT and load-balancer are not supported yet when there are
> multiple
> > > + distributed gateway ports).
> > >
> > > OVN v21.03.0 - 12 Mar 2021
> > > -------------------------
> > > diff --git a/northd/lrouter.dl b/northd/lrouter.dl
> > > index 4a24f3f61..d37350ab8 100644
> > > --- a/northd/lrouter.dl
> > > +++ b/northd/lrouter.dl
> > > @@ -138,14 +138,14 @@ Warning[message] :-
> > > var message = "Bad configuration: distributed gateway port
> configured on "
> > > "port ${lrp.name} on L3 gateway router".
> > >
> > > -/* DistributedGatewayPortCandidate.
> > > +/* Distributed gateway ports.
> > > *
> > > - * Each row pairs a logical router with its distributed gateway port,
> > > - * but without checking that there is at most one DGP per LR.
> > > + * Each row means 'lrp' is a distributed gateway port on 'lr_uuid'.
> > > *
> > > - * (Use DistributedGatewayPort instead, since it guarantees
> uniqueness.) */
> > > -relation DistributedGatewayPortCandidate(lr_uuid: uuid, lrp_uuid: uuid)
> > > -DistributedGatewayPortCandidate(lr_uuid, lrp_uuid) :-
> > > + * A logical router can have multiple distributed gateway ports. */
> > > +relation DistributedGatewayPort(lrp: Intern<nb::Logical_Router_Port>,
> > > + lr_uuid: uuid)
> > > +DistributedGatewayPort(lrp, lr_uuid) :-
> > > lr in nb::Logical_Router(._uuid = lr_uuid),
> > > LogicalRouterPort(lrp_uuid, lr._uuid),
> > > lrp in &nb::Logical_Router_Port(._uuid = lrp_uuid),
> > > @@ -153,30 +153,10 @@ DistributedGatewayPortCandidate(lr_uuid,
> lrp_uuid) :-
> > > var has_hcg = lrp.ha_chassis_group.is_some(),
> > > var has_gc = not lrp.gateway_chassis.is_empty(),
> > > has_hcg or has_gc.
> > > -Warning[message] :-
> > > - DistributedGatewayPortCandidate(lr_uuid, lrp_uuid),
> > > - var lrps = lrp_uuid.group_by(lr_uuid).to_set(),
> > > - lrps.size() > 1,
> > > - lr in nb::Logical_Router(._uuid = lr_uuid),
> > > - var message = "Bad configuration: multiple distributed gateway
> ports on "
> > > - "logical router ${lr.name}; ignoring all of them".
> > > -
> > > -/* Distributed gateway ports.
> > > - *
> > > - * Each row means 'lrp' is the distributed gateway port on 'lr_uuid'.
> > > - *
> > > - * There is at most one distributed gateway port per logical router. */
> > > -relation DistributedGatewayPort(lrp: Intern<nb::Logical_Router_Port>,
> lr_uuid: uuid)
> > > -DistributedGatewayPort(lrp, lr_uuid) :-
> > > - DistributedGatewayPortCandidate(lr_uuid, lrp_uuid),
> > > - var lrps = lrp_uuid.group_by(lr_uuid).to_set(),
> > > - lrps.size() == 1,
> > > - Some{var lrp_uuid} = lrps.nth(0),
> > > - lrp in &nb::Logical_Router_Port(._uuid = lrp_uuid).
> > >
> > > /* HAChassis is an abstraction over nb::Gateway_Chassis and
> nb::HA_Chassis, which
> > > * are different ways to represent the same configuration. Each row is
> > > - * effectively one HA_Chassis record. (Usually, we could associated
> each
> > > + * effectively one HA_Chassis record. (Usually, we could associate
> each
> > > * row with a particular 'lr_uuid', but it's permissible for more than
> one
> > > * logical router to use a HA chassis group, so we omit it so that
> multiple
> > > * references get merged.)
> > > @@ -236,18 +216,20 @@
> HAChassisGroup(ha_chassis_group_uuid(hac_group_uuid),
> > > .name = name,
> > > .external_ids = external_ids).
> > >
> > > -/* Each row maps from a logical router to the name of its
> HAChassisGroup.
> > > - * This level of indirection is needed because multiple logical routers
> > > - * are allowed to reference a given HAChassisGroup. */
> > > -relation LogicalRouterHAChassisGroup(lr_uuid: uuid,
> > > - hacg_uuid: uuid)
> > > -LogicalRouterHAChassisGroup(lr_uuid, ha_chassis_group_uuid(lrp._uuid))
> :-
> > > - DistributedGatewayPort(lrp, lr_uuid),
> > > +/* Each row maps from a distributed gateway logical router port to the
> name of
> > > + * its HAChassisGroup.
> > > + * This level of indirection is needed because multiple distributed
> gateway
> > > + * logical router ports are allowed to reference a given
> HAChassisGroup. */
> > > +relation DistributedGatewayPortHAChassisGroup(
> > > + lrp: Intern<nb::Logical_Router_Port>,
> > > + hacg_uuid: uuid)
> > > +DistributedGatewayPortHAChassisGroup(lrp,
> ha_chassis_group_uuid(lrp._uuid)) :-
> > > + DistributedGatewayPort(.lrp = lrp),
> > > lrp.ha_chassis_group == None,
> > > lrp.gateway_chassis.size() > 0.
> > > -LogicalRouterHAChassisGroup(lr_uuid,
> > > - ha_chassis_group_uuid(hac_group_uuid)) :-
> > > - DistributedGatewayPort(lrp, lr_uuid),
> > > +DistributedGatewayPortHAChassisGroup(lrp,
> > > +
> ha_chassis_group_uuid(hac_group_uuid)) :-
> > > + DistributedGatewayPort(.lrp = lrp),
> > > Some{var hac_group_uuid} = lrp.ha_chassis_group,
> > > nb::HA_Chassis_Group(._uuid = hac_group_uuid).
> > >
> > > @@ -259,14 +241,19 @@ RouterPortIsRedirect(lrp, false) :-
> > > &nb::Logical_Router_Port(._uuid = lrp),
> > > not DistributedGatewayPort(&nb::Logical_Router_Port{._uuid = lrp},
> _).
> > >
> > > -relation LogicalRouterRedirectPort(lr: uuid, has_redirect_port:
> Option<Intern<nb::Logical_Router_Port>>)
> > > -
> > > -LogicalRouterRedirectPort(lr, Some{lrp}) :-
> > > - DistributedGatewayPort(lrp, lr).
> > > -
> > > -LogicalRouterRedirectPort(lr, None) :-
> > > - nb::Logical_Router(._uuid = lr),
> > > - not DistributedGatewayPort(_, lr).
> > > +/*
> > > + * LogicalRouterDGWPorts maps from each logical router UUID
> > > + * to the logical router's set of distributed gateway (or redirect)
> ports. */
> > > +relation LogicalRouterDGWPorts(
> > > + lr_uuid: uuid,
> > > + l3dgw_ports: Vec<Intern<nb::Logical_Router_Port>>)
> > > +LogicalRouterDGWPorts(lr_uuid, l3dgw_ports) :-
> > > + DistributedGatewayPort(lrp, lr_uuid),
> > > + var l3dgw_ports = lrp.group_by(lr_uuid).to_vec().
> > > +LogicalRouterDGWPorts(lr_uuid, vec_empty()) :-
> > > + lr in nb::Logical_Router(),
> > > + var lr_uuid = lr._uuid,
> > > + not DistributedGatewayPort(_, lr_uuid).
> > >
> > > typedef ExceptionalExtIps = AllowedExtIps{ips: Intern<nb::Address_Set>}
> > > | ExemptedExtIps{ips:
> Intern<nb::Address_Set>}
> > > @@ -450,9 +437,7 @@ LogicalRouterCopp0(lr, meters) :-
> > >
> > > /* Router relation collects all attributes of a logical router.
> > > *
> > > - * `l3dgw_port` - optional redirect port (see `DistributedGatewayPort`)
> > > - * `redirect_port_name` - derived redirect port name (or empty string
> if
> > > - * router does not have a redirect port)
> > > + * `l3dgw_ports` - optional redirect ports (see
> `DistributedGatewayPort`)
> > > * `is_gateway` - true iff the router is a gateway router. Together
> with
> > > * `l3dgw_port`, this flag affects the generation of various flows
> > > * related to NAT and load balancing.
> > > @@ -474,8 +459,7 @@ typedef Router = Router {
> > > external_ids: Map<string,string>,
> > >
> > > /* Additional computed fields. */
> > > - l3dgw_port: Option<Intern<nb::Logical_Router_Port>>,
> > > - redirect_port_name: string,
> > > + l3dgw_ports: Vec<Intern<nb::Logical_Router_Port>>,
> > > is_gateway: bool,
> > > nats: Vec<NAT>,
> > > snat_ips: Map<v46_ip, Set<NAT>>,
> > > @@ -498,23 +482,18 @@ Router[Router{
> > > .options = lr.options,
> > > .external_ids = lr.external_ids,
> > >
> > > - .l3dgw_port = l3dgw_port,
> > > - .redirect_port_name =
> > > - match (l3dgw_port) {
> > > - Some{rport} ->
> json_string_escape(chassis_redirect_name(rport.name)),
> > > - _ -> ""
> > > - },
> > > - .is_gateway = lr.options.contains_key("chassis"),
> > > - .nats = nats,
> > > - .snat_ips = snat_ips,
> > > - .lbs = lbs,
> > > - .mcast_cfg = mcast_cfg,
> > > + .l3dgw_ports = l3dgw_ports,
> > > + .is_gateway = lr.options.contains_key("chassis"),
> > > + .nats = nats,
> > > + .snat_ips = snat_ips,
> > > + .lbs = lbs,
> > > + .mcast_cfg = mcast_cfg,
> > > .learn_from_arp_request = learn_from_arp_request,
> > > .force_lb_snat = force_lb_snat,
> > > .copp = copp}.intern()] :-
> > > lr in nb::Logical_Router(),
> > > lr.is_enabled(),
> > > - LogicalRouterRedirectPort(lr._uuid, l3dgw_port),
> > > + LogicalRouterDGWPorts(lr._uuid, l3dgw_ports),
> > > LogicalRouterNATs(lr._uuid, nats),
> > > LogicalRouterLBs(lr._uuid, lbs),
> > > LogicalRouterSnatIPs(lr._uuid, snat_ips),
> > > diff --git a/northd/ovn-northd.8.xml b/northd/ovn-northd.8.xml
> > > index 99a19f853..8a368ef61 100644
> > > --- a/northd/ovn-northd.8.xml
> > > +++ b/northd/ovn-northd.8.xml
> > > @@ -3764,10 +3764,10 @@ icmp6 {
> > > <h3>Ingress Table 17: Gateway Redirect</h3>
> > >
> > > <p>
> > > - For distributed logical routers where one of the logical router
> > > + For distributed logical routers where one or more of the logical
> router
> > > ports specifies a gateway chassis, this table redirects
> > > - certain packets to the distributed gateway port instance on the
> > > - gateway chassis. This table has the following flows:
> > > + certain packets to the distributed gateway port instances on the
> > > + gateway chassises. This table has the following flows:
> > > </p>
> > >
> > > <ul>
> > > diff --git a/northd/ovn-northd.c b/northd/ovn-northd.c
> > > index ebe12cace..87c4478fa 100644
> > > --- a/northd/ovn-northd.c
> > > +++ b/northd/ovn-northd.c
> > > @@ -655,13 +655,12 @@ struct ovn_datapath {
> > > bool is_gw_router;
> > >
> > > /* OVN northd only needs to know about the logical router gateway
> port for
> > > - * NAT on a distributed router. This "distributed gateway port" is
> > > - * populated only when there is a gateway chassis specified for
> one of
> > > - * the ports on the logical router. Otherwise this will be NULL.
> */
> > > - struct ovn_port *l3dgw_port;
> > > - /* The "derived" OVN port representing the instance of l3dgw_port
> on
> > > - * the gateway chassis. */
> > > - struct ovn_port *l3redirect_port;
> > > + * NAT on a distributed router. The "distributed gateway ports"
> are
> > > + * populated only when there is a gateway chassis or ha chassis
> group
> > > + * specified for some of the ports on the logical router.
> Otherwise this
> > > + * will be NULL. */
> > > + struct ovn_port **l3dgw_ports;
> > > + size_t n_l3dgw_ports;
> > >
> > > /* NAT entries configured on the router. */
> > > struct ovn_nat *nat_entries;
> > > @@ -802,6 +801,16 @@ init_nat_entries(struct ovn_datapath *od)
> > > return;
> > > }
> > >
> > > + if (od->n_l3dgw_ports > 1) {
> > > + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1);
> > > + VLOG_WARN_RL(&rl, "NAT is configured on logical router %s,
> which has %"
> > > + PRIuSIZE" distributed gateway ports. NAT is not
> supported"
> > > + " yet when there is more than one distributed
> gateway "
> > > + "port on the router.",
> > > + od->nbr->name, od->n_l3dgw_ports);
> > > + return;
> > > + }
> > > +
> > > od->nat_entries = xmalloc(od->nbr->n_nat * sizeof
> *od->nat_entries);
> > >
> > > for (size_t i = 0; i < od->nbr->n_nat; i++) {
> > > @@ -941,6 +950,7 @@ ovn_datapath_destroy(struct hmap *datapaths, struct
> ovn_datapath *od)
> > > destroy_lb_ips(od);
> > > free(od->nat_entries);
> > > free(od->localnet_ports);
> > > + free(od->l3dgw_ports);
> > > ovn_ls_port_group_destroy(&od->nb_pgs);
> > > destroy_mcast_info_for_datapath(od);
> > >
> > > @@ -1489,9 +1499,18 @@ struct ovn_port {
> > > /* Logical port multicast data. */
> > > struct mcast_port_info mcast_info;
> > >
> > > - /* This is ordinarily false. It is true if and only if this
> ovn_port is
> > > - * derived from a chassis-redirect port. */
> > > - bool derived;
> > > + /* At most one of l3dgw_port and cr_port can be not NULL. */
> > > +
> > > + /* This is set to a distributed gateway port if and only if this
> ovn_port
> > > + * is "derived" from it. Otherwise this is set to NULL. The derived
> > > + * ovn_port represents the instance of distributed gateway port on
> the
> > > + * gateway chassis.*/
> > > + struct ovn_port *l3dgw_port;
> > > +
> > > + /* This is set to the "derived" chassis-redirect port of this port
> if and
> > > + * only if this port is a distributed gateway port. Otherwise this
> is set
> > > + * to NULL. */
> > > + struct ovn_port *cr_port;
> >
> > In my opinion, the above 2 variables - 'l3dgw_port' and 'cr_port' are
> > confusing.
> >
> > For the code - if(op->l3dgw_port), It is not immediately obvious to
> > me that "op" is a derived
> > "chassisredirect" port.
> >
> > I'd suggest to have the bool - derived to be preserved.
> >
>
> OK, maybe a more specific name: is_derived_crp? is_ makes it clear from the
> name it is a bool, and _crp in case there may be other forms of derived
> port in the future.
>
> > How about this ?
> >
> > bool derived;
> > union {
> > struct ovn_port *cr_port;
> > struct ovn_port *l3gw_port;
> > };
> >
> > If 'derived' is true then 'l3gw_port' would be not NULL. Otherwise it
> > would be NULL.
>
> If it is a union, it won't be NULL even if "derived" is false when it is a
> DGP.
> But I think using union is good.
>
> >
> > Instance of 'struct ovn_port' for distributed router port would have
> > "cr_port" set.
> >
> >
> > >
> > > bool has_unknown; /* If the addresses have 'unknown' defined. */
> > >
> > > @@ -1578,7 +1597,7 @@ ovn_port_create(struct hmap *ports, const char
> *key,
> > > op->key = xstrdup(key);
> > > op->sb = sb;
> > > ovn_port_set_nb(op, nbsp, nbrp);
> > > - op->derived = false;
> > > + op->l3dgw_port = op->cr_port = NULL;
> > > hmap_insert(ports, &op->key_node, hash_string(op->key, 0));
> > > return op;
> > > }
> > > @@ -1682,7 +1701,7 @@ lrport_is_enabled(const struct
> nbrec_logical_router_port *lrport)
> > > static struct ovn_port *
> > > ovn_port_get_peer(struct hmap *ports, struct ovn_port *op)
> > > {
> > > - if (!op->nbsp || !lsp_is_router(op->nbsp) || op->derived) {
> > > + if (!op->nbsp || !lsp_is_router(op->nbsp) || op->l3dgw_port) {
> > > return NULL;
> > > }
> > >
> > > @@ -2426,6 +2445,7 @@ join_logical_ports(struct northd_context *ctx,
> > > tag_alloc_add_existing_tags(tag_alloc_table, nbsp);
> > > }
> > > } else {
> > > + size_t n_allocated_l3dgw_ports = 0;
> > > for (size_t i = 0; i < od->nbr->n_ports; i++) {
> > > const struct nbrec_logical_router_port *nbrp
> > > = od->nbr->ports[i];
> > > @@ -2481,36 +2501,32 @@ join_logical_ports(struct northd_context *ctx,
> > > "on L3 gateway router",
> nbrp->name);
> > > continue;
> > > }
> > > - if (od->l3dgw_port || od->l3redirect_port) {
> > > - static struct vlog_rate_limit rl
> > > - = VLOG_RATE_LIMIT_INIT(1, 1);
> > > - VLOG_WARN_RL(&rl, "Bad configuration: multiple
> "
> > > - "distributed gateway ports on
> logical "
> > > - "router %s", od->nbr->name);
> > > - continue;
> > > - }
> > >
> > > char *redirect_name =
> > > ovn_chassis_redirect_name(nbrp->name);
> > > struct ovn_port *crp = ovn_port_find(ports,
> redirect_name);
> > > if (crp && crp->sb && crp->sb->datapath == od->sb)
> {
> > > - crp->derived = true;
> > > ovn_port_set_nb(crp, NULL, nbrp);
> > > ovs_list_remove(&crp->list);
> > > ovs_list_push_back(both, &crp->list);
> > > } else {
> > > crp = ovn_port_create(ports, redirect_name,
> > > NULL, nbrp, NULL);
> > > - crp->derived = true;
> > > ovs_list_push_back(nb_only, &crp->list);
> > > }
> > > + crp->l3dgw_port = op;
> > > + op->cr_port = crp;
> > > crp->od = od;
> > > free(redirect_name);
> > >
> > > - /* Set l3dgw_port and l3redirect_port in od, for
> later
> > > - * use during flow creation. */
> > > - od->l3dgw_port = op;
> > > - od->l3redirect_port = crp;
> > > + /* Add to l3dgw_ports in od, for later use during
> flow
> > > + * creation. */
> > > + if (od->n_l3dgw_ports == n_allocated_l3dgw_ports) {
> > > + od->l3dgw_ports = x2nrealloc(od->l3dgw_ports,
> > > +
> &n_allocated_l3dgw_ports,
> > > + sizeof
> *od->l3dgw_ports);
> > > + }
> > > + od->l3dgw_ports[od->n_l3dgw_ports++] = op;
> > >
> > > assign_routable_addresses(op);
> > > }
> > > @@ -2522,7 +2538,7 @@ join_logical_ports(struct northd_context *ctx,
> > > * to their peers. */
> > > struct ovn_port *op;
> > > HMAP_FOR_EACH (op, key_node, ports) {
> > > - if (op->nbsp && lsp_is_router(op->nbsp) && !op->derived) {
> > > + if (op->nbsp && lsp_is_router(op->nbsp) && !op->l3dgw_port) {
> > > struct ovn_port *peer = ovn_port_get_peer(ports, op);
> > > if (!peer || !peer->nbrp) {
> > > continue;
> > > @@ -2553,7 +2569,7 @@ join_logical_ports(struct northd_context *ctx,
> > > if (peer->od && peer->od->mcast_info.rtr.relay) {
> > > op->od->mcast_info.sw.flood_relay = true;
> > > }
> > > - } else if (op->nbrp && op->nbrp->peer && !op->derived) {
> > > + } else if (op->nbrp && op->nbrp->peer && !op->l3dgw_port) {
> > > struct ovn_port *peer = ovn_port_find(ports,
> op->nbrp->peer);
> > > if (peer) {
> > > if (peer->nbrp) {
> > > @@ -2598,7 +2614,8 @@ get_nat_addresses(const struct ovn_port *op,
> size_t *n, bool routable_only)
> > > struct eth_addr mac;
> > > if (!op || !op->nbrp || !op->od || !op->od->nbr
> > > || (!op->od->nbr->n_nat && !op->od->nbr->n_load_balancer)
> > > - || !eth_addr_from_string(op->nbrp->mac, &mac)) {
> > > + || !eth_addr_from_string(op->nbrp->mac, &mac)
> > > + || op->od->n_l3dgw_ports > 1) {
> > > *n = n_nats;
> > > return NULL;
> > > }
> > > @@ -2629,7 +2646,7 @@ get_nat_addresses(const struct ovn_port *op,
> size_t *n, bool routable_only)
> > >
> > > /* Determine whether this NAT rule satisfies the conditions for
> > > * distributed NAT processing. */
> > > - if (op->od->l3redirect_port && !strcmp(nat->type,
> "dnat_and_snat")
> > > + if (op->od->l3dgw_ports && !strcmp(nat->type, "dnat_and_snat")
> >
> > small nit: I'd suggest to use - 'op->od->n_l3dgw_ports' in the above if.
> > And also suggest to use in other places.
> >
>
> Ack.
>
> >
> > > && nat->logical_port && nat->external_mac) {
> > > /* Distributed NAT rule. */
> > > if (eth_addr_from_string(nat->external_mac, &mac)) {
> > > @@ -2695,9 +2712,9 @@ get_nat_addresses(const struct ovn_port *op,
> size_t *n, bool routable_only)
> > > if (central_ip_address) {
> > > /* Gratuitous ARP for centralized NAT rules on distributed
> gateway
> > > * ports should be restricted to the gateway chassis. */
> > > - if (op->od->l3redirect_port) {
> > > + if (op->od->l3dgw_ports) {
> > > ds_put_format(&c_addresses, " is_chassis_resident(%s)",
> > > - op->od->l3redirect_port->json_key);
> > > + op->od->l3dgw_ports[0]->cr_port->json_key);
> > > }
> > >
> > > addresses[n_nats++] = ds_steal_cstr(&c_addresses);
> > > @@ -3010,7 +3027,7 @@ ovn_port_update_sbrec(struct northd_context *ctx,
> > > /* If the router is for l3 gateway, it resides on a chassis
> > > * and its port type is "l3gateway". */
> > > const char *chassis_name = smap_get(&op->od->nbr->options,
> "chassis");
> > > - if (op->derived) {
> > > + if (op->l3dgw_port) {
> > > sbrec_port_binding_set_type(op->sb, "chassisredirect");
> > > } else if (chassis_name) {
> > > sbrec_port_binding_set_type(op->sb, "l3gateway");
> > > @@ -3020,7 +3037,7 @@ ovn_port_update_sbrec(struct northd_context *ctx,
> > >
> > > struct smap new;
> > > smap_init(&new);
> > > - if (op->derived) {
> > > + if (op->l3dgw_port) {
> > > const char *redirect_type = smap_get(&op->nbrp->options,
> > > "redirect-type");
> > >
> > > @@ -3200,7 +3217,7 @@ ovn_port_update_sbrec(struct northd_context *ctx,
> > > char **nats = NULL;
> > > if (nat_addresses && !strcmp(nat_addresses, "router")) {
> > > if (op->peer && op->peer->od
> > > - && (chassis || op->peer->od->l3redirect_port)) {
> > > + && (chassis || op->peer->od->l3dgw_ports)) {
> > > nats = get_nat_addresses(op->peer, &n_nats, false);
> > > }
> > > /* Only accept manual specification of ethernet address
> > > @@ -3236,12 +3253,26 @@ ovn_port_update_sbrec(struct northd_context
> *ctx,
> > > * sending the GARPs for the router port IPs.
> > > * */
> > > bool add_router_port_garp = false;
> > > - if (op->peer && op->peer->nbrp && op->peer->od->l3dgw_port
> &&
> > > - op->peer->od->l3redirect_port &&
> > > - (smap_get_bool(&op->peer->nbrp->options,
> > > - "reside-on-redirect-chassis", false) ||
> > > - op->peer == op->peer->od->l3dgw_port)) {
> > > - add_router_port_garp = true;
> > > + if (op->peer && op->peer->nbrp &&
> op->peer->od->l3dgw_ports) {
> > > + if (op->peer->cr_port) {
> > > + add_router_port_garp = true;
> > > + } else if (smap_get_bool(&op->peer->nbrp->options,
> > > + "reside-on-redirect-chassis", false)) {
> > > + if (op->peer->od->n_l3dgw_ports == 1) {
> > > + add_router_port_garp = true;
> > > + } else {
> > > + static struct vlog_rate_limit rl =
> > > + VLOG_RATE_LIMIT_INIT(1, 1);
> > > + VLOG_WARN_RL(&rl,
> "\"reside-on-redirect-chassis\" is "
> > > + "set on logical router port %s,
> which "
> > > + "is on logical router %s, which
> has %"
> > > + PRIuSIZE" distributed gateway
> ports. This"
> > > + "option can only be used when
> there is "
> > > + "a single distributed gateway
> port.",
> > > + op->peer->key,
> op->peer->od->nbr->name,
> > > + op->peer->od->n_l3dgw_ports);
> > > + }
> > > + }
> > > } else if (chassis && op->od->n_localnet_ports) {
> > > add_router_port_garp = true;
> > > }
> > > @@ -3256,9 +3287,10 @@ ovn_port_update_sbrec(struct northd_context *ctx,
> > >
> op->peer->lrp_networks.ipv4_addrs[i].addr_s);
> > > }
> > >
> > > - if (op->peer->od->l3redirect_port) {
> > > + if (op->peer->od->l3dgw_ports) {
> > > ds_put_format(&garp_info, "
> is_chassis_resident(%s)",
> > > -
> op->peer->od->l3redirect_port->json_key);
> > > + op->peer->od->l3dgw_ports[0]
> > > + ->cr_port->json_key);
> > > }
> > >
> > > n_nats++;
> > > @@ -3531,7 +3563,17 @@ build_ovn_lr_lbs(struct hmap *datapaths, struct
> hmap *lbs)
> > > if (!od->nbr) {
> > > continue;
> > > }
> > > - if (!smap_get(&od->nbr->options, "chassis") &&
> !od->l3dgw_port) {
> > > + if (!smap_get(&od->nbr->options, "chassis")
> > > + && od->n_l3dgw_ports != 1) {
> > > + if (od->n_l3dgw_ports > 1 && od->nbr->n_load_balancer) {
> > > + static struct vlog_rate_limit rl =
> VLOG_RATE_LIMIT_INIT(1, 1);
> > > + VLOG_WARN_RL(&rl, "Load-balancers are configured on
> logical "
> > > + "router %s, which has %"PRIuSIZE"
> distributed "
> > > + "gateway ports. Load-balancer is not
> supported "
> > > + "yet when there is more than one
> distributed "
> > > + "gateway port on the router.",
> > > + od->nbr->name, od->n_l3dgw_ports);
> > > + }
> > > continue;
> > > }
> > >
> > > @@ -6440,13 +6482,16 @@ build_lrouter_groups__(struct hmap *ports,
> struct ovn_datapath *od)
> > > {
> > > ovs_assert((od && od->nbr && od->lr_group));
> > >
> > > - if (od->l3dgw_port && od->l3redirect_port) {
> > > - /* It's a logical router with gateway port. If it
> > > + if (od->l3dgw_ports) {
> >
> > This "if" check can be dropped as its not necessary.
> >
>
> Ack.
>
> > > + /* It's a logical router with gateway ports. If it
> > > * has HA_Chassis_Group associated to it in SB DB, then store
> the
> > > * ha chassis group name. */
> > > - if (od->l3redirect_port->sb->ha_chassis_group) {
> > > - sset_add(&od->lr_group->ha_chassis_groups,
> > > - od->l3redirect_port->sb->ha_chassis_group->name);
> > > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) {
> > > + struct ovn_port *crp = od->l3dgw_ports[i]->cr_port;
> > > + if (crp->sb->ha_chassis_group) {
> > > + sset_add(&od->lr_group->ha_chassis_groups,
> > > + crp->sb->ha_chassis_group->name);
> > > + }
> > > }
> > > }
> > >
> > > @@ -7833,16 +7878,17 @@ build_lswitch_ip_unicast_lookup(struct ovn_port
> *op,
> > > ds_clear(match);
> > > ds_put_format(match, "eth.dst == "ETH_ADDR_FMT,
> > > ETH_ADDR_ARGS(mac));
> > > - if (op->peer->od->l3dgw_port
> > > - && op->peer->od->l3redirect_port
> > > + if (op->peer->od->l3dgw_ports
> > > && op->od->n_localnet_ports) {
> > > bool add_chassis_resident_check = false;
> > > - if (op->peer == op->peer->od->l3dgw_port) {
> > > + const char *json_key;
> > > + if (op->peer->cr_port) {
> > > /* The peer of this port represents a
> distributed
> > > * gateway port. The destination lookup flow
> for the
> > > * router's distributed gateway port MAC
> address should
> > > * only be programmed on the gateway chassis.
> */
> > > add_chassis_resident_check = true;
> > > + json_key = op->peer->cr_port->json_key;
> > > } else {
> > > /* Check if the option
> 'reside-on-redirect-chassis'
> > > * is set to true on the peer port. If set to
> true
> > > @@ -7853,12 +7899,15 @@ build_lswitch_ip_unicast_lookup(struct ovn_port
> *op,
> > > */
> > > add_chassis_resident_check = smap_get_bool(
> > > &op->peer->nbrp->options,
> > > - "reside-on-redirect-chassis", false);
> > > + "reside-on-redirect-chassis", false) &&
> > > + op->peer->od->n_l3dgw_ports == 1;
> > > + json_key =
> > > +
> op->peer->od->l3dgw_ports[0]->cr_port->json_key;
> > > }
> > >
> > > if (add_chassis_resident_check) {
> > > ds_put_format(match, " &&
> is_chassis_resident(%s)",
> > > -
> op->peer->od->l3redirect_port->json_key);
> > > + json_key);
> > > }
> > > }
> > >
> > > @@ -7871,8 +7920,7 @@ build_lswitch_ip_unicast_lookup(struct ovn_port
> *op,
> > >
> > > /* Add ethernet addresses specified in NAT rules on
> > > * distributed logical routers. */
> > > - if (op->peer->od->l3dgw_port
> > > - && op->peer == op->peer->od->l3dgw_port) {
> > > + if (op->peer->cr_port) {
> > > for (int j = 0; j < op->peer->od->nbr->n_nat; j++)
> {
> > > const struct nbrec_nat *nat
> > > =
> op->peer->od->nbr->nat[j];
> > > @@ -9139,14 +9187,14 @@ build_lrouter_nat_flows_for_lb(struct
> ovn_lb_vip *lb_vip,
> > > &lb->nlb->header_);
> > > }
> > >
> > > - if (od->l3redirect_port &&
> > > + if (od->l3dgw_ports &&
> > > (lb_vip->n_backends || !lb_vip->empty_backend_rej)) {
> > > new_match_p = xasprintf("%s && is_chassis_resident(%s)",
> > > new_match,
> > > - od->l3redirect_port->json_key);
> > > +
> od->l3dgw_ports[0]->cr_port->json_key);
> > > est_match_p = xasprintf("%s && is_chassis_resident(%s)",
> > > est_match,
> > > - od->l3redirect_port->json_key);
> > > +
> od->l3dgw_ports[0]->cr_port->json_key);
> > > }
> > >
> > > if (snat_type == NO_FORCE_SNAT &&
> > > @@ -9191,15 +9239,15 @@ build_lrouter_nat_flows_for_lb(struct
> ovn_lb_vip *lb_vip,
> > > free(est_match_p);
> > > }
> > >
> > > - if (!od->l3dgw_port || !od->l3redirect_port ||
> !lb_vip->n_backends) {
> > > + if (!od->l3dgw_ports || !lb_vip->n_backends) {
> > > goto next;
> > > }
> > >
> > > - char *undnat_match_p = xasprintf("%s) && outport == %s && "
> > > - "is_chassis_resident(%s)",
> > > - ds_cstr(&undnat_match),
> > > - od->l3dgw_port->json_key,
> > > -
> od->l3redirect_port->json_key);
> > > + char *undnat_match_p = xasprintf(
> > > + "%s) && outport == %s && is_chassis_resident(%s)",
> > > + ds_cstr(&undnat_match),
> > > + od->l3dgw_ports[0]->json_key,
> > > + od->l3dgw_ports[0]->cr_port->json_key);
> > > if (snat_type == SKIP_SNAT) {
> > > ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT,
> 120,
> > > undnat_match_p,
> skip_snat_est_action,
> > > @@ -9695,9 +9743,9 @@ build_lrouter_port_nat_arp_nd_flow(struct
> ovn_port *op,
> > > * upstream MAC learning points to the gateway chassis.
> > > * Also need to avoid generation of multiple ARP responses
> > > * from different chassis. */
> > > - if (op->od->l3redirect_port) {
> > > + if (op->od->l3dgw_ports) {
> > > ds_put_format(&match, "is_chassis_resident(%s)",
> > > - op->od->l3redirect_port->json_key);
> > > + op->od->l3dgw_ports[0]->cr_port->json_key);
> > > }
> > > }
> > >
> > > @@ -9968,7 +10016,7 @@ build_adm_ctrl_flows_for_lrouter_port(
> > > return;
> > > }
> > >
> > > - if (op->derived) {
> > > + if (op->l3dgw_port) {
> > > /* No ingress packets should be received on a
> chassisredirect
> > > * port. */
> > > return;
> > > @@ -9991,12 +10039,11 @@ build_adm_ctrl_flows_for_lrouter_port(
> > > ds_clear(match);
> > > ds_put_format(match, "eth.dst == %s && inport == %s",
> > > op->lrp_networks.ea_s, op->json_key);
> > > - if (op->od->l3dgw_port && op == op->od->l3dgw_port
> > > - && op->od->l3redirect_port) {
> > > + if (op->cr_port) {
> > > /* Traffic with eth.dst = l3dgw_port->lrp_networks.ea_s
> > > * should only be received on the gateway chassis. */
> > > ds_put_format(match, " && is_chassis_resident(%s)",
> > > - op->od->l3redirect_port->json_key);
> > > + op->cr_port->json_key);
> > > }
> > > ovn_lflow_add_with_hint(lflows, op->od, S_ROUTER_IN_ADMISSION,
> 50,
> > > ds_cstr(match), ds_cstr(actions),
> > > @@ -10135,10 +10182,9 @@ build_neigh_learning_flows_for_lrouter_port(
> > > op->lrp_networks.ipv4_addrs[i].network_s,
> > > op->lrp_networks.ipv4_addrs[i].plen,
> > > op->lrp_networks.ipv4_addrs[i].addr_s);
> > > - if (op->od->l3dgw_port && op == op->od->l3dgw_port
> > > - && op->od->l3redirect_port) {
> > > + if (op->cr_port) {
> > > ds_put_format(match, " && is_chassis_resident(%s)",
> > > - op->od->l3redirect_port->json_key);
> > > + op->cr_port->json_key);
> > > }
> > > const char *actions_s = REGBIT_LOOKUP_NEIGHBOR_RESULT
> > > " = lookup_arp(inport, arp.spa,
> arp.sha); "
> > > @@ -10155,10 +10201,9 @@ build_neigh_learning_flows_for_lrouter_port(
> > > op->json_key,
> > > op->lrp_networks.ipv4_addrs[i].network_s,
> > > op->lrp_networks.ipv4_addrs[i].plen);
> > > - if (op->od->l3dgw_port && op == op->od->l3dgw_port
> > > - && op->od->l3redirect_port) {
> > > + if (op->cr_port) {
> > > ds_put_format(match, " && is_chassis_resident(%s)",
> > > - op->od->l3redirect_port->json_key);
> > > + op->cr_port->json_key);
> > > }
> > > ds_clear(actions);
> > > ds_put_format(actions, REGBIT_LOOKUP_NEIGHBOR_RESULT
> > > @@ -10643,7 +10688,7 @@ build_arp_resolve_flows_for_lrouter_port(
> > > }
> > > }
> > >
> > > - if (!op->derived && op->od->l3redirect_port) {
> > > + if (op->cr_port) {
> > > const char *redirect_type = smap_get(&op->nbrp->options,
> > > "redirect-type");
> > > if (redirect_type && !strcasecmp(redirect_type,
> "bridged")) {
> > > @@ -10656,7 +10701,7 @@ build_arp_resolve_flows_for_lrouter_port(
> > > ds_clear(match);
> > > ds_put_format(match, "outport == %s && "
> > > "!is_chassis_resident(%s)", op->json_key,
> > > - op->od->l3redirect_port->json_key);
> > > + op->cr_port->json_key);
> > > ds_clear(actions);
> > > ds_put_format(actions, "eth.dst = %s; next;",
> > > op->lrp_networks.ea_s);
> > > @@ -10904,8 +10949,8 @@ build_arp_resolve_flows_for_lrouter_port(
> > > &op->nbsp->header_);
> > > }
> > >
> > > - if (smap_get(&peer->od->nbr->options, "chassis") ||
> > > - (peer->od->l3dgw_port && peer ==
> peer->od->l3dgw_port)) {
> > > + if (smap_get(&peer->od->nbr->options, "chassis")
> > > + || peer->cr_port) {
> > > routable_addresses_to_lflows(lflows, router_port, peer,
> > > match, actions);
> > > }
> > > @@ -10934,110 +10979,111 @@ build_check_pkt_len_flows_for_lrouter(
> > > struct ds *match, struct ds *actions,
> > > struct shash *meter_groups)
> > > {
> > > - if (od->nbr) {
> > > -
> > > - /* Packets are allowed by default. */
> > > - ovn_lflow_add(lflows, od, S_ROUTER_IN_CHK_PKT_LEN, 0, "1",
> > > - "next;");
> > > - ovn_lflow_add(lflows, od, S_ROUTER_IN_LARGER_PKTS, 0, "1",
> > > - "next;");
> > > + if (!od->nbr) {
> > > + return;
> > > + }
> > >
> > > - if (od->l3dgw_port && od->l3redirect_port) {
> > > - int gw_mtu = 0;
> > > - if (od->l3dgw_port->nbrp) {
> > > - gw_mtu = smap_get_int(&od->l3dgw_port->nbrp->options,
> > > - "gateway_mtu", 0);
> > > - }
> > > - /* Add the flows only if gateway_mtu is configured. */
> > > - if (gw_mtu <= 0) {
> > > - return;
> > > - }
> > > + /* Packets are allowed by default. */
> > > + ovn_lflow_add(lflows, od, S_ROUTER_IN_CHK_PKT_LEN, 0, "1",
> > > + "next;");
> > > + ovn_lflow_add(lflows, od, S_ROUTER_IN_LARGER_PKTS, 0, "1",
> > > + "next;");
> > > + for (size_t dgp = 0; dgp < od->n_l3dgw_ports; dgp++) {
> > > + int gw_mtu = 0;
> > > + if (od->l3dgw_ports[dgp]->nbrp) {
> > > + gw_mtu =
> smap_get_int(&od->l3dgw_ports[dgp]->nbrp->options,
> > > + "gateway_mtu", 0);
> > > + }
> > > + /* Add the flows only if gateway_mtu is configured. */
> > > + if (gw_mtu <= 0) {
> > > + continue;
> > > + }
> > >
> > > - ds_clear(match);
> > > - ds_put_format(match, "outport == %s",
> od->l3dgw_port->json_key);
> > > + ds_clear(match);
> > > + ds_put_format(match, "outport == %s",
> > > + od->l3dgw_ports[dgp]->json_key);
> > >
> > > - ds_clear(actions);
> > > - ds_put_format(actions,
> > > - REGBIT_PKT_LARGER" = check_pkt_larger(%d);"
> > > - " next;", gw_mtu + VLAN_ETH_HEADER_LEN);
> > > - ovn_lflow_add_with_hint(lflows, od,
> S_ROUTER_IN_CHK_PKT_LEN, 50,
> > > - ds_cstr(match), ds_cstr(actions),
> > > - &od->l3dgw_port->nbrp->header_);
> > > + ds_clear(actions);
> > > + ds_put_format(actions,
> > > + REGBIT_PKT_LARGER" = check_pkt_larger(%d);"
> > > + " next;", gw_mtu + VLAN_ETH_HEADER_LEN);
> > > + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_CHK_PKT_LEN,
> 50,
> > > + ds_cstr(match), ds_cstr(actions),
> > > + &od->l3dgw_ports[dgp]->nbrp->header_);
> > >
> > > - for (size_t i = 0; i < od->nbr->n_ports; i++) {
> > > - struct ovn_port *rp = ovn_port_find(ports,
> > > -
> od->nbr->ports[i]->name);
> > > - if (!rp || rp == od->l3dgw_port) {
> > > - continue;
> > > - }
> > > + for (size_t i = 0; i < od->nbr->n_ports; i++) {
> > > + struct ovn_port *rp = ovn_port_find(ports,
> > > +
> od->nbr->ports[i]->name);
> > > + if (!rp || rp->cr_port) {
> > > + continue;
> > > + }
> > >
> > > - if (rp->lrp_networks.ipv4_addrs) {
> > > - ds_clear(match);
> > > - ds_put_format(match, "inport == %s && outport ==
> %s"
> > > - " && ip4 && "REGBIT_PKT_LARGER,
> > > - rp->json_key,
> od->l3dgw_port->json_key);
> > > + if (rp->lrp_networks.ipv4_addrs) {
> > > + ds_clear(match);
> > > + ds_put_format(match, "inport == %s && outport == %s"
> > > + " && ip4 && "REGBIT_PKT_LARGER,
> > > + rp->json_key,
> od->l3dgw_ports[dgp]->json_key);
> > >
> > > - ds_clear(actions);
> > > - /* Set icmp4.frag_mtu to gw_mtu */
> > > - ds_put_format(actions,
> > > - "icmp4_error {"
> > > - REGBIT_EGRESS_LOOPBACK" = 1; "
> > > - "eth.dst = %s; "
> > > - "ip4.dst = ip4.src; "
> > > - "ip4.src = %s; "
> > > - "ip.ttl = 255; "
> > > - "icmp4.type = 3; /* Destination Unreachable.
> */ "
> > > - "icmp4.code = 4; /* Frag Needed and DF was
> Set. */ "
> > > - "icmp4.frag_mtu = %d; "
> > > - "next(pipeline=ingress, table=%d); };",
> > > - rp->lrp_networks.ea_s,
> > > - rp->lrp_networks.ipv4_addrs[0].addr_s,
> > > - gw_mtu,
> > > - ovn_stage_get_table(S_ROUTER_IN_ADMISSION));
> > > - ovn_lflow_add_with_hint__(lflows, od,
> > > - S_ROUTER_IN_LARGER_PKTS,
> 50,
> > > - ds_cstr(match),
> ds_cstr(actions),
> > > - NULL,
> > > - copp_meter_get(
> > > - COPP_ICMP4_ERR,
> > > - rp->od->nbr->copp,
> > > - meter_groups),
> > > - &rp->nbrp->header_);
> > > - }
> > > + ds_clear(actions);
> > > + /* Set icmp4.frag_mtu to gw_mtu */
> > > + ds_put_format(actions,
> > > + "icmp4_error {"
> > > + REGBIT_EGRESS_LOOPBACK" = 1; "
> > > + "eth.dst = %s; "
> > > + "ip4.dst = ip4.src; "
> > > + "ip4.src = %s; "
> > > + "ip.ttl = 255; "
> > > + "icmp4.type = 3; /* Destination Unreachable. */ "
> > > + "icmp4.code = 4; /* Frag Needed and DF was Set. */
> "
> > > + "icmp4.frag_mtu = %d; "
> > > + "next(pipeline=ingress, table=%d); };",
> > > + rp->lrp_networks.ea_s,
> > > + rp->lrp_networks.ipv4_addrs[0].addr_s,
> > > + gw_mtu,
> > > + ovn_stage_get_table(S_ROUTER_IN_ADMISSION));
> > > + ovn_lflow_add_with_hint__(lflows, od,
> > > + S_ROUTER_IN_LARGER_PKTS, 50,
> > > + ds_cstr(match),
> ds_cstr(actions),
> > > + NULL,
> > > + copp_meter_get(
> > > + COPP_ICMP4_ERR,
> > > + rp->od->nbr->copp,
> > > + meter_groups),
> > > + &rp->nbrp->header_);
> > > + }
> > >
> > > - if (rp->lrp_networks.ipv6_addrs) {
> > > - ds_clear(match);
> > > - ds_put_format(match, "inport == %s && outport ==
> %s"
> > > - " && ip6 && "REGBIT_PKT_LARGER,
> > > - rp->json_key,
> od->l3dgw_port->json_key);
> > > + if (rp->lrp_networks.ipv6_addrs) {
> > > + ds_clear(match);
> > > + ds_put_format(match, "inport == %s && outport == %s"
> > > + " && ip6 && "REGBIT_PKT_LARGER,
> > > + rp->json_key,
> od->l3dgw_ports[dgp]->json_key);
> > >
> > > - ds_clear(actions);
> > > - /* Set icmp6.frag_mtu to gw_mtu */
> > > - ds_put_format(actions,
> > > - "icmp6_error {"
> > > - REGBIT_EGRESS_LOOPBACK" = 1; "
> > > - "eth.dst = %s; "
> > > - "ip6.dst = ip6.src; "
> > > - "ip6.src = %s; "
> > > - "ip.ttl = 255; "
> > > - "icmp6.type = 2; /* Packet Too Big. */ "
> > > - "icmp6.code = 0; "
> > > - "icmp6.frag_mtu = %d; "
> > > - "next(pipeline=ingress, table=%d); };",
> > > - rp->lrp_networks.ea_s,
> > > - rp->lrp_networks.ipv6_addrs[0].addr_s,
> > > - gw_mtu,
> > > - ovn_stage_get_table(S_ROUTER_IN_ADMISSION));
> > > - ovn_lflow_add_with_hint__(lflows, od,
> > > - S_ROUTER_IN_LARGER_PKTS,
> 50,
> > > - ds_cstr(match),
> ds_cstr(actions),
> > > - NULL,
> > > - copp_meter_get(
> > > - COPP_ICMP6_ERR,
> > > - rp->od->nbr->copp,
> > > - meter_groups),
> > > - &rp->nbrp->header_);
> > > - }
> > > + ds_clear(actions);
> > > + /* Set icmp6.frag_mtu to gw_mtu */
> > > + ds_put_format(actions,
> > > + "icmp6_error {"
> > > + REGBIT_EGRESS_LOOPBACK" = 1; "
> > > + "eth.dst = %s; "
> > > + "ip6.dst = ip6.src; "
> > > + "ip6.src = %s; "
> > > + "ip.ttl = 255; "
> > > + "icmp6.type = 2; /* Packet Too Big. */ "
> > > + "icmp6.code = 0; "
> > > + "icmp6.frag_mtu = %d; "
> > > + "next(pipeline=ingress, table=%d); };",
> > > + rp->lrp_networks.ea_s,
> > > + rp->lrp_networks.ipv6_addrs[0].addr_s,
> > > + gw_mtu,
> > > + ovn_stage_get_table(S_ROUTER_IN_ADMISSION));
> > > + ovn_lflow_add_with_hint__(lflows, od,
> > > + S_ROUTER_IN_LARGER_PKTS, 50,
> > > + ds_cstr(match),
> ds_cstr(actions),
> > > + NULL,
> > > + copp_meter_get(
> > > + COPP_ICMP6_ERR,
> > > + rp->od->nbr->copp,
> > > + meter_groups),
> > > + &rp->nbrp->header_);
> > > }
> > > }
> > > }
> > > @@ -11055,32 +11101,32 @@ build_gateway_redirect_flows_for_lrouter(
> > > struct ovn_datapath *od, struct hmap *lflows,
> > > struct ds *match, struct ds *actions)
> > > {
> > > - if (od->nbr) {
> > > - if (od->l3dgw_port && od->l3redirect_port) {
> > > - const struct ovsdb_idl_row *stage_hint = NULL;
> > > -
> > > - if (od->l3dgw_port->nbrp) {
> > > - stage_hint = &od->l3dgw_port->nbrp->header_;
> > > - }
> > > + if (!od->nbr) {
> > > + return;
> > > + }
> > > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) {
> > > + const struct ovsdb_idl_row *stage_hint = NULL;
> > >
> > > - /* For traffic with outport == l3dgw_port, if the
> > > - * packet did not match any higher priority redirect
> > > - * rule, then the traffic is redirected to the central
> > > - * instance of the l3dgw_port. */
> > > - ds_clear(match);
> > > - ds_put_format(match, "outport == %s",
> > > - od->l3dgw_port->json_key);
> > > - ds_clear(actions);
> > > - ds_put_format(actions, "outport = %s; next;",
> > > - od->l3redirect_port->json_key);
> > > - ovn_lflow_add_with_hint(lflows, od,
> S_ROUTER_IN_GW_REDIRECT, 50,
> > > - ds_cstr(match), ds_cstr(actions),
> > > - stage_hint);
> > > + if (od->l3dgw_ports[i]->nbrp) {
> > > + stage_hint = &od->l3dgw_ports[i]->nbrp->header_;
> > > }
> > >
> > > - /* Packets are allowed by default. */
> > > - ovn_lflow_add(lflows, od, S_ROUTER_IN_GW_REDIRECT, 0, "1",
> "next;");
> > > + /* For traffic with outport == l3dgw_port, if the
> > > + * packet did not match any higher priority redirect
> > > + * rule, then the traffic is redirected to the central
> > > + * instance of the l3dgw_port. */
> > > + ds_clear(match);
> > > + ds_put_format(match, "outport == %s",
> > > + od->l3dgw_ports[i]->json_key);
> > > + ds_clear(actions);
> > > + ds_put_format(actions, "outport = %s; next;",
> > > + od->l3dgw_ports[i]->cr_port->json_key);
> > > + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_GW_REDIRECT,
> 50,
> > > + ds_cstr(match), ds_cstr(actions),
> > > + stage_hint);
> > > }
> > > + /* Packets are allowed by default. */
> > > + ovn_lflow_add(lflows, od, S_ROUTER_IN_GW_REDIRECT, 0, "1",
> "next;");
> > > }
> > >
> > > /* Local router ingress table ARP_REQUEST: ARP request.
> > > @@ -11179,7 +11225,7 @@ build_egress_delivery_flows_for_lrouter_port(
> > > return;
> > > }
> > >
> > > - if (op->derived) {
> > > + if (op->l3dgw_port) {
> > > /* No egress packets should be processed in the context of
> > > * a chassisredirect port. The chassisredirect port should
> > > * be replaced by the l3dgw port in the local output
> > > @@ -11269,7 +11315,7 @@ build_dhcpv6_reply_flows_for_lrouter_port(
> > > struct ovn_port *op, struct hmap *lflows,
> > > struct ds *match)
> > > {
> > > - if (op->nbrp && (!op->derived)) {
> > > + if (op->nbrp && (!op->l3dgw_port)) {
> > > for (size_t i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) {
> > > ds_clear(match);
> > > ds_put_format(match, "ip6.dst == %s && udp.src == 547 &&"
> > > @@ -11289,7 +11335,7 @@ build_ipv6_input_flows_for_lrouter_port(
> > > struct ds *match, struct ds *actions,
> > > struct shash *meter_groups)
> > > {
> > > - if (op->nbrp && (!op->derived)) {
> > > + if (op->nbrp && (!op->l3dgw_port)) {
> > > /* No ingress packets are accepted on a chassisredirect
> > > * port, so no need to program flows for that port. */
> > > if (op->lrp_networks.n_ipv6_addrs) {
> > > @@ -11315,15 +11361,14 @@ build_ipv6_input_flows_for_lrouter_port(
> > > * router's own IP address. */
> > > for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) {
> > > ds_clear(match);
> > > - if (op->od->l3dgw_port && op == op->od->l3dgw_port
> > > - && op->od->l3redirect_port) {
> > > + if (op->cr_port) {
> > > /* Traffic with eth.src = l3dgw_port->lrp_networks.ea_s
> > > * should only be sent from the gateway chassi, so that
> > > * upstream MAC learning points to the gateway chassis.
> > > * Also need to avoid generation of multiple ND replies
> > > * from different chassis. */
> > > ds_put_format(match, "is_chassis_resident(%s)",
> > > - op->od->l3redirect_port->json_key);
> > > + op->cr_port->json_key);
> > > }
> > >
> > > build_lrouter_nd_flow(op->od, op, "nd_na_router",
> > > @@ -11334,7 +11379,7 @@ build_ipv6_input_flows_for_lrouter_port(
> > > }
> > >
> > > /* UDP/TCP/SCTP port unreachable */
> > > - if (!op->od->is_gw_router && !op->od->l3dgw_port) {
> > > + if (!op->od->is_gw_router && !op->od->l3dgw_ports) {
> > > for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) {
> > > ds_clear(match);
> > > ds_put_format(match,
> > > @@ -11504,7 +11549,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> > > {
> > > /* No ingress packets are accepted on a chassisredirect
> > > * port, so no need to program flows for that port. */
> > > - if (op->nbrp && (!op->derived)) {
> > > + if (op->nbrp && (!op->l3dgw_port)) {
> > > if (op->lrp_networks.n_ipv4_addrs) {
> > > /* L3 admission control: drop packets that originate from
> an
> > > * IPv4 address owned by the router or a broadcast address
> > > @@ -11574,16 +11619,18 @@ build_lrouter_ipv4_ip_input(struct ovn_port
> *op,
> > > op->lrp_networks.ipv4_addrs[i].network_s,
> > > op->lrp_networks.ipv4_addrs[i].plen);
> > >
> > > - if (op->od->l3dgw_port && op->od->l3redirect_port &&
> op->peer
> > > + if (op->od->l3dgw_ports && op->peer
> > > && op->peer->od->n_localnet_ports) {
> > > bool add_chassis_resident_check = false;
> > > - if (op == op->od->l3dgw_port) {
> > > + const char *json_key;
> > > + if (op->cr_port) {
> > > /* Traffic with eth.src =
> l3dgw_port->lrp_networks.ea_s
> > > * should only be sent from the gateway chassis,
> so that
> > > * upstream MAC learning points to the gateway
> chassis.
> > > * Also need to avoid generation of multiple ARP
> responses
> > > * from different chassis. */
> > > add_chassis_resident_check = true;
> > > + json_key = op->cr_port->json_key;
> > > } else {
> > > /* Check if the option 'reside-on-redirect-chassis'
> > > * is set to true on the router port. If set to
> true
> > > @@ -11595,12 +11642,14 @@ build_lrouter_ipv4_ip_input(struct ovn_port
> *op,
> > > */
> > > add_chassis_resident_check = smap_get_bool(
> > > &op->nbrp->options,
> > > - "reside-on-redirect-chassis", false);
> > > + "reside-on-redirect-chassis", false) &&
> > > + op->od->n_l3dgw_ports == 1;
> > > + json_key =
> op->od->l3dgw_ports[0]->cr_port->json_key;
> > > }
> > >
> > > if (add_chassis_resident_check) {
> > > ds_put_format(match, " && is_chassis_resident(%s)",
> > > - op->od->l3redirect_port->json_key);
> > > + json_key);
> > > }
> > > }
> > >
> > > @@ -11613,9 +11662,9 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> > > const char *ip_address;
> > > if (sset_count(&op->od->lb_ips_v4)) {
> > > ds_clear(match);
> > > - if (op == op->od->l3dgw_port) {
> > > + if (op->cr_port) {
> > > ds_put_format(match, "is_chassis_resident(%s)",
> > > - op->od->l3redirect_port->json_key);
> > > + op->cr_port->json_key);
> > > }
> > >
> > > struct ds load_balancer_ips_v4 = DS_EMPTY_INITIALIZER;
> > > @@ -11633,9 +11682,9 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> > >
> > > SSET_FOR_EACH (ip_address, &op->od->lb_ips_v6) {
> > > ds_clear(match);
> > > - if (op == op->od->l3dgw_port) {
> > > + if (op->cr_port) {
> > > ds_put_format(match, "is_chassis_resident(%s)",
> > > - op->od->l3redirect_port->json_key);
> > > + op->cr_port->json_key);
> > > }
> > >
> > > build_lrouter_nd_flow(op->od, op, "nd_na",
> > > @@ -11644,7 +11693,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> > > lflows, meter_groups);
> > > }
> > >
> > > - if (!op->od->is_gw_router && !op->od->l3dgw_port) {
> > > + if (!op->od->is_gw_router && !op->od->l3dgw_ports) {
> > > /* UDP/TCP/SCTP port unreachable. */
> > > for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) {
> > > ds_clear(match);
> > > @@ -11741,7 +11790,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> > > * exception is on the l3dgw_port where we might need to use a
> > > * different ETH address.
> > > */
> > > - if (op != op->od->l3dgw_port) {
> > > + if (!op->cr_port) {
> > > return;
> > > }
> > >
> > > @@ -11823,12 +11872,12 @@ build_lrouter_in_unsnat_flow(struct hmap
> *lflows, struct ovn_datapath *od,
> > > ds_clear(actions);
> > > ds_put_format(match, "ip && ip%s.dst == %s && inport == %s",
> > > is_v6 ? "6" : "4", nat->external_ip,
> > > - od->l3dgw_port->json_key);
> > > - if (!distributed && od->l3redirect_port) {
> > > + od->l3dgw_ports[0]->json_key);
> > > + if (!distributed && od->l3dgw_ports) {
> > > /* Flows for NAT rules that are centralized are only
> > > * programmed on the gateway chassis. */
> > > ds_put_format(match, " && is_chassis_resident(%s)",
> > > - od->l3redirect_port->json_key);
> > > + od->l3dgw_ports[0]->cr_port->json_key);
> > > }
> > >
> > > if (!strcmp(nat->type, "dnat_and_snat") && stateless) {
> > > @@ -11900,12 +11949,12 @@ build_lrouter_in_dnat_flow(struct hmap
> *lflows, struct ovn_datapath *od,
> > > ds_clear(match);
> > > ds_put_format(match, "ip && ip%s.dst == %s && inport ==
> %s",
> > > is_v6 ? "6" : "4", nat->external_ip,
> > > - od->l3dgw_port->json_key);
> > > - if (!distributed && od->l3redirect_port) {
> > > + od->l3dgw_ports[0]->json_key);
> > > + if (!distributed && od->l3dgw_ports) {
> > > /* Flows for NAT rules that are centralized are only
> > > * programmed on the gateway chassis. */
> > > ds_put_format(match, " && is_chassis_resident(%s)",
> > > - od->l3redirect_port->json_key);
> > > + od->l3dgw_ports[0]->cr_port->json_key);
> > > }
> > > ds_clear(actions);
> > > if (nat->allowed_ext_ips || nat->exempted_ext_ips) {
> > > @@ -11944,7 +11993,7 @@ build_lrouter_out_undnat_flow(struct hmap
> *lflows, struct ovn_datapath *od,
> > > *
> > > * Note that this only applies for NAT on a distributed router.
> > > */
> > > - if (!od->l3dgw_port ||
> > > + if (!od->l3dgw_ports ||
> > > (strcmp(nat->type, "dnat") && strcmp(nat->type,
> "dnat_and_snat"))) {
> > > return;
> > > }
> > > @@ -11952,12 +12001,12 @@ build_lrouter_out_undnat_flow(struct hmap
> *lflows, struct ovn_datapath *od,
> > > ds_clear(match);
> > > ds_put_format(match, "ip && ip%s.src == %s && outport == %s",
> > > is_v6 ? "6" : "4", nat->logical_ip,
> > > - od->l3dgw_port->json_key);
> > > - if (!distributed && od->l3redirect_port) {
> > > + od->l3dgw_ports[0]->json_key);
> > > + if (!distributed && od->l3dgw_ports) {
> > > /* Flows for NAT rules that are centralized are only
> > > * programmed on the gateway chassis. */
> > > ds_put_format(match, " && is_chassis_resident(%s)",
> > > - od->l3redirect_port->json_key);
> > > + od->l3dgw_ports[0]->cr_port->json_key);
> > > }
> > > ds_clear(actions);
> > > if (distributed) {
> > > @@ -12030,13 +12079,13 @@ build_lrouter_out_snat_flow(struct hmap
> *lflows, struct ovn_datapath *od,
> > > ds_clear(match);
> > > ds_put_format(match, "ip && ip%s.src == %s && outport == %s",
> > > is_v6 ? "6" : "4", nat->logical_ip,
> > > - od->l3dgw_port->json_key);
> > > - if (!distributed && od->l3redirect_port) {
> > > + od->l3dgw_ports[0]->json_key);
> > > + if (!distributed && od->l3dgw_ports) {
> > > /* Flows for NAT rules that are centralized are only
> > > * programmed on the gateway chassis. */
> > > priority += 128;
> > > ds_put_format(match, " && is_chassis_resident(%s)",
> > > - od->l3redirect_port->json_key);
> > > + od->l3dgw_ports[0]->cr_port->json_key);
> > > }
> > > ds_clear(actions);
> > >
> > > @@ -12077,11 +12126,11 @@ build_lrouter_ingress_flow(struct hmap
> *lflows, struct ovn_datapath *od,
> > > struct ds *actions, struct eth_addr mac,
> > > bool distributed, bool is_v6)
> > > {
> > > - if (od->l3dgw_port && !strcmp(nat->type, "snat")) {
> > > + if (od->l3dgw_ports && !strcmp(nat->type, "snat")) {
> > > ds_clear(match);
> > > ds_put_format(
> > > match, "inport == %s && %s == %s",
> > > - od->l3dgw_port->json_key,
> > > + od->l3dgw_ports[0]->json_key,
> > > is_v6 ? "ip6.src" : "ip4.src", nat->external_ip);
> > > ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_INPUT,
> > > 120, ds_cstr(match), "next;",
> > > @@ -12099,14 +12148,14 @@ build_lrouter_ingress_flow(struct hmap
> *lflows, struct ovn_datapath *od,
> > > */
> > > ds_clear(actions);
> > > ds_put_format(actions, REG_INPORT_ETH_ADDR " = %s; next;",
> > > - od->l3dgw_port->lrp_networks.ea_s);
> > > + od->l3dgw_ports[0]->lrp_networks.ea_s);
> > >
> > > ds_clear(match);
> > > ds_put_format(match,
> > > "eth.dst == "ETH_ADDR_FMT" && inport == %s"
> > > " && is_chassis_resident(\"%s\")",
> > > ETH_ADDR_ARGS(mac),
> > > - od->l3dgw_port->json_key,
> > > + od->l3dgw_ports[0]->json_key,
> > > nat->logical_port);
> > > ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_ADMISSION, 50,
> > > ds_cstr(match), ds_cstr(actions),
> > > @@ -12188,7 +12237,7 @@ lrouter_check_nat_entry(struct ovn_datapath
> *od, const struct nbrec_nat *nat,
> > > /* For distributed router NAT, determine whether this NAT rule
> > > * satisfies the conditions for distributed NAT processing. */
> > > *distributed = false;
> > > - if (od->l3dgw_port && !strcmp(nat->type, "dnat_and_snat") &&
> > > + if (od->l3dgw_ports && !strcmp(nat->type, "dnat_and_snat") &&
> > > nat->logical_port && nat->external_mac) {
> > > if (eth_addr_from_string(nat->external_mac, mac)) {
> > > *distributed = true;
> > > @@ -12233,7 +12282,7 @@ build_lrouter_nat_defrag_and_lb(struct
> ovn_datapath *od, struct hmap *lflows,
> > > * not committed, it would produce ongoing datapath flows with the
> ct.new
> > > * flag set. Some NICs are unable to offload these flows.
> > > */
> > > - if ((od->is_gw_router || od->l3dgw_port) &&
> > > + if ((od->is_gw_router || od->l3dgw_ports) &&
> > > (od->nbr->n_nat || od->nbr->n_load_balancer)) {
> > > ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 50,
> > > "ip", "flags.loopback = 1; ct_dnat;");
> > > @@ -12249,7 +12298,7 @@ build_lrouter_nat_defrag_and_lb(struct
> ovn_datapath *od, struct hmap *lflows,
> > > /* NAT rules are only valid on Gateway routers and routers with
> > > * l3dgw_port (router has a port with gateway chassis
> > > * specified). */
> > > - if (!od->is_gw_router && !od->l3dgw_port) {
> > > + if (!od->is_gw_router && !od->l3dgw_ports) {
> > > return;
> > > }
> > >
> > > @@ -12290,14 +12339,14 @@ build_lrouter_nat_defrag_and_lb(struct
> ovn_datapath *od, struct hmap *lflows,
> > > ds_clear(match);
> > > ds_put_format(
> > > match, "outport == %s && %s == %s",
> > > - od->l3dgw_port->json_key,
> > > + od->l3dgw_ports[0]->json_key,
> > > is_v6 ? REG_NEXT_HOP_IPV6 : REG_NEXT_HOP_IPV4,
> > > nat->external_ip);
> > > ds_clear(actions);
> > > ds_put_format(
> > > actions, "eth.dst = %s; next;",
> > > distributed ? nat->external_mac :
> > > - od->l3dgw_port->lrp_networks.ea_s);
> > > + od->l3dgw_ports[0]->lrp_networks.ea_s);
> > > ovn_lflow_add_with_hint(lflows, od,
> > > S_ROUTER_IN_ARP_RESOLVE,
> > > 100, ds_cstr(match),
> > > @@ -12333,7 +12382,7 @@ build_lrouter_nat_defrag_and_lb(struct
> ovn_datapath *od, struct hmap *lflows,
> > > ds_put_format(match,
> > > "ip%s.src == %s && outport == %s",
> > > is_v6 ? "6" : "4", nat->logical_ip,
> > > - od->l3dgw_port->json_key);
> > > + od->l3dgw_ports[0]->json_key);
> > > /* Add a rule to drop traffic from a distributed NAT if
> > > * the virtual port has not claimed yet becaused otherwise
> > > * the traffic will be centralized misconfiguring the TOR
> switch.
> > > @@ -12360,16 +12409,16 @@ build_lrouter_nat_defrag_and_lb(struct
> ovn_datapath *od, struct hmap *lflows,
> > > * gateway port have ip.dst matching a NAT external IP, then
> > > * loop a clone of the packet back to the beginning of the
> > > * ingress pipeline with inport = outport. */
> > > - if (od->l3dgw_port) {
> > > + if (od->l3dgw_ports) {
> > > /* Distributed router. */
> > > ds_clear(match);
> > > ds_put_format(match, "ip%s.dst == %s && outport == %s",
> > > is_v6 ? "6" : "4",
> > > nat->external_ip,
> > > - od->l3dgw_port->json_key);
> > > + od->l3dgw_ports[0]->json_key);
> > > if (!distributed) {
> > > ds_put_format(match, " && is_chassis_resident(%s)",
> > > - od->l3redirect_port->json_key);
> > > + od->l3dgw_ports[0]->cr_port->json_key);
> > > } else {
> > > ds_put_format(match, " && is_chassis_resident(\"%s\")",
> > > nat->logical_port);
> > > diff --git a/northd/ovn_northd.dl b/northd/ovn_northd.dl
> > > index 7bfaae992..c6ebecb81 100644
> > > --- a/northd/ovn_northd.dl
> > > +++ b/northd/ovn_northd.dl
> > > @@ -183,7 +183,7 @@ OutProxy_Port_Binding(._uuid =
> lsp._uuid,
> > > },
> > > Some{var router_port} = lsp.options.get("router-port"),
> > > var opt_chassis = peer.and_then(|p|
> p.router.options.get("chassis")),
> > > - var l3dgw_port = peer.and_then(|p| p.router.l3dgw_port),
> > > + var l3dgw_port = peer.and_then(|p| p.router.l3dgw_ports.nth(0)),
> > > (var __type, var options) = {
> > > var options = ["peer" -> router_port];
> > > match (opt_chassis) {
> > > @@ -239,7 +239,7 @@ OutProxy_Port_Binding(._uuid =
> lsp._uuid,
> > > Some{rport} -> match (
> > >
> (rport.lrp.options.get_bool_def("reside-on-redirect-chassis", false)
> > > and l3dgw_port.is_some()) or
> > > - Some{rport.lrp} == l3dgw_port or
> > > + rport.is_redirect or
> > > (rport.router.options.contains_key("chassis") and
> > > not sw.localnet_ports.is_empty())) {
> > > false -> set_empty(),
> > > @@ -333,7 +333,7 @@ function get_router_load_balancer_ips(router:
> Intern<Router>,
> > > function get_nat_addresses(rport: Intern<RouterPort>, routable_only:
> bool): Set<string> =
> > > {
> > > var addresses = set_empty();
> > > - var has_redirect = rport.router.l3dgw_port.is_some();
> > > + var has_redirect = not rport.router.l3dgw_ports.is_empty();
> >
> > I didn't understand this. Why is the new code using "is_empty()"
> > where as the deleted code has "is_some()" ?
>
> The new code l3dgw_ports is a set, so using the form "not xxx.is_empty()",
> meaning the set is NOT empty, which is equivalent to the old code
> xxx.is_some() meaning xxx exists.
Oops. My bad. I totally missed the "not".
Thanks
Numan
>
> >
> >
> > > match (eth_addr_from_string(rport.lrp.mac)) {
> > > None -> addresses,
> > > Some{mac} -> {
> > > @@ -400,7 +400,10 @@ function get_nat_addresses(rport:
> Intern<RouterPort>, routable_only: bool): Set<
> > > /* Gratuitous ARP for centralized NAT rules on
> distributed gateway
> > > * ports should be restricted to the gateway chassis.
> */
> > > if (has_redirect) {
> > > - c_addresses = c_addresses ++ "
> is_chassis_resident(${rport.router.redirect_port_name})"
> > > + c_addresses = c_addresses ++ match
> (rport.router.l3dgw_ports.nth(0)) {
> > > + None -> "",
> > > + Some {var gw_port} -> "
> is_chassis_resident(${json_string_escape(chassis_redirect_name(gw_port.name
> ))})"
> > > + }
> > > } else ();
> > >
> > > addresses.insert(c_addresses)
> > > @@ -415,8 +418,10 @@ function get_garp_nat_addresses(rport:
> Intern<RouterPort>): string = {
> > > for (ipv4_addr in rport.networks.ipv4_addrs) {
> > > garp_info.push("${ipv4_addr.addr}")
> > > };
> > > - if (rport.router.redirect_port_name != "") {
> > > -
> garp_info.push("is_chassis_resident(${rport.router.redirect_port_name})")
> > > + match (rport.router.l3dgw_ports.nth(0)) {
> > > + None -> (),
> > > + Some {var gw_port} -> garp_info.push(
> > > +
> "is_chassis_resident(${json_string_escape(chassis_redirect_name(
> gw_port.name))})")
> > > };
> > > garp_info.join(" ")
> > > }
> > > @@ -453,7 +458,7 @@ OutProxy_Port_Binding(// lrp._uuid is already in
> use; generate a new UUID by
> > > .nat_addresses = set_empty(),
> > > .external_ids = lrp.external_ids) :-
> > > DistributedGatewayPort(lrp, lr_uuid),
> > > - LogicalRouterHAChassisGroup(lr_uuid, hacg_uuid),
> > > + DistributedGatewayPortHAChassisGroup(lrp, hacg_uuid),
> > > var redirect_type = match (lrp.options.get("redirect-type")) {
> > > Some{var value} -> ["redirect-type" -> value],
> > > _ -> map_empty()
> > > @@ -509,7 +514,8 @@ sb::Out_Port_Binding(._uuid =
> pbinding._uuid,
> > > * chassis. RefChassisSet has a row for every logical router. */
> > > relation RefChassis(lr_uuid: uuid, chassis_uuid: uuid)
> > > RefChassis(lr_uuid, chassis_uuid) :-
> > > - LogicalRouterHAChassisGroup(lr_uuid, _),
> > > + DistributedGatewayPortHAChassisGroup(lrp, _),
> > > + DistributedGatewayPort(lrp, lr_uuid),
> > > ConnectedLogicalRouter[(lr_uuid, set_uuid)],
> > > ConnectedLogicalRouter[(lr2_uuid, set_uuid)],
> > > FirstHopLogicalRouter(lr2_uuid, ls_uuid),
> > > @@ -536,7 +542,8 @@ RefChassisSet(lr_uuid, set_empty()) :-
> > > relation HAChassisGroupRefChassisSet(hacg_uuid: uuid,
> > > chassis_uuids: Set<uuid>)
> > > HAChassisGroupRefChassisSet(hacg_uuid, chassis_uuids) :-
> > > - LogicalRouterHAChassisGroup(lr_uuid, hacg_uuid),
> > > + DistributedGatewayPortHAChassisGroup(lrp, hacg_uuid),
> > > + DistributedGatewayPort(lrp, lr_uuid),
> > > RefChassisSet(lr_uuid, chassis_uuids),
> > > var chassis_uuids = chassis_uuids.group_by(hacg_uuid).union().
> > >
> > > @@ -4453,7 +4460,7 @@ for (&SwitchPort(.lsp = lsp,
> > > .peer = Some{&RouterPort{.lrp = lrp,
> > > .is_redirect = is_redirect,
> > > .router = &Router{._uuid =
> lr_uuid,
> > > -
> .redirect_port_name = redirect_port_name}}})
> > > +
> .l3dgw_ports = l3dgw_ports}}})
> > > if (lsp.addresses.contains("router") and lsp.__type !=
> "external"))
> > > {
> > > Some{var mac} = scan_eth_addr(lrp.mac) in {
> > > @@ -4473,6 +4480,14 @@ for (&SwitchPort(.lsp = lsp,
> > > */
> > > lrp.options.get_bool_def("reside-on-redirect-chassis",
> false)) in
> > > var __match = if (add_chassis_resident_check) {
> > > + var redirect_port_name = if (is_redirect) {
> > > + json_string_escape(chassis_redirect_name(lrp.name))
> > > + } else {
> > > + match (l3dgw_ports.nth(0)) {
> > > + Some {var gw_port} ->
> json_string_escape(chassis_redirect_name(gw_port.name)),
> > > + None -> ""
> > > + }
> > > + };
> > > /* The destination lookup flow for the router's
> > > * distributed gateway port MAC address should only be
> > > * programmed on the "redirect-chassis". */
> > > @@ -4872,13 +4887,8 @@ var rLNIR = rEGBIT_LOOKUP_NEIGHBOR_IP_RESULT() in
> > >
> > > /* Check if we need to learn mac-binding from ARP requests. */
> > > for (RouterPortNetworksIPv4Addr(rp@&RouterPort{.router = router},
> addr)) {
> > > - var is_l3dgw_port = match (router.l3dgw_port) {
> > > - Some{l3dgw_lrp} -> l3dgw_lrp._uuid == rp.lrp._uuid,
> > > - None -> false
> > > - } in
> > > - var has_redirect_port = router.redirect_port_name != "" in
> > > - var chassis_residence = match (is_l3dgw_port and
> has_redirect_port) {
> > > - true -> " &&
> is_chassis_resident(${router.redirect_port_name})",
> > > + var chassis_residence = match (rp.is_redirect) {
> > > + true -> " &&
> is_chassis_resident(${json_string_escape(chassis_redirect_name(rp.lrp.name
> ))})",
> > > false -> ""
> > > } in
> > > var rLNR = rEGBIT_LOOKUP_NEIGHBOR_RESULT() in
> > > @@ -5038,7 +5048,7 @@ relation AddChassisResidentCheck_(lrp: uuid,
> add_check: bool)
> > > AddChassisResidentCheck_(lrp._uuid, res) :-
> > > &SwitchPort(.peer = Some{&RouterPort{.lrp = lrp, .router = router,
> .is_redirect = is_redirect}},
> > > .sw = sw),
> > > - router.l3dgw_port.is_some(),
> > > + not router.l3dgw_ports.is_empty(),
> > > not sw.localnet_ports.is_empty(),
> > > var res = if (is_redirect) {
> > > /* Traffic with eth.src = l3dgw_port->lrp_networks.ea
> > > @@ -5143,7 +5153,8 @@ LogicalRouterArpNdFlow(router, nat, None,
> rEG_INPORT_ETH_ADDR(), None, false, 90
> > > * different ETH address.
> > > */
> > > LogicalRouterPortNatArpNdFlow(router, nat, l3dgw_port) :-
> > > - router in &Router(._uuid = lr_uuid, .l3dgw_port =
> Some{l3dgw_port}),
> > > + router in &Router(._uuid = lr_uuid, .l3dgw_ports = l3dgw_ports),
> > > + Some {var l3dgw_port} = l3dgw_ports.nth(0),
> > > LogicalRouterNAT(lr_uuid, nat),
> > > /* Skip SNAT entries for now, we handle unique SNAT IPs separately
> > > * below.
> > > @@ -5151,7 +5162,8 @@ LogicalRouterPortNatArpNdFlow(router, nat,
> l3dgw_port) :-
> > > nat.nat.__type != "snat".
> > > /* Now handle SNAT entries too, one per unique SNAT IP. */
> > > LogicalRouterPortNatArpNdFlow(router, nat, l3dgw_port) :-
> > > - router in &Router(.l3dgw_port = Some{l3dgw_port}, .snat_ips =
> snat_ips),
> > > + router in &Router(.l3dgw_ports = l3dgw_ports, .snat_ips =
> snat_ips),
> > > + Some {var l3dgw_port} = l3dgw_ports.nth(0),
> > > var snat_ip = FlatMap(snat_ips),
> > > (var ip, var nats) = snat_ip,
> > > Some{var nat} = nats.nth(0).
> > > @@ -5181,9 +5193,9 @@ LogicalRouterArpNdFlow(router, nat, Some{lrp},
> mac, None, true, 91) :-
> > > * upstream MAC learning points to the gateway chassis.
> > > * Also need to avoid generation of multiple ARP responses
> > > * from different chassis. */
> > > - match (router.redirect_port_name) {
> > > - "" -> "",
> > > - s -> "is_chassis_resident(${s})"
> > > + match (router.l3dgw_ports.nth(0)) {
> > > + None -> "",
> > > + Some {var gw_port} ->
> "is_chassis_resident(${json_string_escape(chassis_redirect_name(gw_port.name
> ))})"
> > > }
> > > )
> > > }.
> > > @@ -5328,7 +5340,15 @@ for (RouterPortNetworksIPv4Addr(.port =
> &RouterPort{.lrp = lrp,
> > > var __match =
> > > "arp.spa == ${addr.match_network()}" ++
> > > if (add_chassis_resident_check) {
> > > - " && is_chassis_resident(${router.redirect_port_name})"
> > > + var redirect_port_name = if (is_redirect) {
> > > + json_string_escape(chassis_redirect_name(lrp.name))
> > > + } else {
> > > + match (router.l3dgw_ports.nth(0)) {
> > > + None -> "",
> > > + Some {var gw_port} ->
> json_string_escape(chassis_redirect_name(gw_port.name))
> > > + }
> > > + };
> > > + " && is_chassis_resident(${redirect_port_name})"
> > > } else "" in
> > > LogicalRouterArpFlow(.lr = router,
> > > .lrp = Some{lrp},
> > > @@ -5347,7 +5367,7 @@ for (&RouterPort(.lrp = lrp,
> > > .networks = networks,
> > > .is_redirect = is_redirect))
> > > var residence_check = match (is_redirect) {
> > > - true -> Some{"is_chassis_resident(${router.redirect_port_name})"},
> > > + true ->
> Some{"is_chassis_resident(${json_string_escape(chassis_redirect_name(
> lrp.name))})"},
> > > false -> None
> > > } in {
> > > (var all_ips_v4, _) = get_router_load_balancer_ips(router, false)
> in {
> > > @@ -5417,7 +5437,7 @@ Flow(.logical_datapath = lr_uuid,
> > > for (RouterPortNetworksIPv4Addr(
> > > .port = &RouterPort{
> > > .router = &Router{._uuid = lr_uuid,
> > > - .l3dgw_port = None,
> > > + .l3dgw_ports = vec_empty(),
> > > .is_gateway = false,
> > > .copp = copp},
> > > .lrp = lrp},
> > > @@ -5553,7 +5573,7 @@ for (RouterPortNetworksIPv6Addr(.port =
> &RouterPort{.lrp = lrp,
> > > /* UDP/TCP/SCTP port unreachable */
> > > for (RouterPortNetworksIPv6Addr(
> > > .port = &RouterPort{.router = &Router{._uuid = lr_uuid,
> > > - .l3dgw_port = None,
> > > + .l3dgw_ports =
> vec_empty(),
> > > .is_gateway = false,
> > > .copp = copp},
> > > .lrp = lrp,
> > > @@ -5681,11 +5701,11 @@ for (r in &Router(._uuid = lr_uuid)) {
> > > }
> > >
> > > for (r in &Router(._uuid = lr_uuid,
> > > - .l3dgw_port = l3dgw_port,
> > > + .l3dgw_ports = l3dgw_ports,
> > > .is_gateway = is_gateway,
> > > .nat = nat,
> > > .load_balancer = load_balancer)
> > > - if (l3dgw_port.is_some() or is_gateway) and (not is_empty(nat) or
> not is_empty(load_balancer))) {
> > > + if (l3dgw_ports.len() > 0 or is_gateway) and (not is_empty(nat)
> or not is_empty(load_balancer))) {
> > > /* If the router has load balancer or DNAT rules, re-circulate
> every packet
> > > * through the DNAT zone so that packets that need to be unDNATed
> in the
> > > * reverse direction get unDNATed.
> > > @@ -5768,7 +5788,7 @@ function lrouter_nat_add_ext_ip_match(
> > > },
> > > false -> {
> > > /* S_ROUTER_OUT_SNAT uses priority (mask + 1 + 128
> + 1) */
> > > - var is_gw_router = router.l3dgw_port == None;
> > > + var is_gw_router = router.l3dgw_ports.is_empty();
> > > var mask_1bits = mask.cidr_bits().unwrap_or(8'd0)
> as integer;
> > > mask_1bits + 2 + { if (not is_gw_router) 128 else
> 0 }
> > > }
> > > @@ -5873,10 +5893,9 @@ VirtualLogicalPort(Some{logical_port}) :-
> > > * l3dgw_port (router has a port with "redirect-chassis"
> > > * specified). */
> > > for (r in &Router(._uuid = lr_uuid,
> > > - .l3dgw_port = l3dgw_port,
> > > - .redirect_port_name = redirect_port_name,
> > > + .l3dgw_ports = l3dgw_ports,
> > > .is_gateway = is_gateway)
> > > - if l3dgw_port.is_some() or is_gateway)
> > > + if not l3dgw_ports.is_empty() or is_gateway)
> > > {
> > > for (LogicalRouterNAT(.lr = lr_uuid, .nat = nat)) {
> > > var ipX = nat.external_ip.ipX() in
> > > @@ -5894,7 +5913,7 @@ for (r in &Router(._uuid = lr_uuid,
> > > } in
> > > /* For distributed router NAT, determine whether this NAT rule
> > > * satisfies the conditions for distributed NAT processing. */
> > > - var mac = match ((l3dgw_port.is_some() and nat.nat.__type ==
> "dnat_and_snat",
> > > + var mac = match ((not l3dgw_ports.is_empty() and
> nat.nat.__type == "dnat_and_snat",
> > > nat.nat.logical_port, nat.external_mac)) {
> > > (true, Some{_}, Some{mac}) -> Some{mac},
> > > _ -> None
> > > @@ -5912,7 +5931,7 @@ for (r in &Router(._uuid = lr_uuid,
> > > * not know about the possibility of eventual additional
> SNAT in
> > > * egress pipeline. */
> > > if (nat.nat.__type == "snat" or nat.nat.__type ==
> "dnat_and_snat") {
> > > - if (l3dgw_port == None) {
> > > + if (l3dgw_ports.is_empty()) {
> > > /* Gateway router. */
> > > var actions = if (stateless) {
> > > "${ipX}.dst=${nat.nat.logical_ip}; next;"
> > > @@ -5926,7 +5945,7 @@ for (r in &Router(._uuid = lr_uuid,
> > > .actions = actions,
> > > .external_ids = stage_hint(nat.nat._uuid))
> > > };
> > > - Some{var gwport} = l3dgw_port in {
> > > + Some {var gwport} = l3dgw_ports.nth(0) in {
> > > /* Distributed router. */
> > >
> > > /* Traffic received on l3dgw_port is subject to
> NAT. */
> > > @@ -5936,7 +5955,7 @@ for (r in &Router(._uuid = lr_uuid,
> > > if (mac == None) {
> > > /* Flows for NAT rules that are
> centralized are only
> > > * programmed on the "redirect-chassis". */
> > > - " &&
> is_chassis_resident(${redirect_port_name})"
> > > + " &&
> is_chassis_resident(${json_string_escape(chassis_redirect_name(gwport.name
> ))})"
> > > } else { "" } in
> > > var actions = if (stateless) {
> > > "${ipX}.dst=${nat.nat.logical_ip}; next;"
> > > @@ -5962,7 +5981,7 @@ for (r in &Router(._uuid = lr_uuid,
> > > ""
> > > } in
> > > if (nat.nat.__type == "dnat" or nat.nat.__type ==
> "dnat_and_snat") {
> > > - None = l3dgw_port in
> > > + l3dgw_ports.is_empty() in
> > > var __match = "ip && ${ipX}.dst ==
> ${nat.nat.external_ip}" in
> > > (var ext_ip_match, var ext_flow) =
> lrouter_nat_add_ext_ip_match(
> > > r, nat, __match, ipX, true, mask) in
> > > @@ -5994,14 +6013,14 @@ for (r in &Router(._uuid = lr_uuid,
> > > .external_ids = stage_hint(nat.nat._uuid))
> > > };
> > >
> > > - Some{var gwport} = l3dgw_port in
> > > + Some {var gwport} = l3dgw_ports.nth(0) in
> > > var __match =
> > > "ip && ${ipX}.dst == ${nat.nat.external_ip}"
> > > " && inport == ${json_string_escape(gwport.name)}"
> ++
> > > if (mac == None) {
> > > /* Flows for NAT rules that are centralized
> are only
> > > * programmed on the "redirect-chassis". */
> > > - " &&
> is_chassis_resident(${redirect_port_name})"
> > > + " &&
> is_chassis_resident(${json_string_escape(chassis_redirect_name(gwport.name
> ))})"
> > > } else { "" } in
> > > (var ext_ip_match, var ext_flow) =
> lrouter_nat_add_ext_ip_match(
> > > r, nat, __match, ipX, true, mask) in
> > > @@ -6025,7 +6044,7 @@ for (r in &Router(._uuid = lr_uuid,
> > > };
> > >
> > > /* ARP resolve for NAT IPs. */
> > > - Some{var gwport} = l3dgw_port in {
> > > + Some {var gwport} = l3dgw_ports.nth(0) in {
> > > var gwport_name = json_string_escape(gwport.name) in {
> > > if (nat.nat.__type == "snat") {
> > > var __match = "inport == ${gwport_name} && "
> > > @@ -6062,14 +6081,14 @@ for (r in &Router(._uuid = lr_uuid,
> > > * Note that this only applies for NAT on a distributed
> router.
> > > */
> > > if ((nat.nat.__type == "dnat" or nat.nat.__type ==
> "dnat_and_snat")) {
> > > - Some{var gwport} = l3dgw_port in
> > > + Some {var gwport} = l3dgw_ports.nth(0) in
> > > var __match =
> > > "ip && ${ipX}.src == ${nat.nat.logical_ip}"
> > > " && outport == ${json_string_escape(gwport.name)}"
> ++
> > > if (mac == None) {
> > > /* Flows for NAT rules that are centralized
> are only
> > > * programmed on the "redirect-chassis". */
> > > - " &&
> is_chassis_resident(${redirect_port_name})"
> > > + " &&
> is_chassis_resident(${json_string_escape(chassis_redirect_name(gwport.name
> ))})"
> > > } else { "" } in
> > > var actions =
> > > match (mac) {
> > > @@ -6099,7 +6118,7 @@ for (r in &Router(._uuid = lr_uuid,
> > > ""
> > > } in
> > > if (nat.nat.__type == "snat" or nat.nat.__type ==
> "dnat_and_snat") {
> > > - None = l3dgw_port in
> > > + l3dgw_ports.is_empty() in
> > > var __match = "ip && ${ipX}.src ==
> ${nat.nat.logical_ip}" in
> > > (var ext_ip_match, var ext_flow) =
> lrouter_nat_add_ext_ip_match(
> > > r, nat, __match, ipX, false, mask) in
> > > @@ -6124,14 +6143,14 @@ for (r in &Router(._uuid = lr_uuid,
> > > .external_ids = stage_hint(nat.nat._uuid))
> > > };
> > >
> > > - Some{var gwport} = l3dgw_port in
> > > + Some {var gwport} = l3dgw_ports.nth(0) in
> > > var __match =
> > > "ip && ${ipX}.src == ${nat.nat.logical_ip}"
> > > " && outport == ${json_string_escape(gwport.name)}"
> ++
> > > if (mac == None) {
> > > /* Flows for NAT rules that are centralized
> are only
> > > * programmed on the "redirect-chassis". */
> > > - " &&
> is_chassis_resident(${redirect_port_name})"
> > > + " &&
> is_chassis_resident(${json_string_escape(chassis_redirect_name(gwport.name
> ))})"
> > > } else { "" } in
> > > (var ext_ip_match, var ext_flow) =
> lrouter_nat_add_ext_ip_match(
> > > r, nat, __match, ipX, false, mask) in
> > > @@ -6169,7 +6188,7 @@ for (r in &Router(._uuid = lr_uuid,
> > > * on the l3dgw_port instance where nat->logical_port is
> > > * resident. */
> > > Some{var mac_addr} = mac in
> > > - Some{var gwport} = l3dgw_port in
> > > + Some{var gwport} = l3dgw_ports.nth(0) in
> > > Some{var logical_port} = nat.nat.logical_port in
> > > var __match =
> > > "eth.dst == ${mac_addr} && inport ==
> ${json_string_escape(gwport.name)}"
> > > @@ -6195,7 +6214,7 @@ for (r in &Router(._uuid = lr_uuid,
> > > * stage is sent out with proper IP/MAC src addresses
> > > */
> > > Some{var mac_addr} = mac in
> > > - Some{var gwport} = l3dgw_port in
> > > + Some{var gwport} = l3dgw_ports.nth(0) in
> > > Some{var logical_port} = nat.nat.logical_port in
> > > Some{var external_mac} = nat.nat.external_mac in
> > > var __match =
> > > @@ -6214,7 +6233,7 @@ for (r in &Router(._uuid = lr_uuid,
> > > .external_ids = stage_hint(nat.nat._uuid));
> > >
> > > for (VirtualLogicalPort(nat.nat.logical_port)) {
> > > - Some{var gwport} = l3dgw_port in
> > > + Some{var gwport} = l3dgw_ports.nth(0) in
> > > Flow(.logical_datapath = lr_uuid,
> > > .stage = s_ROUTER_IN_GW_REDIRECT(),
> > > .priority = 80,
> > > @@ -6229,14 +6248,14 @@ for (r in &Router(._uuid = lr_uuid,
> > > * gateway port have ip.dst matching a NAT external IP,
> then
> > > * loop a clone of the packet back to the beginning of the
> > > * ingress pipeline with inport = outport. */
> > > - Some{var gwport} = l3dgw_port in
> > > + Some{var gwport} = l3dgw_ports.nth(0) in
> > > /* Distributed router. */
> > > Some{var port} = match (mac) {
> > > Some{_} -> match (nat.nat.logical_port) {
> > > Some{name} ->
> Some{json_string_escape(name)},
> > > None -> None: Option<string>
> > > },
> > > - None -> Some{redirect_port_name}
> > > + None -> Some{json_string_escape(chassis_redirect_name(
> gwport.name))}
> > > } in
> > > var __match = "${ipX}.dst == ${nat.nat.external_ip} &&
> outport == ${json_string_escape(gwport.name)} &&
> is_chassis_resident(${port})" in
> > > var regs = {
> > > @@ -6264,7 +6283,7 @@ for (r in &Router(._uuid = lr_uuid,
> > > };
> > >
> > > /* Handle force SNAT options set in the gateway router. */
> > > - if (l3dgw_port == None) {
> > > + if (l3dgw_ports.is_empty()) {
> > > var dnat_force_snat_ips = get_force_snat_ip(r.options, "dnat")
> in
> > > if (not dnat_force_snat_ips.is_empty())
> > > LogicalRouterForceSnatFlows(.logical_router = lr_uuid,
> > > @@ -6292,14 +6311,13 @@ function nats_contain_vip(nats: Vec<NAT>, vip:
> v46_ip): bool {
> > > * Gateway routers or router with gateway port. */
> > > for (RouterLBVIP(
> > > .router = r@&Router{._uuid = lr_uuid,
> > > - .l3dgw_port = l3dgw_port,
> > > - .redirect_port_name = redirect_port_name,
> > > + .l3dgw_ports = l3dgw_ports,
> > > .is_gateway = is_gateway,
> > > .nats = nats},
> > > .lb = lb,
> > > .vip = vip,
> > > .backends = backends)
> > > - if l3dgw_port.is_some() or is_gateway)
> > > + if not l3dgw_ports.is_empty() or is_gateway)
> > > {
> > > if (backends == "" and not lb.options.get_bool_def("reject",
> false)) {
> > > for (LoadBalancerEmptyEvents(lb)) {
> > > @@ -6368,8 +6386,8 @@ for (RouterLBVIP(
> > > (110, "")
> > > } in
> > > var __match = match1 ++ match2 ++
> > > - match ((l3dgw_port, backends != "" or
> lb.options.get_bool_def("reject", false))) {
> > > - (Some{gwport}, true) -> " &&
> is_chassis_resident(${redirect_port_name})",
> > > + match ((l3dgw_ports.nth(0), backends != "" or
> lb.options.get_bool_def("reject", false))) {
> > > + (Some{gw_port}, true) -> " &&
> is_chassis_resident(${json_string_escape(chassis_redirect_name(gw_port.name
> ))})",
> > > _ -> ""
> > > } in
> > > var snat_for_lb = snat_for_lb(r.options, lb) in
> > > @@ -6381,8 +6399,8 @@ for (RouterLBVIP(
> > > } else {
> > > ""
> > > } ++
> > > - match ((l3dgw_port, backends != "" or
> lb.options.get_bool_def("reject", false))) {
> > > - (Some{gwport}, true) -> " &&
> is_chassis_resident(${redirect_port_name})",
> > > + match ((l3dgw_ports.nth(0), backends != "" or
> lb.options.get_bool_def("reject", false))) {
> > > + (Some {var gw_port}, true) -> " &&
> is_chassis_resident(${json_string_escape(chassis_redirect_name(gw_port.name
> ))})",
> > > _ -> ""
> > > } in
> > > var actions =
> > > @@ -6421,7 +6439,7 @@ for (RouterLBVIP(
> > > .external_ids = stage_hint(lb._uuid))
> > > };
> > >
> > > - Some{var gwport} = l3dgw_port in
> > > + Some{var gwport} = l3dgw_ports.nth(0) in
> > > /* Add logical flows to UNDNAT the load balanced reverse
> traffic in
> > > * the router egress pipleine stage - S_ROUTER_OUT_UNDNAT
> if the logical
> > > * router has a gateway router port associated.
> > > @@ -6446,7 +6464,7 @@ for (RouterLBVIP(
> > > var undnat_match =
> > > "${ip_address.ipX()} && (" ++ conds.join(" || ") ++
> > > ") && outport == ${json_string_escape(gwport.name)} &&
> "
> > > - "is_chassis_resident(${redirect_port_name})" in
> > > +
>
> "is_chassis_resident(${json_string_escape(chassis_redirect_name(gwport.name))})"
> in
> > > var action =
> > > match (snat_for_lb) {
> > > SkipSNAT -> "flags.skip_snat_for_lb = 1; ct_dnat;",
> > > @@ -6477,14 +6495,14 @@ MeteredFlow(.logical_datapath = r._uuid,
> > > .controller_meter = meter,
> > > .external_ids = stage_hint(lb._uuid)) :-
> > > r in &Router(),
> > > - r.l3dgw_port.is_some() or r.is_gateway,
> > > + r.l3dgw_ports.len() > 0 or r.is_gateway,
> > > LBVIPWithStatus[lbvip@&LBVIPWithStatus{.lb = lb}],
> > > r.load_balancer.contains(lb._uuid),
> > > var __match
> > > = "ct.new && " ++
> > > get_match_for_lb_key(lbvip.vip_addr, lbvip.vip_port,
> lb.protocol, true, true) ++
> > > - match (r.l3dgw_port) {
> > > - Some{gwport} -> " &&
> is_chassis_resident(${r.redirect_port_name})",
> > > + match (r.l3dgw_ports.nth(0)) {
> > > + Some{gw_port} -> " &&
> is_chassis_resident(${json_string_escape(chassis_redirect_name(gw_port.name
> ))})",
> > > _ -> ""
> > > },
> > > var priority = if (lbvip.vip_port != 0) 120 else 110,
> > > @@ -7290,11 +7308,11 @@ Flow(.logical_datapath = router._uuid,
> > > .stage = s_ROUTER_IN_ARP_RESOLVE(),
> > > .priority = 50,
> > > .__match = "outport == ${rp.json_name} && "
> > > -
> "!is_chassis_resident(${router.redirect_port_name})",
> > > +
> "!is_chassis_resident(${json_string_escape(chassis_redirect_name(
> l3dgw_port.name))})",
> > > .actions = "eth.dst = ${rp.networks.ea}; next;",
> > > .external_ids = stage_hint(lrp._uuid)) :-
> > > rp in &RouterPort(.lrp = lrp, .router = router),
> > > - router.redirect_port_name != "",
> > > + Some{var l3dgw_port} = router.l3dgw_ports.nth(0),
> > > Some{"bridged"} = lrp.options.get("redirect-type").
> > >
> > >
> > > @@ -7537,9 +7555,8 @@ Flow(.logical_datapath = lr_uuid,
> > > "next;",
> > > .external_ids = stage_hint(l3dgw_port._uuid)) :-
> > > r in &Router(._uuid = lr_uuid),
> > > - Some{var l3dgw_port} = r.l3dgw_port,
> > > + Some{var l3dgw_port} = r.l3dgw_ports.nth(0),
> > > var l3dgw_port_json_name = json_string_escape(l3dgw_port.name),
> > > - r.redirect_port_name != "",
> > > var gw_mtu = l3dgw_port.options.get_int_def("gateway_mtu", 0),
> > > gw_mtu > 0,
> > > var mtu = gw_mtu + vLAN_ETH_HEADER_LEN().
> > > @@ -7564,9 +7581,8 @@ MeteredFlow(.logical_datapath = lr_uuid,
> > > .controller_meter = r.copp.get(cOPP_ICMP4_ERR()),
> > > .external_ids = stage_hint(rp.lrp._uuid)) :-
> > > r in &Router(._uuid = lr_uuid),
> > > - Some{var l3dgw_port} = r.l3dgw_port,
> > > + Some{var l3dgw_port} = r.l3dgw_ports.nth(0),
> > > var l3dgw_port_json_name = json_string_escape(l3dgw_port.name),
> > > - r.redirect_port_name != "",
> > > var gw_mtu = l3dgw_port.options.get_int_def("gateway_mtu", 0),
> > > gw_mtu > 0,
> > > rp in &RouterPort(.router = r),
> > > @@ -7593,9 +7609,8 @@ MeteredFlow(.logical_datapath = lr_uuid,
> > > .controller_meter = r.copp.get(cOPP_ICMP6_ERR()),
> > > .external_ids = stage_hint(rp.lrp._uuid)) :-
> > > r in &Router(._uuid = lr_uuid),
> > > - Some{var l3dgw_port} = r.l3dgw_port,
> > > + Some{var l3dgw_port} = r.l3dgw_ports.nth(0),
> > > var l3dgw_port_json_name = json_string_escape(l3dgw_port.name),
> > > - r.redirect_port_name != "",
> > > var gw_mtu = l3dgw_port.options.get_int_def("gateway_mtu", 0),
> > > gw_mtu > 0,
> > > rp in &RouterPort(.router = r),
> > > @@ -7609,21 +7624,20 @@ MeteredFlow(.logical_datapath = lr_uuid,
> > > * of the traffic to the l3redirect_port which represents
> > > * the central instance of the l3dgw_port.
> > > */
> > > -for (&Router(._uuid = lr_uuid,
> > > - .l3dgw_port = l3dgw_port,
> > > - .redirect_port_name = redirect_port_name))
> > > +for (&Router(._uuid = lr_uuid))
> > > {
> > > /* For traffic with outport == l3dgw_port, if the
> > > * packet did not match any higher priority redirect
> > > * rule, then the traffic is redirected to the central
> > > * instance of the l3dgw_port. */
> > > - Some{var gwport} = l3dgw_port in
> > > - Flow(.logical_datapath = lr_uuid,
> > > - .stage = s_ROUTER_IN_GW_REDIRECT(),
> > > - .priority = 50,
> > > - .__match = "outport == ${json_string_escape(
> gwport.name)}",
> > > - .actions = "outport = ${redirect_port_name}; next;",
> > > - .external_ids = stage_hint(gwport._uuid));
> > > + for (DistributedGatewayPort(lrp, lr_uuid)) {
> > > + Flow(.logical_datapath = lr_uuid,
> > > + .stage = s_ROUTER_IN_GW_REDIRECT(),
> > > + .priority = 50,
> > > + .__match = "outport == ${json_string_escape(
> lrp.name)}",
> > > + .actions = "outport =
> ${json_string_escape(chassis_redirect_name(lrp.name))}; next;",
> > > + .external_ids = stage_hint(lrp._uuid))
> > > + };
> > >
> > > /* Packets are allowed by default. */
> > > Flow(.logical_datapath = lr_uuid,
> > > diff --git a/ovn-architecture.7.xml b/ovn-architecture.7.xml
> > > index 0eef9b739..3598b5073 100644
> > > --- a/ovn-architecture.7.xml
> > > +++ b/ovn-architecture.7.xml
> > > @@ -731,6 +731,13 @@
> > > highest-priority gateway that is online.
> > > </p>
> > >
> > > + <p>
> > > + A logical router can have multiple distributed gateway ports, each
> > > + connecting different external networks. However, some features,
> such as NAT
> > > + and load balancers, are not supported yet for logical routers with
> more
> > > + than one distributed gateway port configured.
> > > + </p>
> > > +
> > > <h4>Physical VLAN MTU Issues</h4>
> > >
> > > <p>
> > > @@ -1968,8 +1975,9 @@
> > >
> > > <p>
> > > If the logical router doesn't have a distributed gateway port
> connecting
> > > - to the localnet logical switch which provides external
> connectivity,
> > > - then this option is ignored by <code>OVN</code>.
> > > + to the localnet logical switch which provides external
> connectivity, or
> > > + if it has more than one distributed gateway ports, then this
> option is
> > > + ignored by <code>OVN</code>.
> > > </p>
> > >
> > > <p>
> > > @@ -2086,6 +2094,13 @@
> > > a tunnel.
> > > </p>
> > >
> > > + <p>
> > > + If the logical router doesn't have a distributed gateway port
> connecting
> > > + to the localnet logical switch which provides external
> connectivity, or
> > > + if it has more than one distributed gateway ports, then this
> option is
> > > + ignored by <code>OVN</code>.
> > > + </p>
> > > +
> > > <p>
> > > Following happens for bridged redirection:
> > > </p>
> > > diff --git a/ovn-nb.xml b/ovn-nb.xml
> > > index c1176e81f..ec51b5608 100644
> > > --- a/ovn-nb.xml
> > > +++ b/ovn-nb.xml
> > > @@ -2032,13 +2032,14 @@
> > >
> > > <column name="nat">
> > > One or more NAT rules for the router. NAT rules only work on
> > > - Gateway routers, and on distributed routers with logical gateway
> ports.
> > > + Gateway routers, and on distributed routers with one and only one
> > > + distributed gateway port.
> > > </column>
> > >
> > > <column name="load_balancer">
> > > Load balance a virtual ip address to a set of logical port ip
> > > addresses. Load balancer rules only work on the Gateway routers
> or
> > > - routers with distributed gateway ports.
> > > + routers with one and only one distributed gateway port.
> > > </column>
> > >
> > > <group title="Naming">
> > > @@ -2453,8 +2454,7 @@
> > > If either of these are set, this logical router port
> represents a
> > > distributed gateway port that connects this router to a
> > > logical switch with a <code>localnet</code> port or a
> > > - connection to another OVN deployment. There may be at most
> > > - one such logical router port on each logical router.
> > > + connection to another OVN deployment.
> > > </p>
> > >
> > > <p>
> > > @@ -2476,8 +2476,16 @@
> > > </p>
> > >
> > > <p>
> > > - When more than one gateway chassis is specified, OVN only uses
> > > - one at a time. OVN can rely on OVS BFD implementation to
> monitor
> > > + There can be more than one distributed gateway ports configured
> > > + on each logical router, each connecting to different L2
> segments.
> > > + However, features such as NAT and load-balancer are not
> supported
> > > + on logical routers with more than one distributed gateway
> ports.
> > > + </p>
> > > +
> > > + <p>
> > > + For each distributed gateway port, it may have more than one
> gateway
> > > + chassises. When more than one gateway chassis is specified,
> OVN only
> > > + uses one at a time. OVN can rely on OVS BFD implementation to
> monitor
> > > gateway connectivity, preferring the highest-priority gateway
> > > that is online. Priorities are specified in the
> <code>priority</code>
> > > column of <ref table="Gateway_Chassis"/> or <ref
> table="HA_Chassis"/>.
> > > @@ -2563,8 +2571,8 @@
> > > </p>
> > >
> > > <p>
> > > - OVN honors this option only if the logical router has a
> distributed
> > > - gateway port and if the LRP's peer switch has a
> > > + OVN honors this option only if the logical router has one
> and only
> > > + one distributed gateway port and if the LRP's peer switch
> has a
> > > <code>localnet</code> port.
> > > </p>
> > > </column>
> > > @@ -2588,7 +2596,8 @@
> > > <p>
> > > Setting this option to <code>overlay</code> or leaving it
> unset has
> > > no effect. This option may usefully be set only on a
> distributed
> > > - gateway port. It is otherwise ignored.
> > > + gateway port when there is one and only one distributed
> gateway
> > > + port on the logical router. It is otherwise ignored.
> > > </p>
> > > </column>
> > > </group>
> > > diff --git a/ovs b/ovs
> > > index daf627f45..e6ad4d8d9 160000
> > > --- a/ovs
> > > +++ b/ovs
> > > @@ -1 +1 @@
> > > -Subproject commit daf627f459ffbc7171d42a2c01f80754bfd54edc
> > > +Subproject commit e6ad4d8d9c9273f226ec9a993b64fccfb50bdf4c
> > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at
> > > index 94405349e..dfb5b70f9 100644
> > > --- a/tests/ovn-northd.at
> > > +++ b/tests/ovn-northd.at
> > > @@ -4815,3 +4815,95 @@ AT_CHECK([ovn-sbctl --columns=tags list
> logical_flow | grep lsp0 -c], [0], [dnl
> > >
> > > AT_CLEANUP
> > > ])
> > > +
> > > +OVN_FOR_EACH_NORTHD([
> > > +AT_SETUP([ovn-northd -- lr multiple gw ports])
> > > +AT_KEYWORDS([multiple-l3dgw-ports])
> > > +ovn_start
> > > +
> > > +# Logical network:
> > > +# 1 Logical Router, 3 bridged Logical Switches,
> > > +# 1 gateway chassis attached to each corresponding LRP.
> > > +#
> > > +# | S1 (gw1)
> > > +# |
> > > +# ls ---- DR -- S3 (gw3)
> > > +# (20.0.0.0/24) |
> > > +# | S2 (gw2)
> > > +#
> > > +# Validate basic LR logical flows.
> > > +
> > > +check ovn-sbctl chassis-add gw1 geneve 127.0.0.1
> > > +check ovn-sbctl chassis-add gw2 geneve 128.0.0.1
> > > +check ovn-sbctl chassis-add gw3 geneve 129.0.0.1
> > > +
> > > +check ovn-nbctl lr-add DR
> > > +check ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24
> > > +check ovn-nbctl lrp-add DR DR-S2 03:ac:10:01:00:01 10.0.0.1/24
> > > +check ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.1/24
> > > +check ovn-nbctl lrp-add DR DR-ls 05:ac:10:01:00:01 20.0.0.1/24
> > > +
> > > +check ovn-nbctl ls-add S1
> > > +check ovn-nbctl lsp-add S1 S1-DR
> > > +check ovn-nbctl lsp-set-type S1-DR router
> > > +check ovn-nbctl lsp-set-addresses S1-DR router
> > > +check ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1
> > > +
> > > +check ovn-nbctl ls-add S2
> > > +check ovn-nbctl lsp-add S2 S2-DR
> > > +check ovn-nbctl lsp-set-type S2-DR router
> > > +check ovn-nbctl lsp-set-addresses S2-DR router
> > > +check ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2
> > > +
> > > +check ovn-nbctl ls-add S3
> > > +check ovn-nbctl lsp-add S3 S3-DR
> > > +check ovn-nbctl lsp-set-type S3-DR router
> > > +check ovn-nbctl lsp-set-addresses S3-DR router
> > > +check ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3
> > > +
> > > +check ovn-nbctl ls-add ls
> > > +check ovn-nbctl lsp-add ls ls-DR
> > > +check ovn-nbctl lsp-set-type ls-DR router
> > > +check ovn-nbctl lsp-set-addresses ls-DR router
> > > +check ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls
> > > +
> > > +check ovn-nbctl lrp-set-gateway-chassis DR-S1 gw1
> > > +check ovn-nbctl lrp-set-gateway-chassis DR-S2 gw2
> > > +check ovn-nbctl lrp-set-gateway-chassis DR-S3 gw3
> > > +
> > > +check ovn-nbctl --wait=sb sync
> > > +
> > > +ovn-sbctl dump-flows DR > lrflows
> > > +AT_CAPTURE_FILE([lrflows])
> > > +
> > > +# Check the flows in lr_in_admission stage
> > > +AT_CHECK([grep lr_in_admission lrflows | grep cr-DR | wc -l], [0], [3
> > > +])
> > > +AT_CHECK([grep lr_in_admission lrflows | grep cr-DR-S1 | wc -l], [0],
> [1
> > > +])
> > > +AT_CHECK([grep lr_in_admission lrflows | grep cr-DR-S2 | wc -l], [0],
> [1
> > > +])
> > > +AT_CHECK([grep lr_in_admission lrflows | grep cr-DR-S3 | wc -l], [0],
> [1
> > > +])
> > > +
> >
> > I'd suggest to check for the exact flow match actions rather than using
> "wc -l".
> >
> > This would ensure that both northd-c and northd-ddlog generate the
> > exact same flows.
> >
>
> Ack
>
> > > +# Check the flows in lr_in_lookup_neighbor stage
> > > +AT_CHECK([grep lr_in_lookup_neighbor lrflows | grep cr-DR | wc -l],
> [0], [3
> > > +])
> > > +AT_CHECK([grep lr_in_lookup_neighbor lrflows | grep cr-DR-S1 | wc -l],
> [0], [1
> > > +])
> > > +AT_CHECK([grep lr_in_lookup_neighbor lrflows | grep cr-DR-S2 | wc -l],
> [0], [1
> > > +])
> > > +AT_CHECK([grep lr_in_lookup_neighbor lrflows | grep cr-DR-S3 | wc -l],
> [0], [1
> > > +])
> > > +
> > > +# Check the flows in lr_in_gw_redirect stage
> > > +AT_CHECK([grep lr_in_gw_redirect lrflows | grep cr-DR | wc -l], [0], [3
> > > +])
> > > +AT_CHECK([grep lr_in_gw_redirect lrflows | grep cr-DR-S1 | wc -l],
> [0], [1
> > > +])
> > > +AT_CHECK([grep lr_in_gw_redirect lrflows | grep cr-DR-S2 | wc -l],
> [0], [1
> > > +])
> > > +AT_CHECK([grep lr_in_gw_redirect lrflows | grep cr-DR-S3 | wc -l],
> [0], [1
> > > +])
> > > +AT_CLEANUP
> > > +])
> > > diff --git a/tests/ovn.at b/tests/ovn.at
> > > index 13d860f10..13f2a4990 100644
> > > --- a/tests/ovn.at
> > > +++ b/tests/ovn.at
> > > @@ -27351,3 +27351,310 @@ AT_CHECK([ovs-ofctl dump-flows br-int
> table=44 | grep 10.0.0.144], [0], [ignore]
> > > OVN_CLEANUP([hv1])
> > > AT_CLEANUP
> > > ])
> > > +
> > > +OVN_FOR_EACH_NORTHD([
> > > +AT_SETUP([ovn -- lr multiple gw ports])
> > > +AT_KEYWORDS([multiple-l3dgw-ports])
> > > +ovn_start
> > > +
> > > +# Logical network:
> > > +# 1 LR, 3 Logical Switches,
> > > +# 1 gateway chassis attached to each corresponding LRP.
> > > +#
> > > +# | S1 (gw1)
> > > +# |
> > > +# ls ---- DR -- S3 (gw3)
> > > +# (20.0.0.0/24) |
> > > +# | S2 (gw2)
> > > +#
> > > +# S1 - VLAN 1000
> > > +# S2 - VLAN 2000
> > > +# S3 - VLAN 3000
> > > +#
> > > +# 5 chassis(s), HV1----HV5
> > > +#
> > > +# HV1 - VIF11
> > > +# HV2 - Gateway chassis gw1
> > > +# HV3 - Gateway chassis gw2
> > > +# HV4 - Gateway chassis gw3
> > > +# HV5 - North endpoint
> > > +
> > > +ovn-nbctl lr-add DR
> > > +ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24
> > > +ovn-nbctl lrp-add DR DR-S2 08:ac:10:01:00:01 10.0.0.1/24
> > > +ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.1/24
> > > +ovn-nbctl lrp-add DR DR-ls 06:ac:10:01:00:01 20.0.0.1/24
> > > +
> > > +ovn-nbctl ls-add S1
> > > +ovn-nbctl lsp-add S1 S1-DR
> > > +ovn-nbctl lsp-set-type S1-DR router
> > > +ovn-nbctl lsp-set-addresses S1-DR router
> > > +ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1
> > > +ovn-nbctl lsp-add S1 ln1 "" 1000
> > > +ovn-nbctl lsp-set-addresses ln1 unknown
> > > +ovn-nbctl lsp-set-type ln1 localnet
> > > +ovn-nbctl lsp-set-options ln1 network_name=phys
> > > +
> > > +ovn-nbctl ls-add S2
> > > +ovn-nbctl lsp-add S2 S2-DR
> > > +ovn-nbctl lsp-set-type S2-DR router
> > > +ovn-nbctl lsp-set-addresses S2-DR router
> > > +ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2
> > > +ovn-nbctl lsp-add S2 ln2 "" 2000
> > > +ovn-nbctl lsp-set-addresses ln2 unknown
> > > +ovn-nbctl lsp-set-type ln2 localnet
> > > +ovn-nbctl lsp-set-options ln2 network_name=phys
> > > +
> > > +ovn-nbctl ls-add S3
> > > +ovn-nbctl lsp-add S3 S3-DR
> > > +ovn-nbctl lsp-set-type S3-DR router
> > > +ovn-nbctl lsp-set-addresses S3-DR router
> > > +ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3
> > > +ovn-nbctl lsp-add S3 ln3 "" 3000
> > > +ovn-nbctl lsp-set-addresses ln3 unknown
> > > +ovn-nbctl lsp-set-type ln3 localnet
> > > +ovn-nbctl lsp-set-options ln3 network_name=phys
> > > +
> > > +ovn-nbctl ls-add ls
> > > +ovn-nbctl lsp-add ls ls-DR
> > > +ovn-nbctl lsp-set-type ls-DR router
> > > +ovn-nbctl lsp-set-addresses ls-DR router
> > > +ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls
> > > +
> > > +# Add the lsp lp11 to ls. This will map to VIF11.
> > > +ovn-nbctl lsp-add ls lp11
> > > +ovn-nbctl lsp-set-addresses lp11 "f0:00:00:00:00:10 20.0.0.10"
> > > +ovn-nbctl lsp-set-port-security lp11 f0:00:00:00:00:10
> > > +
> > > +# Add the Northbound endpoint, lp-north1
> > > +ovn-nbctl ls-add ls-north1
> > > +ovn-nbctl lsp-add ls-north1 ln4 "" 1000
> > > +ovn-nbctl lsp-set-addresses ln4 unknown
> > > +ovn-nbctl lsp-set-type ln4 localnet
> > > +ovn-nbctl lsp-set-options ln4 network_name=phys
> > > +
> > > +ovn-nbctl lsp-add ls-north1 lp-north1
> > > +ovn-nbctl lsp-set-addresses lp-north1 "f0:f0:00:00:00:11 172.16.1.10"
> > > +ovn-nbctl lsp-set-port-security lp-north1 f0:f0:00:00:00:11
> > > +
> > > +# Add the Northbound endpoint, lp-north2
> > > +ovn-nbctl ls-add ls-north2
> > > +ovn-nbctl lsp-add ls-north2 ln5 "" 2000
> > > +ovn-nbctl lsp-set-addresses ln5 unknown
> > > +ovn-nbctl lsp-set-type ln5 localnet
> > > +ovn-nbctl lsp-set-options ln5 network_name=phys
> > > +
> > > +ovn-nbctl lsp-add ls-north2 lp-north2
> > > +ovn-nbctl lsp-set-addresses lp-north2 "f0:f0:00:00:00:22 10.0.0.10"
> > > +ovn-nbctl lsp-set-port-security lp-north2 f0:f0:00:00:00:22
> > > +
> > > +# Add the Northbound endpoint, lp-north3
> > > +ovn-nbctl ls-add ls-north3
> > > +ovn-nbctl lsp-add ls-north3 ln6 "" 3000
> > > +ovn-nbctl lsp-set-addresses ln6 unknown
> > > +ovn-nbctl lsp-set-type ln6 localnet
> > > +ovn-nbctl lsp-set-options ln6 network_name=phys
> > > +
> > > +ovn-nbctl lsp-add ls-north3 lp-north3
> > > +ovn-nbctl lsp-set-addresses lp-north3 "f0:f0:00:00:00:33 192.168.0.10"
> > > +ovn-nbctl lsp-set-port-security lp-north3 f0:f0:00:00:00:33
> > > +
> > > +# Add 5 chassis
> > > +net_add n1
> > > +for i in 1 2 3 4 5; do
> > > + sim_add hv$i
> > > + as hv$i
> > > + ovs-vsctl add-br br-phys
> > > + ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys
> > > + ovn_attach n1 br-phys 192.168.0.$i 24 $encap
> > > +done
> > > +
> > > +# Add a vif on HV1
> > > +as hv1 ovs-vsctl add-port br-int vif11 -- \
> > > + set Interface vif11 external-ids:iface-id=lp11 \
> > > + options:tx_pcap=hv1/vif11-tx.pcap \
> > > + options:rxq_pcap=hv1/vif11-rx.pcap \
> > > + ofport-request=11
> > > +OVS_WAIT_UNTIL([test x`ovn-nbctl lsp-get-up lp11` = xup])
> > > +
> > > +as hv5 ovs-vsctl add-port br-int vif-north1 -- \
> > > + set Interface vif-north1 external-ids:iface-id=lp-north1 \
> > > + options:tx_pcap=hv5/vif-north1-tx.pcap \
> > > + options:rxq_pcap=hv5/vif-north1-rx.pcap \
> > > + ofport-request=44
> > > +
> > > +as hv5 ovs-vsctl add-port br-int vif-north2 -- \
> > > + set Interface vif-north2 external-ids:iface-id=lp-north2 \
> > > + options:tx_pcap=hv5/vif-north2-tx.pcap \
> > > + options:rxq_pcap=hv5/vif-north2-rx.pcap \
> > > + ofport-request=45
> > > +
> > > +as hv5 ovs-vsctl add-port br-int vif-north3 -- \
> > > + set Interface vif-north3 external-ids:iface-id=lp-north3 \
> > > + options:tx_pcap=hv5/vif-north3-tx.pcap \
> > > + options:rxq_pcap=hv5/vif-north3-rx.pcap \
> > > + ofport-request=46
> > > +
> > > +ovn-nbctl lrp-set-gateway-chassis DR-S1 hv2
> > > +ovn-nbctl lrp-set-gateway-chassis DR-S2 hv3
> > > +ovn-nbctl lrp-set-gateway-chassis DR-S3 hv4
> > > +
> > > +ovn-nbctl --wait=sb sync
> > > +OVN_POPULATE_ARP
> > > +
> > > +vif_to_ls () {
> > > + case ${1} in dnl (
> > > + vif?[[11]]) echo ls ;; dnl (
> > > + vif-north1) echo ls-north1 ;; dnl (
> > > + vif-north2) echo ls-north2 ;; dnl (
> > > + vif-north3) echo ls-north3 ;; dnl (
> > > + *) AT_FAIL_IF([:]) ;;
> > > + esac
> > > +}
> > > +
> > > +vif_to_hv () {
> > > + case ${1} in dnl (
> > > + vif[[1]]?) echo hv1 ;; dnl (
> > > + vif-north1) echo hv5 ;; dnl (
> > > + vif-north2) echo hv5 ;; dnl (
> > > + vif-north3) echo hv5 ;; dnl (
> > > + *) AT_FAIL_IF([:]) ;;
> > > + esac
> > > +}
> > > +
> > > +vif_to_lrp () {
> > > + case ${1} in dnl (
> > > + vif?[[11]]) echo DR-ls ;; dnl (
> > > + *) AT_FAIL_IF([:]) ;;
> > > + esac
> > > +
> > > +}
> > > +
> > > +ip_to_hex() {
> > > + printf "%02x%02x%02x%02x" "${@}"
> > > +}
> > > +
> > > +# test_arp INPORT SHA SPA TPA
> > > +#
> > > +# Causes a packet to be received on INPORT. The packet is an ARP
> > > +# request with SHA, SPA, and TPA as specified.
> > > +test_arp() {
> > > + local inport=$1 sha=$2 spa=$3 tpa=$4
> > > + local
> request=ffffffffffff${sha}08060001080006040001${sha}${spa}ffffffffffff${tpa}
> > > + hv=`vif_to_hv $inport`
> > > + as $hv ovs-appctl netdev-dummy/receive $inport $request
> > > +}
> > > +
> > > +
> > > +test_ip() {
> > > + # This packet has bad checksums but logical L3 routing doesn't
> check.
> > > + local inport=${1} src_mac=${2} dst_mac=${3} src_ip=${4}
> dst_ip=${5} outport=${6}
> > > + local
> packet=${dst_mac}${src_mac}08004500001c0000000040110000${src_ip}${dst_ip}0035111100080000
> > > + shift; shift; shift; shift; shift
> > > + hv=`vif_to_hv $inport`
> > > + as $hv ovs-appctl netdev-dummy/receive $inport $packet
> > > + in_ls=`vif_to_ls $inport`
> > > + for outport; do
> > > + out_ls=`vif_to_ls $outport`
> > > + if test $in_ls = $out_ls; then
> > > + # Ports on the same logical switch receive exactly the
> same packet.
> > > + echo $packet
> > > + else
> > > + # Routing decrements TTL and updates source and dest
> MAC
> > > + # (and checksum).
> > > + # For North-South, packet will come via gateway
> chassis, i.e hv3
> > > + if test $inport = vif-north1; then
> > > + echo
> f0000000001006ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000
> >> $outport.expected
> > > + fi
> > > + if test $outport = vif-north1; then
> > > + echo
> f0f00000001102ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000
> >> $outport.expected
> > > + fi
> > > + if test $outport = vif-north2; then
> > > + echo
> f0f00000002208ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000
> >> $outport.expected
> > > + fi
> > > + if test $outport = vif-north3; then
> > > + echo
> f0f00000003304ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000
> >> $outport.expected
> > > + fi
> > > + fi >> $outport.expected
> > > + done
> > > +}
> > > +
> > > +echo "------ OVN dump ------"
> > > +ovn-nbctl show
> > > +ovn-sbctl show
> > > +ovn-sbctl list port_binding
> > > +ovn-sbctl list mac_binding
> > > +ovn-sbctl list datapath_binding
> > > +
> > > +ovn-sbctl dump-flows DR
> > > +ovn-sbctl dump-flows S1
> > > +ovn-sbctl dump-flows ls
> > > +
> > > +echo "------ hv1 dump ------"
> > > +as hv1 ovs-vsctl show
> > > +as hv1 ovs-vsctl list Open_Vswitch
> > > +as hv1 ovs-ofctl dump-flows br-int
> > > +
> > > +echo "------ hv2 dump ------"
> > > +as hv2 ovs-vsctl show
> > > +as hv2 ovs-vsctl list Open_Vswitch
> > > +as hv2 ovs-ofctl dump-flows br-int
> > > +
> > > +echo "------ hv3 dump ------"
> > > +as hv3 ovs-vsctl show
> > > +as hv3 ovs-vsctl list Open_Vswitch
> > > +as hv3 ovs-ofctl dump-flows br-int
> > > +
> > > +echo "------ hv4 dump ------"
> > > +as hv4 ovs-vsctl show
> > > +as hv4 ovs-vsctl list Open_Vswitch
> > > +as hv5 ovs-ofctl dump-flows br-int
> > > +
> > > +# N-S with lp-north1
> > > +echo "Send Dummy ARP"
> > > +sip=`ip_to_hex 172 16 1 10`
> > > +tip=`ip_to_hex 172 16 1 50`
> > > +test_arp vif-north1 f0f000000011 $sip $tip
> > > +
> > > +echo "Send traffic North to South"
> > > +sip=`ip_to_hex 172 16 1 10`
> > > +dip=`ip_to_hex 20 0 0 10`
> > > +test_ip vif-north1 f0f000000011 02ac10010001 $sip $dip vif11
> > > +# Confirm that North to south traffic works fine.
> > > +OVN_CHECK_PACKETS_REMOVE_BROADCAST([hv1/vif11-tx.pcap],
> [vif11.expected])
> > > +
> > > +echo "Send traffic South to North1"
> > > +sip=`ip_to_hex 20 0 0 10`
> > > +dip=`ip_to_hex 172 16 1 10`
> > > +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north1
> > > +# Confirm that South to North traffic works fine.
> > > +OVN_CHECK_PACKETS_REMOVE_BROADCAST([hv5/vif-north1-tx.pcap],
> [vif-north1.expected])
> > > +
> > > +# N-S with lp-north2
> > > +echo "Send Dummy ARP"
> > > +sip=`ip_to_hex 10 0 0 10`
> > > +tip=`ip_to_hex 10 0 0 50`
> > > +test_arp vif-north2 f0f000000022 $sip $tip
> > > +
> > > +echo "Send traffic South to North2"
> > > +sip=`ip_to_hex 20 0 0 10`
> > > +dip=`ip_to_hex 10 0 0 10`
> > > +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north2
> > > +# Confirm that South to North traffic works fine.
> > > +OVN_CHECK_PACKETS_REMOVE_BROADCAST([hv5/vif-north2-tx.pcap],
> [vif-north2.expected])
> > > +
> > > +# N-S with lp-north3
> > > +echo "Send Dummy ARP"
> > > +sip=`ip_to_hex 192 168 0 10`
> > > +tip=`ip_to_hex 192 168 0 50`
> > > +test_arp vif-north3 f0f000000033 $sip $tip
> > > +
> > > +echo "Send traffic South to North3"
> > > +sip=`ip_to_hex 20 0 0 10`
> > > +dip=`ip_to_hex 192 168 0 10`
> > > +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north3
> > > +# Confirm that South to North traffic works fine.
> > > +OVN_CHECK_PACKETS_REMOVE_BROADCAST([hv5/vif-north3-tx.pcap],
> [vif-north3.expected])
> > > +
> > > +AT_CLEANUP
> > > +])
> > > --
> > > 2.30.2
> > >
> > > _______________________________________________
> > > dev mailing list
> > > [email protected]
> > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > >
> _______________________________________________
> dev mailing list
> [email protected]
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
