On Fri, Mar 2, 2018 at 5:01 AM, Numan Siddique <[email protected]> wrote:
> > > On Feb 28, 2018 6:12 PM, "Guru Shetty" <[email protected]> wrote: > > This is a little more complicated for a quick review. I think it will need > a round from me to understand the implications for gateway routers and one > round from Numan (or someone from openstack) that use distributed gateway > routers. > > > > Ack. I will take a look at the patch as well. > > Thanks > Numan > > > On 14 February 2018 at 16:42, Ben Pfaff <[email protected]> wrote: > > > Hi Guru. Are you willing to take a look at this patch? > > > > Thanks, > > > > Ben. > > > > On Fri, Feb 09, 2018 at 03:34:55PM +0800, Guoshuai Li wrote: > > > The main application scenario of this patch is that the user flow wants > > to > > > different destination addresses through different external networks. > > > This scenario requires a distributed route to be associated with > > > multiple external network logical switches. > > > > > > Change l3dgw_port to l3dgw_ports in ovn_datapath,and change > > > l3redirect_port to ovn_port. Then in a distributed router, the NAT > > > logical flow table is generated based on the external IP lookup > > > distributed router port, otherwise not generated. And LB is the same. > > > > > > When the destination address of the packet is an external IP of the NAT > > rule, > > > and the ingress port is not a gateway, it is necessary to route to the > > actual > > > outgoing port. > > > > > > Signed-off-by: Guoshuai Li <[email protected]> > > Hi Guoshuai Li, I am getting the below compilation errors with "-Werror" option ********** ../ovn/northd/ovn-northd.c: In function ‘build_lrouter_flows’: ../ovn/northd/ovn-northd.c:5394:29: error: declaration of ‘i’ shadows a previous local [-Werror=shadow] for (size_t i = 0; i < od->n_l3dgw_ports; i++) { ^ ../ovn/northd/ovn-northd.c:5139:18: note: shadowed declaration is here for (int i = 0; i < od->nbr->n_nat; i++) { ^ ../ovn/northd/ovn-northd.c:5395:38: error: declaration of ‘l3dgw_port’ shadows a previous local [-Werror=shadow] struct ovn_port *l3dgw_port = od->l3dgw_ports[i]; ^~~~~~~~~~ ../ovn/northd/ovn-northd.c:5197:30: note: shadowed declaration is here struct ovn_port *l3dgw_port = find_l3dgw_port(od, ^~~~~~~~~~ ../ovn/northd/ovn-northd.c:5413:29: error: declaration of ‘i’ shadows a previous local [-Werror=shadow] for (size_t i = 0; i < od->n_l3dgw_ports; i++) { ^ ../ovn/northd/ovn-northd.c:5139:18: note: shadowed declaration is here for (int i = 0; i < od->nbr->n_nat; i++) { ^ ../ovn/northd/ovn-northd.c:5414:38: error: declaration of ‘l3dgw_port’ shadows a previous local [-Werror=shadow] struct ovn_port *l3dgw_port = od->l3dgw_ports[i]; ^~~~~~~~~~ ../ovn/northd/ovn-northd.c:5197:30: note: shadowed declaration is here struct ovn_port *l3dgw_port = find_l3dgw_port(od, ********* Thanks Numan > > > --- > > > ovn/northd/ovn-northd.8.xml | 22 +--- > > > ovn/northd/ovn-northd.c | 273 +++++++++++++++++++++++++----- > > -------------- > > > ovn/ovn-nb.xml | 12 +- > > > tests/system-ovn.at | 162 +++++++++++++++++++++++--- > > > 4 files changed, 315 insertions(+), 154 deletions(-) > > > > > > --- > > > > > > I submitted this patch long ago, I think it might be useful, > > > so resend it for more comments, thanks. > > > > > > v1 -> v2: > > > 1. rebase from master. > > > 2. add test case. > > > v2 -> v3: > > > fixed build failed. > > > > > > --- > > > > > > diff --git a/ovn/northd/ovn-northd.8.xml b/ovn/northd/ovn-northd.8.xml > > > index 6bc2dd6af..ff523614a 100644 > > > --- a/ovn/northd/ovn-northd.8.xml > > > +++ b/ovn/northd/ovn-northd.8.xml > > > @@ -1732,16 +1732,6 @@ output; > > > <ul> > > > <li> > > > <p> > > > - For distributed logical routers where one of the logical > > router > > > - ports specifies a <code>redirect-chassis</code>, a > > priority-300 > > > - logical flow with match <code>REGBIT_NAT_REDIRECT == > 1</code> > > has > > > - actions <code>ip.ttl--; next;</code>. The > > <code>outport</code> > > > - will be set later in the Gateway Redirect table. > > > - </p> > > > - </li> > > > - > > > - <li> > > > - <p> > > > IPv4 routing table. For each route to IPv4 network > > <var>N</var> with > > > netmask <var>M</var>, on router port <var>P</var> with IP > > address > > > <var>A</var> and Ethernet > > > @@ -1827,12 +1817,12 @@ next; > > > <ul> > > > <li> > > > <p> > > > - For distributed logical routers where one of the logical > > router > > > - ports specifies a <code>redirect-chassis</code>, a > > priority-200 > > > - logical flow with match <code>REGBIT_NAT_REDIRECT == > 1</code> > > has > > > - actions <code>eth.dst = <var>E</var>; next;</code>, where > > > - <var>E</var> is the ethernet address of the router's > > distributed > > > - gateway port. > > > + For distributed logical routers where router port > <var>P</var> > > > + specifies a <code>redirect-chassis</code>, a priority-200 > > > + logical flow with match <code>REGBIT_NAT_REDIRECT == > 1</code> > > > + and outport == <var>P</var> has actions > > > + <code>eth.dst = <var>E</var>; next;</code>, where > <var>E</var> > > > + is the ethernet address of the router's distributed gateway > > port. > > > </p> > > > </li> > > > > > > diff --git a/ovn/northd/ovn-northd.c b/ovn/northd/ovn-northd.c > > > index 4d95a3d9d..d38efcbed 100644 > > > --- a/ovn/northd/ovn-northd.c > > > +++ b/ovn/northd/ovn-northd.c > > > @@ -418,12 +418,10 @@ struct ovn_datapath { > > > > > > /* OVN northd only needs to know about the logical router gateway > > port for > > > * NAT on a distributed router. This "distributed gateway port" > is > > > - * populated only when there is a "redirect-chassis" specified for > > one of > > > - * the ports on the logical router. Otherwise this will be NULL. > */ > > > - struct ovn_port *l3dgw_port; > > > - /* The "derived" OVN port representing the instance of l3dgw_port > on > > > - * the "redirect-chassis". */ > > > - struct ovn_port *l3redirect_port; > > > + * populated only when there is a "redirect-chassis" specified for > > the > > > + * ports on the logical router. Otherwise this will be NULL. */ > > > + struct ovn_port **l3dgw_ports; > > > + size_t n_l3dgw_ports; > > > struct ovn_port *localnet_port; > > > }; > > > > > > @@ -472,6 +470,7 @@ ovn_datapath_destroy(struct hmap *datapaths, struct > > ovn_datapath *od) > > > free(od->ipam_info); > > > } > > > free(od->router_ports); > > > + free(od->l3dgw_ports); > > > free(od); > > > } > > > } > > > @@ -796,6 +795,10 @@ struct ovn_port { > > > bool derived; /* Indicates whether this is an additional port > > > * derived from nbsp or nbrp. */ > > > > > > + /* The "derived" OVN port representing the instance of l3dgw_port > on > > > + * the "redirect-chassis". Otherwise this will be NULL. */ > > > + struct ovn_port *l3redirect_port; > > > + > > > /* The port's peer: > > > * > > > * - A switch port S of type "router" has a router port R as a > > peer, > > > @@ -1479,7 +1482,7 @@ join_logical_ports(struct northd_context *ctx, > > > "on L3 gateway router", > > nbrp->name); > > > continue; > > > } > > > - if (od->l3dgw_port || od->l3redirect_port) { > > > + if (op->l3redirect_port) { > > > static struct vlog_rate_limit rl > > > = VLOG_RATE_LIMIT_INIT(1, 1); > > > VLOG_WARN_RL(&rl, "Bad configuration: multiple > > ports " > > > @@ -1506,8 +1509,10 @@ join_logical_ports(struct northd_context *ctx, > > > > > > /* Set l3dgw_port and l3redirect_port in od, for > > later > > > * use during flow creation. */ > > > - od->l3dgw_port = op; > > > - od->l3redirect_port = crp; > > > + op->l3redirect_port = crp; > > > + od->l3dgw_ports = xrealloc(od->l3dgw_ports, > > > + sizeof *od->l3dgw_ports * (od->n_l3dgw_ports + > > 1)); > > > + od->l3dgw_ports[od->n_l3dgw_ports++] = op; > > > } > > > } > > > } > > > @@ -1604,6 +1609,9 @@ get_router_load_balancer_ips(const struct > > ovn_datapath *od, > > > } > > > } > > > > > > +static const char * > > > +find_lrp_member_ip(const struct ovn_port *op, const char *ip_s); > > > + > > > /* Returns an array of strings, each consisting of a MAC address > > followed > > > * by one or more IP addresses, and if the port is a distributed > gateway > > > * port, followed by 'is_chassis_resident("LPORT_NAME")', where the > > > @@ -1646,7 +1654,7 @@ get_nat_addresses(const struct ovn_port *op, > > size_t *n) > > > > > > /* Determine whether this NAT rule satisfies the conditions > for > > > * distributed NAT processing. */ > > > - if (op->od->l3redirect_port && !strcmp(nat->type, > > "dnat_and_snat") > > > + if (op->l3redirect_port && !strcmp(nat->type, "dnat_and_snat") > > > && nat->logical_port && nat->external_mac) { > > > /* Distributed NAT rule. */ > > > if (eth_addr_from_string(nat->external_mac, &mac)) { > > > @@ -1660,8 +1668,10 @@ get_nat_addresses(const struct ovn_port *op, > > size_t *n) > > > } else { > > > /* Centralized NAT rule, either on gateway router or > > distributed > > > * router. */ > > > - ds_put_format(&c_addresses, " %s", nat->external_ip); > > > - central_ip_address = true; > > > + if (find_lrp_member_ip(op, nat->external_ip)) { > > > + ds_put_format(&c_addresses, " %s", nat->external_ip); > > > + central_ip_address = true; > > > + } > > > } > > > } > > > > > > @@ -1680,9 +1690,9 @@ get_nat_addresses(const struct ovn_port *op, > > size_t *n) > > > if (central_ip_address) { > > > /* Gratuitous ARP for centralized NAT rules on distributed > > gateway > > > * ports should be restricted to the "redirect-chassis". */ > > > - if (op->od->l3redirect_port) { > > > + if (op->l3redirect_port) { > > > ds_put_format(&c_addresses, " is_chassis_resident(%s)", > > > - op->od->l3redirect_port->json_key); > > > + op->l3redirect_port->json_key); > > > } > > > > > > addresses[n_nats++] = ds_steal_cstr(&c_addresses); > > > @@ -2021,8 +2031,7 @@ ovn_port_update_sbrec(struct northd_context *ctx, > > > const char *nat_addresses = smap_get(&op->nbsp->options, > > > "nat-addresses"); > > > if (nat_addresses && !strcmp(nat_addresses, "router")) { > > > - if (op->peer && op->peer->od > > > - && (chassis || op->peer->od->l3redirect_port)) { > > > + if (op->peer && (chassis || > op->peer->l3redirect_port)) > > { > > > size_t n_nats; > > > char **nats = get_nat_addresses(op->peer, > &n_nats); > > > if (n_nats) { > > > @@ -3983,14 +3992,12 @@ build_lswitch_flows(struct hmap *datapaths, > > struct hmap *ports, > > > ds_clear(&match); > > > ds_put_format(&match, "eth.dst == "ETH_ADDR_FMT, > > > ETH_ADDR_ARGS(mac)); > > > - if (op->peer->od->l3dgw_port > > > - && op->peer == op->peer->od->l3dgw_port > > > - && op->peer->od->l3redirect_port) { > > > + if (op->peer->l3redirect_port) { > > > /* The destination lookup flow for the router's > > > * distributed gateway port MAC address should > only > > be > > > * programmed on the "redirect-chassis". */ > > > ds_put_format(&match, " && > is_chassis_resident(%s)", > > > - op->peer->od->l3redirect_port- > > >json_key); > > > + op->peer->l3redirect_port->jso > n_key); > > > } > > > > > > ds_clear(&actions); > > > @@ -4000,8 +4007,7 @@ build_lswitch_flows(struct hmap *datapaths, > struct > > hmap *ports, > > > > > > /* Add ethernet addresses specified in NAT rules on > > > * distributed logical routers. */ > > > - if (op->peer->od->l3dgw_port > > > - && op->peer == op->peer->od->l3dgw_port) { > > > + if (op->peer->l3redirect_port) { > > > for (int j = 0; j < op->peer->od->nbr->n_nat; > j++) { > > > const struct nbrec_nat *nat > > > = > > op->peer->od->nbr->nat[j]; > > > @@ -4156,6 +4162,22 @@ find_lrp_member_ip(const struct ovn_port *op, > > const char *ip_s) > > > return NULL; > > > } > > > > > > +static struct ovn_port * > > > +find_l3dgw_port(struct ovn_datapath *od, const char *external_ip) > > > +{ > > > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) { > > > + struct ovn_port *l3dgw_port = od->l3dgw_ports[i]; > > > + if (find_lrp_member_ip(l3dgw_port, external_ip)) { > > > + return l3dgw_port; > > > + } > > > + } > > > + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); > > > + VLOG_WARN_RL(&rl, "can not find l3dgw port with redirect-chassis " > > > + "for nat, external ip %s in router "UUID_FMT"", > > > + external_ip, UUID_ARGS(&od->key)); > > > + return NULL; > > > +} > > > + > > > static void > > > add_route(struct hmap *lflows, const struct ovn_port *op, > > > const char *lrp_addr_s, const char *network_s, int plen, > > > @@ -4410,7 +4432,8 @@ static void > > > add_router_lb_flow(struct hmap *lflows, struct ovn_datapath *od, > > > struct ds *match, struct ds *actions, int priority, > > > const char *lb_force_snat_ip, char *backend_ips, > > > - bool is_udp, int addr_family) > > > + bool is_udp, int addr_family, > > > + const struct ovn_port *l3dgw_port) > > > { > > > /* A match and actions for new connections. */ > > > char *new_match = xasprintf("ct.new && %s", ds_cstr(match)); > > > @@ -4438,7 +4461,7 @@ add_router_lb_flow(struct hmap *lflows, struct > > ovn_datapath *od, > > > free(new_match); > > > free(est_match); > > > > > > - if (!od->l3dgw_port || !od->l3redirect_port || !backend_ips > > > + if (!l3dgw_port || !l3dgw_port->l3redirect_port || !backend_ips > > > || addr_family != AF_INET) { > > > return; > > > } > > > @@ -4485,8 +4508,8 @@ add_router_lb_flow(struct hmap *lflows, struct > > ovn_datapath *od, > > > ds_chomp(&undnat_match, '|'); > > > ds_chomp(&undnat_match, ' '); > > > ds_put_format(&undnat_match, ") && outport == %s && " > > > - "is_chassis_resident(%s)", od->l3dgw_port->json_key, > > > - od->l3redirect_port->json_key); > > > + "is_chassis_resident(%s)", l3dgw_port->json_key, > > > + l3dgw_port->l3redirect_port->json_key); > > > if (lb_force_snat_ip) { > > > ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 120, > > > ds_cstr(&undnat_match), > > > @@ -4610,12 +4633,11 @@ build_lrouter_flows(struct hmap *datapaths, > > struct hmap *ports, > > > ds_clear(&match); > > > ds_put_format(&match, "eth.dst == %s && inport == %s", > > > op->lrp_networks.ea_s, op->json_key); > > > - if (op->od->l3dgw_port && op == op->od->l3dgw_port > > > - && op->od->l3redirect_port) { > > > + if (op->l3redirect_port) { > > > /* Traffic with eth.dst = l3dgw_port->lrp_networks.ea_s > > > * should only be received on the "redirect-chassis". */ > > > ds_put_format(&match, " && is_chassis_resident(%s)", > > > - op->od->l3redirect_port->json_key); > > > + op->l3redirect_port->json_key); > > > } > > > ovn_lflow_add(lflows, op->od, S_ROUTER_IN_ADMISSION, 50, > > > ds_cstr(&match), "next;"); > > > @@ -4724,15 +4746,14 @@ build_lrouter_flows(struct hmap *datapaths, > > struct hmap *ports, > > > ds_put_format(&match, > > > "inport == %s && arp.tpa == %s && arp.op == > > 1", > > > op->json_key, op->lrp_networks.ipv4_addrs[i] > > .addr_s); > > > - if (op->od->l3dgw_port && op == op->od->l3dgw_port > > > - && op->od->l3redirect_port) { > > > + if (op->l3redirect_port) { > > > /* Traffic with eth.src = > l3dgw_port->lrp_networks.ea_s > > > * should only be sent from the "redirect-chassis", so > > that > > > * upstream MAC learning points to the > > "redirect-chassis". > > > * Also need to avoid generation of multiple ARP > > responses > > > * from different chassis. */ > > > ds_put_format(&match, " && is_chassis_resident(%s)", > > > - op->od->l3redirect_port->json_key); > > > + op->l3redirect_port->json_key); > > > } > > > > > > ds_clear(&actions); > > > @@ -4865,7 +4886,7 @@ build_lrouter_flows(struct hmap *datapaths, > struct > > hmap *ports, > > > "arp.op = 2; /* ARP reply */ " > > > "arp.tha = arp.sha; "); > > > > > > - if (op->od->l3dgw_port && op == op->od->l3dgw_port) { > > > + if (op->l3redirect_port) { > > > struct eth_addr mac; > > > if (nat->external_mac && > > > eth_addr_from_string(nat->external_mac, &mac) > > > @@ -4894,10 +4915,8 @@ build_lrouter_flows(struct hmap *datapaths, > > struct hmap *ports, > > > * upstream MAC learning points to the > > "redirect-chassis". > > > * Also need to avoid generation of multiple ARP > > responses > > > * from different chassis. */ > > > - if (op->od->l3redirect_port) { > > > - ds_put_format(&match, " && > > is_chassis_resident(%s)", > > > - op->od->l3redirect_port->json_ > > key); > > > - } > > > + ds_put_format(&match, " && > is_chassis_resident(%s)", > > > + op->l3redirect_port->json_key); > > > } > > > } else { > > > ds_put_format(&actions, > > > @@ -5007,15 +5026,14 @@ build_lrouter_flows(struct hmap *datapaths, > > struct hmap *ports, > > > op->lrp_networks.ipv6_addrs[i].addr_s, > > > op->lrp_networks.ipv6_addrs[i].sn_addr_s, > > > op->lrp_networks.ipv6_addrs[i].addr_s); > > > - if (op->od->l3dgw_port && op == op->od->l3dgw_port > > > - && op->od->l3redirect_port) { > > > + if (op->l3redirect_port) { > > > /* Traffic with eth.src = > l3dgw_port->lrp_networks.ea_s > > > * should only be sent from the "redirect-chassis", so > > that > > > * upstream MAC learning points to the > > "redirect-chassis". > > > * Also need to avoid generation of multiple ND > replies > > > * from different chassis. */ > > > ds_put_format(&match, " && is_chassis_resident(%s)", > > > - op->od->l3redirect_port->json_key); > > > + op->l3redirect_port->json_key); > > > } > > > > > > ds_clear(&actions); > > > @@ -5056,7 +5074,7 @@ build_lrouter_flows(struct hmap *datapaths, > struct > > hmap *ports, > > > /* NAT rules are only valid on Gateway routers and routers > with > > > * l3dgw_port (router has a port with "redirect-chassis" > > > * specified). */ > > > - if (!smap_get(&od->nbr->options, "chassis") && > !od->l3dgw_port) > > { > > > + if (!smap_get(&od->nbr->options, "chassis") && > > !od->n_l3dgw_ports) { > > > continue; > > > } > > > > > > @@ -5066,6 +5084,8 @@ build_lrouter_flows(struct hmap *datapaths, > struct > > hmap *ports, > > > const char *lb_force_snat_ip = get_force_snat_ip(od, "lb", > > > &snat_ip); > > > > > > + struct smap nat_external_ip = SMAP_INITIALIZER(&nat_ > > external_ip); > > > + > > > for (int i = 0; i < od->nbr->n_nat; i++) { > > > const struct nbrec_nat *nat; > > > > > > @@ -5110,7 +5130,7 @@ build_lrouter_flows(struct hmap *datapaths, > struct > > hmap *ports, > > > * satisfies the conditions for distributed NAT > processing. > > */ > > > bool distributed = false; > > > struct eth_addr mac; > > > - if (od->l3dgw_port && !strcmp(nat->type, "dnat_and_snat") > && > > > + if (od->n_l3dgw_ports && !strcmp(nat->type, > > "dnat_and_snat") && > > > nat->logical_port && nat->external_mac) { > > > if (eth_addr_from_string(nat->external_mac, &mac)) { > > > distributed = true; > > > @@ -5123,6 +5143,13 @@ build_lrouter_flows(struct hmap *datapaths, > > struct hmap *ports, > > > } > > > } > > > > > > + /* find l3dgw port by external ip */ > > > + struct ovn_port *l3dgw_port = find_l3dgw_port(od, > > > + > > nat->external_ip); > > > + > > > + bool first_add = smap_add_once(&nat_external_ip, > > nat->external_ip, > > > + nat->external_ip); > > > + > > > /* Ingress UNSNAT table: It is for already established > > connections' > > > * reverse traffic. i.e., SNAT has already been done in > > egress > > > * pipeline and now the packet has entered the ingress > > pipeline as > > > @@ -5132,16 +5159,16 @@ build_lrouter_flows(struct hmap *datapaths, > > struct hmap *ports, > > > * because when the packet was DNATed in ingress pipeline, > > it did > > > * not know about the possibility of eventual additional > > SNAT in > > > * egress pipeline. */ > > > - if (!strcmp(nat->type, "snat") > > > - || !strcmp(nat->type, "dnat_and_snat")) { > > > - if (!od->l3dgw_port) { > > > + if ((!strcmp(nat->type, "snat") > > > + || !strcmp(nat->type, "dnat_and_snat")) && first_add) > { > > > + if (!od->n_l3dgw_ports) { > > > /* Gateway router. */ > > > ds_clear(&match); > > > ds_put_format(&match, "ip && ip4.dst == %s", > > > nat->external_ip); > > > ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 90, > > > ds_cstr(&match), "ct_snat; next;"); > > > - } else { > > > + } else if (l3dgw_port) { > > > /* Distributed router. */ > > > > > > /* Traffic received on l3dgw_port is subject to > > NAT. */ > > > @@ -5149,12 +5176,12 @@ build_lrouter_flows(struct hmap *datapaths, > > struct hmap *ports, > > > ds_put_format(&match, "ip && ip4.dst == %s" > > > " && inport == %s", > > > nat->external_ip, > > > - od->l3dgw_port->json_key); > > > - if (!distributed && od->l3redirect_port) { > > > + l3dgw_port->json_key); > > > + if (!distributed && l3dgw_port->l3redirect_port) { > > > /* Flows for NAT rules that are centralized > are > > only > > > * programmed on the "redirect-chassis". */ > > > ds_put_format(&match, " && > > is_chassis_resident(%s)", > > > - od->l3redirect_port->json_key); > > > + l3dgw_port->l3redirect_port-> > > json_key); > > > } > > > ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 100, > > > ds_cstr(&match), "ct_snat;"); > > > @@ -5176,7 +5203,7 @@ build_lrouter_flows(struct hmap *datapaths, > struct > > hmap *ports, > > > * to a logical IP address. */ > > > if (!strcmp(nat->type, "dnat") > > > || !strcmp(nat->type, "dnat_and_snat")) { > > > - if (!od->l3dgw_port) { > > > + if (!od->n_l3dgw_ports) { > > > /* Gateway router. */ > > > /* Packet when it goes from the initiator to > > destination. > > > * We need to set flags.loopback because the > router > > can > > > @@ -5196,7 +5223,7 @@ build_lrouter_flows(struct hmap *datapaths, > struct > > hmap *ports, > > > nat->logical_ip); > > > ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 100, > > > ds_cstr(&match), ds_cstr(&actions)); > > > - } else { > > > + } else if (l3dgw_port) { > > > /* Distributed router. */ > > > > > > /* Traffic received on l3dgw_port is subject to > > NAT. */ > > > @@ -5204,12 +5231,12 @@ build_lrouter_flows(struct hmap *datapaths, > > struct hmap *ports, > > > ds_put_format(&match, "ip && ip4.dst == %s" > > > " && inport == %s", > > > nat->external_ip, > > > - od->l3dgw_port->json_key); > > > - if (!distributed && od->l3redirect_port) { > > > + l3dgw_port->json_key); > > > + if (!distributed && l3dgw_port->l3redirect_port) { > > > /* Flows for NAT rules that are centralized > are > > only > > > * programmed on the "redirect-chassis". */ > > > ds_put_format(&match, " && > > is_chassis_resident(%s)", > > > - od->l3redirect_port->json_key); > > > + l3dgw_port->l3redirect_port-> > > json_key); > > > } > > > ds_clear(&actions); > > > ds_put_format(&actions, "ct_dnat(%s);", > > > @@ -5237,18 +5264,18 @@ build_lrouter_flows(struct hmap *datapaths, > > struct hmap *ports, > > > * Note that this only applies for NAT on a distributed > > router. > > > * Undo DNAT on a gateway router is done in the ingress > DNAT > > > * pipeline stage. */ > > > - if (od->l3dgw_port && (!strcmp(nat->type, "dnat") > > > + if (l3dgw_port && (!strcmp(nat->type, "dnat") > > > || !strcmp(nat->type, "dnat_and_snat"))) { > > > ds_clear(&match); > > > ds_put_format(&match, "ip && ip4.src == %s" > > > " && outport == %s", > > > nat->logical_ip, > > > - od->l3dgw_port->json_key); > > > - if (!distributed && od->l3redirect_port) { > > > + l3dgw_port->json_key); > > > + if (!distributed && l3dgw_port->l3redirect_port) { > > > /* Flows for NAT rules that are centralized are > only > > > * programmed on the "redirect-chassis". */ > > > ds_put_format(&match, " && > is_chassis_resident(%s)", > > > - od->l3redirect_port->json_key); > > > + l3dgw_port->l3redirect_port-> > > json_key); > > > } > > > ds_clear(&actions); > > > if (distributed) { > > > @@ -5265,7 +5292,7 @@ build_lrouter_flows(struct hmap *datapaths, > struct > > hmap *ports, > > > * address. */ > > > if (!strcmp(nat->type, "snat") > > > || !strcmp(nat->type, "dnat_and_snat")) { > > > - if (!od->l3dgw_port) { > > > + if (!od->n_l3dgw_ports) { > > > /* Gateway router. */ > > > ds_clear(&match); > > > ds_put_format(&match, "ip && ip4.src == %s", > > > @@ -5279,18 +5306,18 @@ build_lrouter_flows(struct hmap *datapaths, > > struct hmap *ports, > > > ovn_lflow_add(lflows, od, S_ROUTER_OUT_SNAT, > > > count_1bits(ntohl(mask)) + 1, > > > ds_cstr(&match), ds_cstr(&actions)); > > > - } else { > > > + } else if (l3dgw_port) { > > > /* Distributed router. */ > > > ds_clear(&match); > > > ds_put_format(&match, "ip && ip4.src == %s" > > > " && outport == %s", > > > nat->logical_ip, > > > - od->l3dgw_port->json_key); > > > - if (!distributed && od->l3redirect_port) { > > > + l3dgw_port->json_key); > > > + if (!distributed && l3dgw_port->l3redirect_port) { > > > /* Flows for NAT rules that are centralized > are > > only > > > * programmed on the "redirect-chassis". */ > > > ds_put_format(&match, " && > > is_chassis_resident(%s)", > > > - od->l3redirect_port->json_key); > > > + l3dgw_port->l3redirect_port-> > > json_key); > > > } > > > ds_clear(&actions); > > > if (distributed) { > > > @@ -5314,15 +5341,18 @@ build_lrouter_flows(struct hmap *datapaths, > > struct hmap *ports, > > > * on the l3dgw_port instance where nat->logical_port is > > > * resident. */ > > > if (distributed) { > > > - ds_clear(&match); > > > - ds_put_format(&match, > > > - "eth.dst == "ETH_ADDR_FMT" && inport == > > %s" > > > - " && is_chassis_resident(\"%s\")", > > > - ETH_ADDR_ARGS(mac), > > > - od->l3dgw_port->json_key, > > > - nat->logical_port); > > > - ovn_lflow_add(lflows, od, S_ROUTER_IN_ADMISSION, 50, > > > - ds_cstr(&match), "next;"); > > > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) { > > > + struct ovn_port *l3dgw_port = od->l3dgw_ports[i]; > > > + ds_clear(&match); > > > + ds_put_format(&match, > > > + "eth.dst == "ETH_ADDR_FMT" && inport > > == %s" > > > + " && is_chassis_resident(\"%s\")", > > > + ETH_ADDR_ARGS(mac), > > > + l3dgw_port->json_key, > > > + nat->logical_port); > > > + ovn_lflow_add(lflows, od, S_ROUTER_IN_ADMISSION, > 50, > > > + ds_cstr(&match), "next;"); > > > + } > > > } > > > > > > /* Ingress Gateway Redirect Table: For NAT on a > distributed > > > @@ -5330,12 +5360,15 @@ build_lrouter_flows(struct hmap *datapaths, > > struct hmap *ports, > > > * flows indicate the presence of an applicable NAT rule > > that > > > * can be applied in a distributed manner. */ > > > if (distributed) { > > > - ds_clear(&match); > > > - ds_put_format(&match, "ip4.src == %s && outport == > %s", > > > - nat->logical_ip, > > > - od->l3dgw_port->json_key); > > > - ovn_lflow_add(lflows, od, S_ROUTER_IN_GW_REDIRECT, > 100, > > > - ds_cstr(&match), "next;"); > > > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) { > > > + struct ovn_port *l3dgw_port = od->l3dgw_ports[i]; > > > + ds_clear(&match); > > > + ds_put_format(&match, "ip4.src == %s && outport == > > %s", > > > + nat->logical_ip, > > > + l3dgw_port->json_key); > > > + ovn_lflow_add(lflows, od, S_ROUTER_IN_GW_REDIRECT, > > 100, > > > + ds_cstr(&match), "next;"); > > > + } > > > } > > > > > > /* Egress Loopback table: For NAT on a distributed router. > > > @@ -5343,12 +5376,12 @@ build_lrouter_flows(struct hmap *datapaths, > > struct hmap *ports, > > > * gateway port have ip.dst matching a NAT external IP, > then > > > * loop a clone of the packet back to the beginning of the > > > * ingress pipeline with inport = outport. */ > > > - if (od->l3dgw_port) { > > > + if (l3dgw_port && first_add) { > > > /* Distributed router. */ > > > ds_clear(&match); > > > ds_put_format(&match, "ip4.dst == %s && outport == > %s", > > > nat->external_ip, > > > - od->l3dgw_port->json_key); > > > + l3dgw_port->json_key); > > > ds_clear(&actions); > > > ds_put_format(&actions, > > > "clone { ct_clear; " > > > @@ -5365,7 +5398,7 @@ build_lrouter_flows(struct hmap *datapaths, > struct > > hmap *ports, > > > } > > > > > > /* Handle force SNAT options set in the gateway router. */ > > > - if (dnat_force_snat_ip && !od->l3dgw_port) { > > > + if (dnat_force_snat_ip && !od->n_l3dgw_ports) { > > > /* If a packet with destination IP address as that of the > > > * gateway router (as set in options:dnat_force_snat_ip) > is > > seen, > > > * UNSNAT it. */ > > > @@ -5384,7 +5417,7 @@ build_lrouter_flows(struct hmap *datapaths, > struct > > hmap *ports, > > > ovn_lflow_add(lflows, od, S_ROUTER_OUT_SNAT, 100, > > > ds_cstr(&match), ds_cstr(&actions)); > > > } > > > - if (lb_force_snat_ip && !od->l3dgw_port) { > > > + if (lb_force_snat_ip && !od->n_l3dgw_ports) { > > > /* If a packet with destination IP address as that of the > > > * gateway router (as set in options:lb_force_snat_ip) is > > seen, > > > * UNSNAT it. */ > > > @@ -5403,7 +5436,7 @@ build_lrouter_flows(struct hmap *datapaths, > struct > > hmap *ports, > > > ds_cstr(&match), ds_cstr(&actions)); > > > } > > > > > > - if (!od->l3dgw_port) { > > > + if (!od->n_l3dgw_ports) { > > > /* For gateway router, re-circulate every packet through > > > * the DNAT zone. This helps with two things. > > > * > > > @@ -5422,40 +5455,38 @@ build_lrouter_flows(struct hmap *datapaths, > > struct hmap *ports, > > > "ip", "flags.loopback = 1; ct_dnat;"); > > > } else { > > > /* For NAT on a distributed router, add flows to Ingress > > > - * IP Routing table, Ingress ARP Resolution table, and > > > - * Ingress Gateway Redirect Table that are not specific > to a > > > - * NAT rule. */ > > > - > > > - /* The highest priority IN_IP_ROUTING rule matches packets > > > - * with REGBIT_NAT_REDIRECT (set in DNAT or UNSNAT > stages), > > > - * with action "ip.ttl--; next;". The IN_GW_REDIRECT > table > > > - * will take care of setting the outport. */ > > > - ovn_lflow_add(lflows, od, S_ROUTER_IN_IP_ROUTING, 300, > > > - REGBIT_NAT_REDIRECT" == 1", "ip.ttl--; > > next;"); > > > - > > > - /* The highest priority IN_ARP_RESOLVE rule matches > packets > > > - * with REGBIT_NAT_REDIRECT (set in DNAT or UNSNAT > stages), > > > - * then sets eth.dst to the distributed gateway port's > > > - * ethernet address. */ > > > - ds_clear(&actions); > > > - ds_put_format(&actions, "eth.dst = %s; next;", > > > - od->l3dgw_port->lrp_networks.ea_s); > > > - ovn_lflow_add(lflows, od, S_ROUTER_IN_ARP_RESOLVE, 200, > > > - REGBIT_NAT_REDIRECT" == 1", > > ds_cstr(&actions)); > > > - > > > - /* The highest priority IN_GW_REDIRECT rule redirects > > packets > > > - * with REGBIT_NAT_REDIRECT (set in DNAT or UNSNAT stages) > > to > > > - * the central instance of the l3dgw_port for NAT > > processing. */ > > > - ds_clear(&actions); > > > - ds_put_format(&actions, "outport = %s; next;", > > > - od->l3redirect_port->json_key); > > > - ovn_lflow_add(lflows, od, S_ROUTER_IN_GW_REDIRECT, 200, > > > - REGBIT_NAT_REDIRECT" == 1", > > ds_cstr(&actions)); > > > + * ARP Resolution table, and Ingress Gateway Redirect > Table > > > + * that are not specific to a NAT rule. */ > > > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) { > > > + struct ovn_port *l3dgw_port = od->l3dgw_ports[i]; > > > + /* The highest priority IN_ARP_RESOLVE rule matches > > packets > > > + * with REGBIT_NAT_REDIRECT (set in DNAT or UNSNAT > > stages), > > > + * then sets eth.dst to the distributed gateway port's > > > + * ethernet address. */ > > > + ds_clear(&match); > > > + ds_put_format(&match, REGBIT_NAT_REDIRECT" == 1 && " > > > + "outport == %s", l3dgw_port->json_key); > > > + ds_clear(&actions); > > > + ds_put_format(&actions, "eth.dst = %s; next;", > > > + l3dgw_port->lrp_networks.ea_s); > > > + ovn_lflow_add(lflows, od, S_ROUTER_IN_ARP_RESOLVE, > 200, > > > + ds_cstr(&match), ds_cstr(&actions)); > > > + > > > + /* The highest priority IN_GW_REDIRECT rule redirects > > packets > > > + * with REGBIT_NAT_REDIRECT (set in DNAT or UNSNAT > > stages) to > > > + * the central instance of the l3dgw_port for NAT > > processing. > > > + */ > > > + ds_clear(&actions); > > > + ds_put_format(&actions, "outport = %s; next;", > > > + l3dgw_port->l3redirect_port->json_key); > > > + ovn_lflow_add(lflows, od, S_ROUTER_IN_GW_REDIRECT, > 200, > > > + ds_cstr(&match), ds_cstr(&actions)); > > > + } > > > } > > > > > > /* Load balancing and packet defrag are only valid on > > > * Gateway routers or router with gateway port. */ > > > - if (!smap_get(&od->nbr->options, "chassis") && > !od->l3dgw_port) > > { > > > + if (!smap_get(&od->nbr->options, "chassis") && > > !od->n_l3dgw_ports) { > > > continue; > > > } > > > > > > @@ -5517,6 +5548,9 @@ build_lrouter_flows(struct hmap *datapaths, > struct > > hmap *ports, > > > ds_put_format(&match, "ip && ip6.dst == %s", > > > ip_address); > > > } > > > + /* find l3dgw port by lb vip */ > > > + const struct ovn_port *l3dgw_port > > > + = find_l3dgw_port(od, ip_address); > > > free(ip_address); > > > > > > int prio = 110; > > > @@ -5533,13 +5567,13 @@ build_lrouter_flows(struct hmap *datapaths, > > struct hmap *ports, > > > prio = 120; > > > } > > > > > > - if (od->l3redirect_port) { > > > + if (l3dgw_port) { > > > ds_put_format(&match, " && > is_chassis_resident(%s)", > > > - od->l3redirect_port->json_key); > > > + l3dgw_port->l3redirect_port-> > > json_key); > > > } > > > add_router_lb_flow(lflows, od, &match, &actions, prio, > > > lb_force_snat_ip, node->value, > > is_udp, > > > - addr_family); > > > + addr_family, l3dgw_port); > > > } > > > } > > > sset_destroy(&all_ips); > > > @@ -5898,17 +5932,18 @@ build_lrouter_flows(struct hmap *datapaths, > > struct hmap *ports, > > > if (!od->nbr) { > > > continue; > > > } > > > - if (od->l3dgw_port && od->l3redirect_port) { > > > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) { > > > + struct ovn_port *l3dgw_port = od->l3dgw_ports[i]; > > > /* For traffic with outport == l3dgw_port, if the > > > * packet did not match any higher priority redirect > > > * rule, then the traffic is redirected to the central > > > * instance of the l3dgw_port. */ > > > ds_clear(&match); > > > ds_put_format(&match, "outport == %s", > > > - od->l3dgw_port->json_key); > > > + l3dgw_port->json_key); > > > ds_clear(&actions); > > > ds_put_format(&actions, "outport = %s; next;", > > > - od->l3redirect_port->json_key); > > > + l3dgw_port->l3redirect_port->json_key); > > > ovn_lflow_add(lflows, od, S_ROUTER_IN_GW_REDIRECT, 50, > > > ds_cstr(&match), ds_cstr(&actions)); > > > > > > diff --git a/ovn/ovn-nb.xml b/ovn/ovn-nb.xml > > > index b7a5b6bf2..c0c6a45f0 100644 > > > --- a/ovn/ovn-nb.xml > > > +++ b/ovn/ovn-nb.xml > > > @@ -1430,8 +1430,7 @@ > > > <p> > > > If set, this indicates that this logical router port > > represents > > > a distributed gateway port that connects this router to a > > logical > > > - switch with a localnet port. There may be at most one such > > > - logical router port on each logical router. > > > + switch with a localnet port. > > > </p> > > > > > > <p> > > > @@ -1617,7 +1616,14 @@ > > > </column> > > > > > > <column name="external_ip"> > > > - An IPv4 address. > > > + <p> > > > + An IPv4 address. > > > + </p> > > > + > > > + <p> > > > + On distributed router, This address must be within the subnet > of > > > + the gateway port instance on the > <code>redirect-chassis</code>. > > > + </p> > > > </column> > > > > > > <column name="external_mac"> > > > diff --git a/tests/system-ovn.at b/tests/system-ovn.at > > > index 638c0b661..6f30c1ec6 100644 > > > --- a/tests/system-ovn.at > > > +++ b/tests/system-ovn.at > > > @@ -1097,23 +1097,27 @@ start_daemon ovn-controller > > > > > > # Logical network: > > > # One LR R1 with switches foo (192.168.1.0/24), bar (192.168.2.0/24), > > > -# and alice (172.16.1.0/24) connected to it. The port between R1 and > > > -# alice is the router gateway port where the R1 LB rules are applied. > > > +# alice (172.16.1.0/24) and outsite (172.16.2.0/24) connected to it. > > > +# The port between R1 and alice/outsite is the router gateway port > > > +# where the R1 LB rules are applied. > > > # > > > -# foo -- R1 -- bar > > > -# | > > > -# alice ---- > > > +# foo ---+--- bar > > > +# R1 > > > +# alice --+-- outsite > > > > > > ovn-nbctl lr-add R1 > > > > > > ovn-nbctl ls-add foo > > > ovn-nbctl ls-add bar > > > ovn-nbctl ls-add alice > > > +ovn-nbctl ls-add outsite > > > > > > ovn-nbctl lrp-add R1 foo 00:00:01:01:02:03 192.168.1.1/24 > > > ovn-nbctl lrp-add R1 bar 00:00:01:01:02:04 192.168.2.1/24 > > > ovn-nbctl lrp-add R1 alice 00:00:02:01:02:03 172.16.1.1/24 \ > > > -- set Logical_Router_Port alice options:redirect-chassis=hv1 > > > +ovn-nbctl lrp-add R1 outsite 00:00:03:01:02:01 172.16.2.1/24 \ > > > + -- set Logical_Router_Port outsite options:redirect-chassis=hv1 > > > > > > # Connect foo to R1 > > > ovn-nbctl lsp-add foo rp-foo -- set Logical_Switch_Port rp-foo \ > > > @@ -1130,6 +1134,11 @@ ovn-nbctl lsp-add alice rp-alice -- set > > Logical_Switch_Port rp-alice \ > > > type=router options:router-port=alice \ > > > -- lsp-set-addresses rp-alice router > > > > > > +# Connect outsite to R1 > > > +ovn-nbctl lsp-add outsite rp-outsite -- set Logical_Switch_Port > > rp-outsite \ > > > + type=router options:router-port=outsite \ > > > + -- lsp-set-addresses rp-outsite router > > > + > > > # Logical port 'foo1' in switch 'foo'. > > > ADD_NAMESPACES(foo1) > > > ADD_VETH(foo1, foo1, br-int, "192.168.1.2/24", "f0:00:00:01:02:03", \ > > > @@ -1158,12 +1167,21 @@ ADD_VETH(alice1, alice1, br-int, " > 172.16.1.2/24", > > "f0:00:00:01:02:05", \ > > > ovn-nbctl lsp-add alice alice1 \ > > > -- lsp-set-addresses alice1 "f0:00:00:01:02:05 172.16.1.2" > > > > > > +# Logical port 'outsite1' in switch 'outsite'. > > > +ADD_NAMESPACES(outsite1) > > > +ADD_VETH(outsite1, outsite1, br-int, "172.16.2.2/24", > > "f0:00:00:01:02:07", \ > > > + "172.16.2.1") > > > +ovn-nbctl lsp-add outsite outsite1 \ > > > +-- lsp-set-addresses outsite1 "f0:00:00:01:02:07 172.16.2.2" > > > + > > > # Config OVN load-balancer with a VIP. > > > uuid=`ovn-nbctl create load_balancer vips:172.16.1.10="192.168.1.2, > > 192.168.2.2"` > > > -ovn-nbctl set logical_router R1 load_balancer=$uuid > > > +uuid2=`ovn-nbctl create load_balancer vips:172.16.2.10="192.168.1.2, > > 192.168.2.2"` > > > +ovn-nbctl set logical_router R1 load_balancer=$uuid,$uuid2 > > > > > > # Config OVN load-balancer with another VIP (this time with ports). > > > ovn-nbctl set load_balancer $uuid vips:'"172.16.1.11:8000"'='"19 > > 2.168.1.2:80,192.168.2.2:80"' > > > +ovn-nbctl set load_balancer $uuid2 vips:'"172.16.2.11:8000"'='"19 > > 2.168.1.2:80,192.168.2.2:80"' > > > > > > # Wait for ovn-controller to catch up. > > > ovn-nbctl --wait=hv sync > > > @@ -1179,6 +1197,10 @@ for i in `seq 1 20`; do > > > echo Request $i > > > NS_CHECK_EXEC([alice1], [wget 172.16.1.10 -t 5 -T 1 > > --retry-connrefused -v -o wget$i.log]) > > > done > > > +for i in `seq 1 20`; do > > > + echo Request $i > > > + NS_CHECK_EXEC([outsite1], [wget 172.16.2.10 -t 5 -T 1 > > --retry-connrefused -v -o wget$i.log]) > > > +done > > > > > > dnl Each server should have at least one connection. > > > AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.10) | > > > @@ -1186,12 +1208,21 @@ sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], > [0], > > [dnl > > > tcp,orig=(src=172.16.1.2,dst=172.16.1.10,sport=<cleared>, > > dport=<cleared>),reply=(src=192.168.1.2,dst=172.16.1.2, > > sport=<cleared>,dport=<cleared>),zone=<cleared>, > > protoinfo=(state=<cleared>) > > > tcp,orig=(src=172.16.1.2,dst=172.16.1.10,sport=<cleared>, > > dport=<cleared>),reply=(src=192.168.2.2,dst=172.16.1.2, > > sport=<cleared>,dport=<cleared>),zone=<cleared>, > > protoinfo=(state=<cleared>) > > > ]) > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.2.10) | > > > +sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl > > > +tcp,orig=(src=172.16.2.2,dst=172.16.2.10,sport=<cleared>, > > dport=<cleared>),reply=(src=192.168.1.2,dst=172.16.2.2, > > sport=<cleared>,dport=<cleared>),zone=<cleared>, > > protoinfo=(state=<cleared>) > > > +tcp,orig=(src=172.16.2.2,dst=172.16.2.10,sport=<cleared>, > > dport=<cleared>),reply=(src=192.168.2.2,dst=172.16.2.2, > > sport=<cleared>,dport=<cleared>),zone=<cleared>, > > protoinfo=(state=<cleared>) > > > +]) > > > > > > dnl Test load-balancing that includes L4 ports in NAT. > > > for i in `seq 1 20`; do > > > echo Request $i > > > NS_CHECK_EXEC([alice1], [wget 172.16.1.11:8000 -t 5 -T 1 > > --retry-connrefused -v -o wget$i.log]) > > > done > > > +for i in `seq 1 20`; do > > > + echo Request $i > > > + NS_CHECK_EXEC([outsite1], [wget 172.16.2.11:8000 -t 5 -T 1 > > --retry-connrefused -v -o wget$i.log]) > > > +done > > > > > > dnl Each server should have at least one connection. > > > AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.11) | > > > @@ -1199,6 +1230,11 @@ sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], > > [dnl > > > tcp,orig=(src=172.16.1.2,dst=172.16.1.11,sport=<cleared>, > > dport=<cleared>),reply=(src=192.168.1.2,dst=172.16.1.2, > > sport=<cleared>,dport=<cleared>),zone=<cleared>, > > protoinfo=(state=<cleared>) > > > tcp,orig=(src=172.16.1.2,dst=172.16.1.11,sport=<cleared>, > > dport=<cleared>),reply=(src=192.168.2.2,dst=172.16.1.2, > > sport=<cleared>,dport=<cleared>),zone=<cleared>, > > protoinfo=(state=<cleared>) > > > ]) > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.2.11) | > > > +sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl > > > +tcp,orig=(src=172.16.2.2,dst=172.16.2.11,sport=<cleared>, > > dport=<cleared>),reply=(src=192.168.1.2,dst=172.16.2.2, > > sport=<cleared>,dport=<cleared>),zone=<cleared>, > > protoinfo=(state=<cleared>) > > > +tcp,orig=(src=172.16.2.2,dst=172.16.2.11,sport=<cleared>, > > dport=<cleared>),reply=(src=192.168.2.2,dst=172.16.2.2, > > sport=<cleared>,dport=<cleared>),zone=<cleared>, > > protoinfo=(state=<cleared>) > > > +]) > > > > > > OVS_APP_EXIT_AND_WAIT([ovn-controller]) > > > > > > @@ -1238,23 +1274,27 @@ start_daemon ovn-controller > > > > > > # Logical network: > > > # One LR R1 with switches foo (192.168.1.0/24), bar (192.168.2.0/24), > > > -# and alice (172.16.1.0/24) connected to it. The port between R1 and > > > -# alice is the router gateway port where the R1 NAT rules are applied. > > > +# alice (172.16.1.0/24) and outsite (172.16.2.0/24) connected to it. > > > +# The port between R1 and alice/outsite is the router gateway port > > > +# where the R1 NAT rules are applied. > > > # > > > -# foo -- R1 -- alice > > > -# | > > > -# bar ---- > > > +# foo ---+--- bar > > > +# R1 > > > +# alice --+-- outsite > > > > > > ovn-nbctl lr-add R1 > > > > > > ovn-nbctl ls-add foo > > > ovn-nbctl ls-add bar > > > ovn-nbctl ls-add alice > > > +ovn-nbctl ls-add outsite > > > > > > ovn-nbctl lrp-add R1 foo 00:00:01:01:02:03 192.168.1.1/24 > > > ovn-nbctl lrp-add R1 bar 00:00:01:01:02:04 192.168.2.1/24 > > > ovn-nbctl lrp-add R1 alice 00:00:02:01:02:03 172.16.1.1/24 \ > > > -- set Logical_Router_Port alice options:redirect-chassis=hv1 > > > +ovn-nbctl lrp-add R1 outsite 00:00:03:01:02:03 172.16.2.1/24 \ > > > + -- set Logical_Router_Port outsite options:redirect-chassis=hv1 > > > > > > # Connect foo to R1 > > > ovn-nbctl lsp-add foo rp-foo -- set Logical_Switch_Port rp-foo \ > > > @@ -1271,6 +1311,11 @@ ovn-nbctl lsp-add alice rp-alice -- set > > Logical_Switch_Port rp-alice \ > > > type=router options:router-port=alice \ > > > -- lsp-set-addresses rp-alice router > > > > > > +# Connect outsite to R1 > > > +ovn-nbctl lsp-add outsite rp-outsite -- set Logical_Switch_Port > > rp-outsite \ > > > + type=router options:router-port=outsite \ > > > + -- lsp-set-addresses rp-outsite router > > > + > > > # Logical port 'foo1' in switch 'foo'. > > > ADD_NAMESPACES(foo1) > > > ADD_VETH(foo1, foo1, br-int, "192.168.1.2/24", "f0:00:00:01:02:03", \ > > > @@ -1299,15 +1344,26 @@ ADD_VETH(alice1, alice1, br-int, " > 172.16.1.2/24", > > "f0:00:00:01:02:05", \ > > > ovn-nbctl lsp-add alice alice1 \ > > > -- lsp-set-addresses alice1 "f0:00:00:01:02:05 172.16.1.2" > > > > > > +# Logical port 'outsite1' in switch 'outsite'. > > > +ADD_NAMESPACES(outsite1) > > > +ADD_VETH(outsite1, outsite1, br-int, "172.16.2.2/24", > > "f0:00:00:01:02:07", \ > > > + "172.16.2.1") > > > +ovn-nbctl lsp-add outsite outsite1 \ > > > +-- lsp-set-addresses outsite1 "f0:00:00:01:02:07 172.16.2.2" > > > + > > > # Add DNAT rules > > > AT_CHECK([ovn-nbctl lr-nat-add R1 dnat_and_snat 172.16.1.3 192.168.1.2 > > foo1 00:00:02:02:03:04]) > > > AT_CHECK([ovn-nbctl lr-nat-add R1 dnat_and_snat 172.16.1.4 192.168.1.3 > > foo2 00:00:02:02:03:05]) > > > +AT_CHECK([ovn-nbctl lr-nat-add R1 dnat_and_snat 172.16.2.3 192.168.1.2 > > foo1 00:00:02:02:03:04]) > > > +AT_CHECK([ovn-nbctl lr-nat-add R1 dnat_and_snat 172.16.2.4 192.168.1.3 > > foo2 00:00:02:02:03:05]) > > > > > > # Add a SNAT rule > > > AT_CHECK([ovn-nbctl lr-nat-add R1 snat 172.16.1.1 192.168.0.0/16]) > > > +AT_CHECK([ovn-nbctl lr-nat-add R1 snat 172.16.2.1 192.168.2.0/24]) > > > > > > ovn-nbctl --wait=hv sync > > > OVS_WAIT_UNTIL([ovs-ofctl dump-flows br-int | grep > > 'nat(src=172.16.1.1)']) > > > +OVS_WAIT_UNTIL([ovs-ofctl dump-flows br-int | grep > > 'nat(src=172.16.2.1)']) > > > > > > # North-South DNAT: 'alice1' pings 'foo1' using 172.16.1.3. > > > NS_CHECK_EXEC([alice1], [ping -q -c 3 -i 0.3 -w 2 172.16.1.3 | > > FORMAT_PING], \ > > > @@ -1315,11 +1371,21 @@ NS_CHECK_EXEC([alice1], [ping -q -c 3 -i 0.3 -w > > 2 172.16.1.3 | FORMAT_PING], \ > > > 3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > ]) > > > > > > +# North-South DNAT: 'outsite1' pings 'foo1' using 172.16.2.3. > > > +NS_CHECK_EXEC([outsite1], [ping -q -c 3 -i 0.3 -w 2 172.16.2.3 | > > FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > # We verify that DNAT indeed happened via 'dump-conntrack' command. > > > AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.3) | \ > > > sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl > > > icmp,orig=(src=172.16.1.2,dst=172.16.1.3,id=<cleared>,type= > > 8,code=0),reply=(src=192.168.1.2,dst=172.16.1.2,id=< > > cleared>,type=0,code=0),zone=<cleared> > > > ]) > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.2.3) | \ > > > +sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl > > > +icmp,orig=(src=172.16.2.2,dst=172.16.2.3,id=<cleared>, > > type=8,code=0),reply=(src=192.168.1.2,dst=172.16.2.2,id=< > > cleared>,type=0,code=0),zone=<cleared> > > > +]) > > > > > > # South-North SNAT: 'foo2' pings 'alice1'. But 'alice1' receives > traffic > > > # from 172.16.1.4 > > > @@ -1328,11 +1394,22 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 > > 172.16.1.2 | FORMAT_PING], \ > > > 3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > ]) > > > > > > +# South-North SNAT: 'foo2' pings 'outsite1'. But 'outsite1' receives > > traffic > > > +# from 172.16.2.4 > > > +NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 172.16.2.2 | > > FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > # We verify that SNAT indeed happened via 'dump-conntrack' command. > > > AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.4) | \ > > > sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl > > > icmp,orig=(src=192.168.1.3,dst=172.16.1.2,id=<cleared>, > > type=8,code=0),reply=(src=172.16.1.2,dst=172.16.1.4,id=< > > cleared>,type=0,code=0),zone=<cleared> > > > ]) > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.2.4) | \ > > > +sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl > > > +icmp,orig=(src=192.168.1.3,dst=172.16.2.2,id=<cleared>, > > type=8,code=0),reply=(src=172.16.2.2,dst=172.16.2.4,id=< > > cleared>,type=0,code=0),zone=<cleared> > > > +]) > > > > > > # South-North SNAT: 'bar1' pings 'alice1'. But 'alice1' receives > traffic > > > # from 172.16.1.1 > > > @@ -1341,11 +1418,22 @@ NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 > > 172.16.1.2 | FORMAT_PING], \ > > > 3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > ]) > > > > > > +# South-North SNAT: 'bar1' pings 'outsite1'. But 'outsite1' receives > > traffic > > > +# from 172.16.2.1 > > > +NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 172.16.2.2 | > > FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > # We verify that SNAT indeed happened via 'dump-conntrack' command. > > > AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.1) | \ > > > sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl > > > icmp,orig=(src=192.168.2.2,dst=172.16.1.2,id=<cleared>, > > type=8,code=0),reply=(src=172.16.1.2,dst=172.16.1.1,id=< > > cleared>,type=0,code=0),zone=<cleared> > > > ]) > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.2.1) | \ > > > +sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl > > > +icmp,orig=(src=192.168.2.2,dst=172.16.2.2,id=<cleared>, > > type=8,code=0),reply=(src=172.16.2.2,dst=172.16.2.1,id=< > > cleared>,type=0,code=0),zone=<cleared> > > > +]) > > > > > > OVS_APP_EXIT_AND_WAIT([ovn-controller]) > > > > > > @@ -1385,23 +1473,27 @@ start_daemon ovn-controller > > > > > > # Logical network: > > > # One LR R1 with switches foo (192.168.1.0/24), bar (192.168.2.0/24), > > > -# and alice (172.16.1.0/24) connected to it. The port between R1 and > > > -# alice is the router gateway port where the R1 NAT rules are applied. > > > +# alice (172.16.1.0/24) and outsite (172.16.1.0/24) connected to it. > > > +# The port between R1 and alice/outsite is the router gateway port > > > +# where the R1 NAT rules are applied. > > > # > > > -# foo -- R1 -- alice > > > -# | > > > -# bar ---- > > > +# foo ---+--- bar > > > +# R1 > > > +# alice --+-- outsite > > > > > > ovn-nbctl lr-add R1 > > > > > > ovn-nbctl ls-add foo > > > ovn-nbctl ls-add bar > > > ovn-nbctl ls-add alice > > > +ovn-nbctl ls-add outsite > > > > > > ovn-nbctl lrp-add R1 foo 00:00:01:01:02:03 192.168.1.1/24 > > > ovn-nbctl lrp-add R1 bar 00:00:01:01:02:04 192.168.2.1/24 > > > ovn-nbctl lrp-add R1 alice 00:00:02:01:02:03 172.16.1.1/24 \ > > > -- set Logical_Router_Port alice options:redirect-chassis=hv1 > > > +ovn-nbctl lrp-add R1 outsite 00:00:03:01:02:01 172.16.2.1/24 \ > > > + -- set Logical_Router_Port outsite options:redirect-chassis=hv1 > > > > > > # Connect foo to R1 > > > ovn-nbctl lsp-add foo rp-foo -- set Logical_Switch_Port rp-foo \ > > > @@ -1418,6 +1510,11 @@ ovn-nbctl lsp-add alice rp-alice -- set > > Logical_Switch_Port rp-alice \ > > > type=router options:router-port=alice \ > > > -- lsp-set-addresses rp-alice router > > > > > > +# Connect outsite to R1 > > > +ovn-nbctl lsp-add outsite rp-outsite -- set Logical_Switch_Port > > rp-outsite \ > > > + type=router options:router-port=outsite \ > > > + -- lsp-set-addresses rp-outsite router > > > + > > > # Logical port 'foo1' in switch 'foo'. > > > ADD_NAMESPACES(foo1) > > > ADD_VETH(foo1, foo1, br-int, "192.168.1.2/24", "f0:00:00:01:02:03", \ > > > @@ -1446,15 +1543,26 @@ ADD_VETH(alice1, alice1, br-int, " > 172.16.1.2/24", > > "f0:00:00:01:02:05", \ > > > ovn-nbctl lsp-add alice alice1 \ > > > -- lsp-set-addresses alice1 "f0:00:00:01:02:05 172.16.1.2" > > > > > > +# Logical port 'outsite1' in switch 'outsite'. > > > +ADD_NAMESPACES(outsite1) > > > +ADD_VETH(outsite1, outsite1, br-int, "172.16.2.2/24", > > "f0:00:00:01:02:07", \ > > > + "172.16.2.1") > > > +ovn-nbctl lsp-add outsite outsite1 \ > > > +-- lsp-set-addresses outsite1 "f0:00:00:01:02:07 172.16.2.2" > > > + > > > # Add DNAT rules > > > AT_CHECK([ovn-nbctl lr-nat-add R1 dnat_and_snat 172.16.1.3 192.168.1.2 > > foo1 00:00:02:02:03:04]) > > > AT_CHECK([ovn-nbctl lr-nat-add R1 dnat_and_snat 172.16.1.4 192.168.2.2 > > bar1 00:00:02:02:03:05]) > > > +AT_CHECK([ovn-nbctl lr-nat-add R1 dnat_and_snat 172.16.2.3 192.168.1.2 > > foo1 00:00:02:02:03:04]) > > > +AT_CHECK([ovn-nbctl lr-nat-add R1 dnat_and_snat 172.16.2.4 192.168.2.2 > > bar1 00:00:02:02:03:05]) > > > > > > # Add a SNAT rule > > > AT_CHECK([ovn-nbctl lr-nat-add R1 snat 172.16.1.1 192.168.0.0/16]) > > > +AT_CHECK([ovn-nbctl lr-nat-add R1 snat 172.16.2.1 192.168.1.0/24]) > > > > > > ovn-nbctl --wait=hv sync > > > OVS_WAIT_UNTIL([ovs-ofctl dump-flows br-int | grep > > 'nat(src=172.16.1.1)']) > > > +OVS_WAIT_UNTIL([ovs-ofctl dump-flows br-int | grep > > 'nat(src=172.16.2.1)']) > > > > > > echo "------ hv dump ------" > > > ovs-ofctl show br-int > > > @@ -1500,6 +1608,12 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 > > 172.16.1.4 | FORMAT_PING], \ > > > 3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > ]) > > > > > > +# East-West NAT: 'foo1' pings 'bar1' using 172.16.2.4. > > > +NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 172.16.2.4 | > > FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > # Check conntrack entries. First SNAT of 'foo1' address happens. > > > # Then DNAT of 'bar1' address happens (listed first below). > > > AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.3) | \ > > > @@ -1507,6 +1621,11 @@ sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], > > [dnl > > > icmp,orig=(src=172.16.1.3,dst=172.16.1.4,id=<cleared>,type= > > 8,code=0),reply=(src=192.168.2.2,dst=172.16.1.3,id=< > > cleared>,type=0,code=0),zone=<cleared> > > > icmp,orig=(src=192.168.1.2,dst=172.16.1.4,id=<cleared>, > > type=8,code=0),reply=(src=172.16.1.4,dst=172.16.1.3,id=< > > cleared>,type=0,code=0),zone=<cleared> > > > ]) > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.2.3) | \ > > > +sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl > > > +icmp,orig=(src=172.16.2.3,dst=172.16.2.4,id=<cleared>, > > type=8,code=0),reply=(src=192.168.2.2,dst=172.16.2.3,id=< > > cleared>,type=0,code=0),zone=<cleared> > > > +icmp,orig=(src=192.168.1.2,dst=172.16.2.4,id=<cleared>, > > type=8,code=0),reply=(src=172.16.2.4,dst=172.16.2.3,id=< > > cleared>,type=0,code=0),zone=<cleared> > > > +]) > > > > > > # East-West NAT: 'foo2' pings 'bar1' using 172.16.1.4. > > > NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | > > FORMAT_PING], \ > > > @@ -1514,6 +1633,12 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 > > 172.16.1.4 | FORMAT_PING], \ > > > 3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > ]) > > > > > > +# East-West NAT: 'foo2' pings 'bar1' using 172.16.2.4. > > > +NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 172.16.2.4 | > > FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > # Check conntrack entries. First SNAT of 'foo2' address happens. > > > # Then DNAT of 'bar1' address happens (listed first below). > > > AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.1) | \ > > > @@ -1521,6 +1646,11 @@ sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], > > [dnl > > > icmp,orig=(src=172.16.1.1,dst=172.16.1.4,id=<cleared>,type= > > 8,code=0),reply=(src=192.168.2.2,dst=172.16.1.1,id=< > > cleared>,type=0,code=0),zone=<cleared> > > > icmp,orig=(src=192.168.1.3,dst=172.16.1.4,id=<cleared>, > > type=8,code=0),reply=(src=172.16.1.4,dst=172.16.1.1,id=< > > cleared>,type=0,code=0),zone=<cleared> > > > ]) > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.2.1) | \ > > > +sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl > > > +icmp,orig=(src=172.16.2.1,dst=172.16.2.4,id=<cleared>, > > type=8,code=0),reply=(src=192.168.2.2,dst=172.16.2.1,id=< > > cleared>,type=0,code=0),zone=<cleared> > > > +icmp,orig=(src=192.168.1.3,dst=172.16.2.4,id=<cleared>, > > type=8,code=0),reply=(src=172.16.2.4,dst=172.16.2.1,id=< > > cleared>,type=0,code=0),zone=<cleared> > > > +]) > > > > > > OVS_APP_EXIT_AND_WAIT([ovn-controller]) > > > > > > -- > > > 2.13.2.windows.1 > > > > > > _______________________________________________ > > > dev mailing list > > > [email protected] > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > _______________________________________________ > dev mailing list > [email protected] > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
