On Thu, Sep 26, 2024 at 10:55 AM Roberto Bartzen Acosta via dev <[email protected]> wrote: > > Hi Numan, > > Thanks for your feedback and review. > > Em qua., 25 de set. de 2024 às 19:50, Numan Siddique <[email protected]> > escreveu: > > > On Thu, Sep 19, 2024 at 6:12 PM Roberto Bartzen Acosta via dev > > <[email protected]> wrote: > > > > > > This commit fixes the build_distr_lrouter_nat_flows_for_lb function to > > > include a DNAT flow entry for each DGP in use. Since we have added > > support > > > to create multiple gateway ports per logical router, it's necessary to > > > include in the LR NAT rules pipeline a specific entry for each attached > > DGP. > > > Otherwise, the inbound traffic will only be redirected when the incoming > > LRP > > > matches the chassis_resident field. > > > > > > Additionally, this patch includes the ability to use load-balancer with > > DGPs > > > attached to multiple chassis. We can have each of the DGPs associated > > with a > > > different chassis, and in this case the DNAT rules added by default will > > not > > > be enough to guarantee outgoing traffic. > > > > > > To solve the multiple chassis for DGPs problem, this patch include a new > > > config options to be configured in the load-balancer. If the > > use_stateless_nat > > > is set to true, the logical router that references this load-balancer > > will use > > > Stateless NAT rules when the logical router has multiple DGPs. After > > applying > > > this patch and setting the use_stateless_nat option, the inbound and/or > > > outbound traffic can pass through any chassis where the DGP resides > > without > > > having problems with CT state. > > > > > > Reported-at: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2054322 > > > Fixes: 15348b7b806f ("ovn-northd: Multiple distributed gateway port > > support.") > > > > > > Signed-off-by: Roberto Bartzen Acosta <[email protected]> > > > > Hi Roberto, > > > > Thanks for the patch. I tested this patch using the test example in > > multinode.at. > > > > The test case adds the below load balancer > > > > [root@ovn-central ~]# ovn-nbctl lb-list > > UUID LB PROTO > > VIP IPs > > f3e29869-3bb5-4df0-960a-171106f5913a lb0 tcp > > 172.16.0.100:9000 10.0.0.3:80,10.0.0.4:80 > > > > And the below logical flows are generated by this patch > > > > -------- > > [root@ovn-central ~]# ovn-sbctl dump-flows lr0 | grep 172.16.0.100 > > table=6 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst > > == 172.16.0.100), action=(ct_dnat;) > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > > is_chassis_resident("cr-lr0-public-p1")), > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80, > > 10.0.0.4:80);) > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > > is_chassis_resident("cr-lr0-public-p2")), > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80, > > 10.0.0.4:80);) > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > > tcp.src == 80)) && (inport == "lr0-public-p1" || outport == > > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1") && tcp), > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > > tcp.src == 80)) && (inport == "lr0-public-p2" || outport == > > "lr0-public-p2") && is_chassis_resident("cr-lr0-public-p2") && tcp), > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > > -------------- > > > > > > I fail to understand the reason for modifying the ip4.dst before > > calling ct_lb_mark. Can you please explain why ? Because the > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;) will first modify the > > ip4.dst to 10.0.0.3 and > > then to 10.0.0.4 and then the ct_lb_mark will actually do the > > conntrack with NAT to either 10.0.0.3 or 10.0.0.4. > > > > Is it because you want the conntrack entry to not have 172.16.0.100 ? > > > > The only reason I included this ip4.dst action in the DNAT rule is because > it's required to accept packets coming from a chassis that doesn't have > previously created conntrack entries. The main feature introduced in this > patch is to allow the administrator to have multiple DGPs attached to > different chassis (is_chassis_resident...). So, my implementation was based > on the normal behavior when using stateless NAT for external addresses, > where we need to add the ipx.dst in lr_in_dnat for traffic to be received > on the chassis (put the DGP port as chassis_resident match, as is the case > with stateless NAT [1] with DGP[2]). > > The question is, if we only have the ct_lb_mark, packets that pass through > the chassis and are already part of an active flow in another chassis (same > IPs and Ports) will be dropped because there is no correspondence in the > backend. So only packets with the NEW flag will be accepted and sent to the > backend (at least for TCP traffic). If we only have the ip4.dst action, > this will always perform the dnat for the same backend, without balancing. > Therefore, the combination of the two actions allows the packet to always > be received (regardless of whether conntrack is active for it), and > ct_lb_mark will take care of balancing for different backends. > > If we had conntrack sync between different chassis this would not be > necessary, as the ct_lb_mark action could always be executed without > dropping packets due to lack of correspondence in the conntrack table.
I still need to understand it properly. I'll get back to you after reading your reply thoroughly. But what's the point of doing multiple ip4.dst modifications for the same packet ? i.e - action=(ip4.dst = 10.0.0.3; ip4.dst = 10.0.0.4.....) The 2nd ip4.dst action overwrites and the packet's ip4.dst will be 10.0.0.4 before ct_lb_mark. > > [1] > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15737 > [2] > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15726 > > > > > > Also I don't understand why this patch adds the logical flows in > > "lr_out_snat" stage ? > > > > The flow for lr_out_snat is necessary for the correct functioning of > stateless NAT for the same reason explained previously. I mean, if the > outgoing packet is redirected to a chassis that doesn't have an active > conntrack entry, it will not be NATed by ct_lb action because it doesn't > refer to a valid flow (use case with ecmp). > > So it is necessary to create a stateless SNAT rule (similar to this [3]) > with a lower priority than the other router pipeline entries, in this case, > if the packet is not SNATed by ct_lb (conntrack missed) it will be SNATed > by stateless NAT rule. > > [3] > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15884 > > > > > > > Using the system multinode test as an example, the below fails > > (which is a regression) > > > > --- > > root@ovn-chassis-3 ~]# ip netns exec publicp1 nc -vz 10.0.0.3 80 > > ---- > > > > In the above test, publicp1 with IP 20.0.0.3 when it tries to connect > > to one if the backends directly (without the LB VIP), it fails. > > It fails because of the logical flows in "lr_out_snat". > > > > > > Looks to me the solution proposed here is incomplete. > > > > Also please note that in our CI we run the multinode tests > > periodically once a day using the v0.1 of the ovn-fake-multinode > > and the tests you added will fail. This needs to be fixed and until > > we move to the latest version of ovn-fake-multinode. > > > > I imagine that the test you are doing is using the same port as the LB > backend (TCP 80 in this case). So, the stateless lr_out_snat flow will > force the output to be SNATed because this port is in use by the backend. > Traffic to/from other ports will work without problems and will follow the > normal programmed flows (e.g. ICMP). > > This is necessary to ensure the egress traffic because the DGPs are > distributed across multiple chassis. Also, this setup is being validated in > the test ovn-fake-multinode testcase (ICMP from the backends chassis use > the router's default SNAT and not the LB's). I didn't understand the > regression you mentioned because this was programmed to be stateless and > it's traffic that uses the same ports as the LB backend, could you explain > better? > To reproduce the issue I mentioned, first run the multinode test you have added. Don't destroy the resources. And then login to ovn-chassis-3 container and run the command #ip netns exec publicp1 curl -v 10.0.0.3:80 So instead of curling to the LB VIP (which is 172.16.0.100:80), curl directly to the backends. Without your patch it works and with your patch it doesn't. If you run tcpdump you'd see the below 16:51:24.526980 40:54:00:00:00:03 > 00:00:00:00:ff:02, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 44276, offset 0, flags [DF], proto TCP (6), length 60) 20.0.0.3.46978 > 10.0.0.3.http: Flags [S], cksum 0x1e34 (incorrect -> 0x4b96), seq 1989771329, win 65280, options [mss 1360,sackOK,TS val 1753008285 ecr 0,nop,wscale 7], length 0 16:51:24.528559 00:00:00:00:ff:02 > 40:54:00:00:00:03, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60) 172.16.0.100.http > 20.0.0.3.46978: Flags [S.], cksum 0xc0a5 (incorrect -> 0x2094), seq 1088931300, ack 1989771330, win 64704, options [mss 1360,sackOK,TS val 3729696164 ecr 1753008285,nop,wscale 7], length 0 16:51:24.528571 40:54:00:00:00:03 > 00:00:00:00:ff:02, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40) The request is sent to 10.0.0.3 but the reply is received from 172.16.0.100 which is wrong. Thanks Numan > Thanks, > Roberto > > > > Thanks > > Numan > > > > > > > --- > > > northd/en-lr-stateful.c | 12 - > > > northd/northd.c | 116 ++++++-- > > > ovn-nb.xml | 10 + > > > tests/multinode-macros.at | 40 +++ > > > tests/multinode.at | 556 ++++++++++++++++++++++++++++++++++++++ > > > tests/ovn-northd.at | 320 ++++++++++++++++++++++ > > > 6 files changed, 1017 insertions(+), 37 deletions(-) > > > > > > diff --git a/northd/en-lr-stateful.c b/northd/en-lr-stateful.c > > > index baf1bd2f8..f09691af6 100644 > > > --- a/northd/en-lr-stateful.c > > > +++ b/northd/en-lr-stateful.c > > > @@ -516,18 +516,6 @@ lr_stateful_record_create(struct lr_stateful_table > > *table, > > > > > > table->array[od->index] = lr_stateful_rec; > > > > > > - /* Load balancers are not supported (yet) if a logical router has > > multiple > > > - * distributed gateway port. Log a warning. */ > > > - if (lr_stateful_rec->has_lb_vip && lr_has_multiple_gw_ports(od)) { > > > - static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1); > > > - VLOG_WARN_RL(&rl, "Load-balancers are configured on logical " > > > - "router %s, which has %"PRIuSIZE" distributed " > > > - "gateway ports. Load-balancer is not supported " > > > - "yet when there is more than one distributed " > > > - "gateway port on the router.", > > > - od->nbr->name, od->n_l3dgw_ports); > > > - } > > > - > > > return lr_stateful_rec; > > > } > > > > > > diff --git a/northd/northd.c b/northd/northd.c > > > index a267cd5f8..bbe97acf8 100644 > > > --- a/northd/northd.c > > > +++ b/northd/northd.c > > > @@ -11807,31 +11807,30 @@ static void > > > build_distr_lrouter_nat_flows_for_lb(struct lrouter_nat_lb_flows_ctx > > *ctx, > > > enum lrouter_nat_lb_flow_type type, > > > struct ovn_datapath *od, > > > - struct lflow_ref *lflow_ref) > > > + struct lflow_ref *lflow_ref, > > > + struct ovn_port *dgp, > > > + bool stateless_nat) > > > { > > > - struct ovn_port *dgp = od->l3dgw_ports[0]; > > > - > > > - const char *undnat_action; > > > - > > > - switch (type) { > > > - case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > > - undnat_action = "flags.force_snat_for_lb = 1; next;"; > > > - break; > > > - case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > > - undnat_action = "flags.skip_snat_for_lb = 1; next;"; > > > - break; > > > - case LROUTER_NAT_LB_FLOW_NORMAL: > > > - case LROUTER_NAT_LB_FLOW_MAX: > > > - undnat_action = lrouter_use_common_zone(od) > > > - ? "ct_dnat_in_czone;" > > > - : "ct_dnat;"; > > > - break; > > > - } > > > + struct ds dnat_action = DS_EMPTY_INITIALIZER; > > > > > > /* Store the match lengths, so we can reuse the ds buffer. */ > > > size_t new_match_len = ctx->new_match->length; > > > size_t undnat_match_len = ctx->undnat_match->length; > > > > > > + /* dnat_action: Add the LB backend IPs as a destination action of > > the > > > + * lr_in_dnat NAT rule with cumulative effect because > > any > > > + * backend dst IP used in the action list will > > redirect the > > > + * packet to the ct_lb pipeline. > > > + */ > > > + if (stateless_nat) { > > > + for (size_t i = 0; i < ctx->lb_vip->n_backends; i++) { > > > + struct ovn_lb_backend *backend = &ctx->lb_vip->backends[i]; > > > + bool ipv6 = !IN6_IS_ADDR_V4MAPPED(&backend->ip); > > > + ds_put_format(&dnat_action, "%s.dst=%s;", ipv6 ? "ip6" : > > "ip4", > > > + backend->ip_str); > > > + } > > > + } > > > + ds_put_format(&dnat_action, "%s", ctx->new_action[type]); > > > > > > const char *meter = NULL; > > > > > > @@ -11841,20 +11840,46 @@ build_distr_lrouter_nat_flows_for_lb(struct > > lrouter_nat_lb_flows_ctx *ctx, > > > > > > if (ctx->lb_vip->n_backends || !ctx->lb_vip->empty_backend_rej) { > > > ds_put_format(ctx->new_match, " && is_chassis_resident(%s)", > > > - od->l3dgw_ports[0]->cr_port->json_key); > > > + dgp->cr_port->json_key); > > > } > > > > > > ovn_lflow_add_with_hint__(ctx->lflows, od, S_ROUTER_IN_DNAT, > > ctx->prio, > > > - ds_cstr(ctx->new_match), > > ctx->new_action[type], > > > + ds_cstr(ctx->new_match), > > ds_cstr(&dnat_action), > > > NULL, meter, &ctx->lb->nlb->header_, > > > lflow_ref); > > > > > > ds_truncate(ctx->new_match, new_match_len); > > > > > > + ds_destroy(&dnat_action); > > > if (!ctx->lb_vip->n_backends) { > > > return; > > > } > > > > > > + struct ds undnat_action = DS_EMPTY_INITIALIZER; > > > + struct ds snat_action = DS_EMPTY_INITIALIZER; > > > + > > > + switch (type) { > > > + case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > > + ds_put_format(&undnat_action, "flags.force_snat_for_lb = 1; > > next;"); > > > + break; > > > + case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > > + ds_put_format(&undnat_action, "flags.skip_snat_for_lb = 1; > > next;"); > > > + break; > > > + case LROUTER_NAT_LB_FLOW_NORMAL: > > > + case LROUTER_NAT_LB_FLOW_MAX: > > > + ds_put_format(&undnat_action, "%s", > > > + lrouter_use_common_zone(od) ? "ct_dnat_in_czone;" > > > + : "ct_dnat;"); > > > + break; > > > + } > > > + > > > + /* undnat_action: Remove the ct action from the lr_out_undenat NAT > > rule. > > > + */ > > > + if (stateless_nat) { > > > + ds_clear(&undnat_action); > > > + ds_put_format(&undnat_action, "next;"); > > > + } > > > + > > > /* We need to centralize the LB traffic to properly perform > > > * the undnat stage. > > > */ > > > @@ -11873,11 +11898,41 @@ build_distr_lrouter_nat_flows_for_lb(struct > > lrouter_nat_lb_flows_ctx *ctx, > > > ds_put_format(ctx->undnat_match, ") && (inport == %s || outport == > > %s)" > > > " && is_chassis_resident(%s)", dgp->json_key, > > dgp->json_key, > > > dgp->cr_port->json_key); > > > + /* Use the LB protocol as matching criteria for out undnat and snat > > when > > > + * creating LBs with stateless NAT. */ > > > + if (stateless_nat) { > > > + ds_put_format(ctx->undnat_match, " && %s", ctx->lb->proto); > > > + } > > > ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_UNDNAT, 120, > > > - ds_cstr(ctx->undnat_match), undnat_action, > > > - &ctx->lb->nlb->header_, > > > + ds_cstr(ctx->undnat_match), > > > + ds_cstr(&undnat_action), > > &ctx->lb->nlb->header_, > > > lflow_ref); > > > + > > > + /* snat_action: Add a new lr_out_snat rule with the LB VIP as > > source IP > > > + * action to perform the NAT stateless pipeline > > completely. > > > + */ > > > + if (stateless_nat) { > > > + if (ctx->lb_vip->port_str) { > > > + ds_put_format(&snat_action, "%s.src=%s; %s.src=%s; next;", > > > + ctx->lb_vip->address_family == AF_INET6 ? > > > + "ip6" : "ip4", > > > + ctx->lb_vip->vip_str, ctx->lb->proto, > > > + ctx->lb_vip->port_str); > > > + } else { > > > + ds_put_format(&snat_action, "%s.src=%s; next;", > > > + ctx->lb_vip->address_family == AF_INET6 ? > > > + "ip6" : "ip4", > > > + ctx->lb_vip->vip_str); > > > + } > > > + ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_SNAT, 160, > > > + ds_cstr(ctx->undnat_match), > > > + ds_cstr(&snat_action), > > &ctx->lb->nlb->header_, > > > + lflow_ref); > > > + } > > > + > > > ds_truncate(ctx->undnat_match, undnat_match_len); > > > + ds_destroy(&undnat_action); > > > + ds_destroy(&snat_action); > > > } > > > > > > static void > > > @@ -12022,6 +12077,8 @@ build_lrouter_nat_flows_for_lb( > > > * lflow generation for them. > > > */ > > > size_t index; > > > + bool use_stateless_nat = smap_get_bool(&lb->nlb->options, > > > + "use_stateless_nat", false); > > > BITMAP_FOR_EACH_1 (index, bitmap_len, lb_dps->nb_lr_map) { > > > struct ovn_datapath *od = lr_datapaths->array[index]; > > > enum lrouter_nat_lb_flow_type type; > > > @@ -12043,8 +12100,17 @@ build_lrouter_nat_flows_for_lb( > > > if (!od->n_l3dgw_ports) { > > > bitmap_set1(gw_dp_bitmap[type], index); > > > } else { > > > - build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, > > > - lb_dps->lflow_ref); > > > + /* Create stateless LB NAT rules when using multiple DGPs > > and > > > + * use_stateless_nat is true. > > > + */ > > > + bool stateless_nat = (od->n_l3dgw_ports > 1) > > > + ? use_stateless_nat : false; > > > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) { > > > + struct ovn_port *dgp = od->l3dgw_ports[i]; > > > + build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, > > > + lb_dps->lflow_ref, > > dgp, > > > + stateless_nat); > > > + } > > > } > > > > > > if (lb->affinity_timeout) { > > > diff --git a/ovn-nb.xml b/ovn-nb.xml > > > index 2836f58f5..ad03c6214 100644 > > > --- a/ovn-nb.xml > > > +++ b/ovn-nb.xml > > > @@ -2302,6 +2302,16 @@ or > > > local anymore by the ovn-controller. This option is set to > > > <code>false</code> by default. > > > </column> > > > + > > > + <column name="options" key="use_stateless_nat" > > > + type='{"type": "boolean"}'> > > > + If the load balancer is configured with > > <code>use_stateless_nat</code> > > > + option to <code>true</code>, the logical router that references > > this > > > + load balancer will use Stateless NAT rules when the logical > > router > > > + has multiple distributed gateway ports(DGP). Otherwise, the > > outbound > > > + traffic may be dropped in scenarios where we have different > > chassis > > > + for each DGP. This option is set to <code>false</code> by > > default. > > > + </column> > > > </group> > > > </table> > > > > > > diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at > > > index 757917626..2f69433fc 100644 > > > --- a/tests/multinode-macros.at > > > +++ b/tests/multinode-macros.at > > > @@ -40,6 +40,27 @@ m4_define([M_START_TCPDUMP], > > > ] > > > ) > > > > > > +# M_EXEC([fake_node], [command]) > > > +# > > > +# Execute 'command' in 'fakenode' > > > +m4_define([M_EXEC], > > > + [podman exec $1 $2]) > > > + > > > +# M_CHECK_EXEC([fake_node], [command], other_params...) > > > +# > > > +# Wrapper for AT_CHECK that executes 'command' inside 'fake_node''s'. > > > +# 'other_params' as passed as they are to AT_CHECK. > > > +m4_define([M_CHECK_EXEC], > > > + [ AT_CHECK([M_EXEC([$1], [$2])], m4_shift(m4_shift(m4_shift($@)))) ] > > > +) > > > + > > > +# M_FORMAT_CT([ip-addr]) > > > +# > > > +# Strip content from the piped input which would differ from test to > > test > > > +# and limit the output to the rows containing 'ip-addr'. > > > +# > > > +m4_define([M_FORMAT_CT], > > > + [[grep -F "dst=$1," | sed -e 's/id=[0-9]*/id=<cleared>/g' -e > > 's/state=[0-9_A-Z]*/state=<cleared>/g' | sort | uniq | sed -e > > 's/zone=[[0-9]]*/zone=<cleared>/' -e 's/mark=[[0-9]]*/mark=<cleared>/' ]]) > > > > > > OVS_START_SHELL_HELPERS > > > > > > @@ -76,6 +97,25 @@ multinode_nbctl () { > > > m_as ovn-central ovn-nbctl "$@" > > > } > > > > > > +check_fake_multinode_setup_by_nodes() { > > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > > + for c in $1 > > > + do > > > + AT_CHECK([m_as $c ovn-appctl -t ovn-controller version], [0], > > [ignore]) > > > + done > > > +} > > > + > > > +cleanup_multinode_resources_by_nodes() { > > > + m_as ovn-central rm -f /etc/ovn/ovnnb_db.db > > > + m_as ovn-central /usr/share/ovn/scripts/ovn-ctl restart_northd > > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > > + for c in $1 > > > + do > > > + m_as $c ovs-vsctl del-br br-int > > > + m_as $c ip --all netns delete > > > + done > > > +} > > > + > > > # m_count_rows TABLE [CONDITION...] > > > # > > > # Prints the number of rows in TABLE (that satisfy CONDITION). > > > diff --git a/tests/multinode.at b/tests/multinode.at > > > index a0eb8fc67..b1beb4d97 100644 > > > --- a/tests/multinode.at > > > +++ b/tests/multinode.at > > > @@ -1591,3 +1591,559 @@ AT_CHECK([cat ch1_eth2.tcpdump], [0], [dnl > > > ]) > > > > > > AT_CLEANUP > > > + > > > +AT_SETUP([ovn multinode load-balancer with multiple DGPs and multiple > > chassis]) > > > + > > > +# Check that ovn-fake-multinode setup is up and running - requires > > additional nodes > > > +check_fake_multinode_setup_by_nodes 'ovn-chassis-1 ovn-chassis-2 > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > > + > > > +# Delete the multinode NB and OVS resources before starting the test. > > > +cleanup_multinode_resources_by_nodes 'ovn-chassis-1 ovn-chassis-2 > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > > + > > > +# Network topology > > > +# > > > +# publicp1 (ovn-chassis-3) (20.0.0.3/24) > > > +# | > > > +# overlay > > > +# | > > > +# DGP public1 (ovn-gw-1) (20.0.0.1/24) > > > +# | > > > +# | > > > +# | > > > +# lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) 10.0.0.3/24 > > > +# | | > > > +# | + --- sw0p2 (ovn-chassis-2) 10.0.0.4/24 > > > +# | > > > +# DGP public2 (ovn-gw-2) (30.0.0.1/24) > > > +# | > > > +# overlay > > > +# | > > > +# publicp2 (ovn-chassis-4) (30.0.0.3/24) > > > + > > > +# Delete already used ovs-ports > > > +m_as ovn-chassis-1 ovs-vsctl del-port br-int sw0p1-p > > > +m_as ovn-chassis-2 ovs-vsctl del-port br-int sw0p2-p > > > +m_as ovn-chassis-1 ip link del sw0p1-p > > > +m_as ovn-chassis-2 ip link del sw0p2-p > > > +m_as ovn-chassis-3 ovs-vsctl del-port br-int publicp1-p > > > +m_as ovn-chassis-4 ovs-vsctl del-port br-int publicp2-p > > > +m_as ovn-chassis-3 ip link del publicp1-p > > > +m_as ovn-chassis-4 ip link del publicp2-p > > > + > > > +# Create East-West switch for LB backends > > > +check multinode_nbctl ls-add sw0 > > > +check multinode_nbctl lsp-add sw0 sw0-port1 > > > +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 > > 10.0.0.3 1000::3" > > > +check multinode_nbctl lsp-add sw0 sw0-port2 > > > +check multinode_nbctl lsp-set-addresses sw0-port2 "50:54:00:00:00:04 > > 10.0.0.4 1000::4" > > > + > > > +m_as ovn-chassis-1 /data/create_fake_vm.sh sw0-port1 sw0p1 > > 50:54:00:00:00:03 1400 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a > > > +m_as ovn-chassis-2 /data/create_fake_vm.sh sw0-port2 sw0p2 > > 50:54:00:00:00:04 1400 10.0.0.4 24 10.0.0.1 1000::4/64 1000::a > > > + > > > +m_wait_for_ports_up > > > + > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 > > 10.0.0.4 | FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 > > 10.0.0.3 | FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > +# Create a logical router and attach to sw0 > > > +check multinode_nbctl lr-add lr0 > > > +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24 > > 1000::a/64 > > > +check multinode_nbctl lsp-add sw0 sw0-lr0 > > > +check multinode_nbctl lsp-set-type sw0-lr0 router > > > +check multinode_nbctl lsp-set-addresses sw0-lr0 router > > > +check multinode_nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0 > > > + > > > +# create external connection for N/S traffic using multiple DGPs > > > +check multinode_nbctl ls-add public > > > + > > > +# DGP public1 > > > +check multinode_nbctl lsp-add public ln-public-1 > > > +check multinode_nbctl lsp-set-type ln-public-1 localnet > > > +check multinode_nbctl lsp-set-addresses ln-public-1 unknown > > > +check multinode_nbctl lsp-set-options ln-public-1 network_name=public1 > > > + > > > +# DGP public2 > > > +# create exteranl connection for N/S traffic > > > +check multinode_nbctl lsp-add public ln-public-2 > > > +check multinode_nbctl lsp-set-type ln-public-2 localnet > > > +check multinode_nbctl lsp-set-addresses ln-public-2 unknown > > > +check multinode_nbctl lsp-set-options ln-public-2 network_name=public2 > > > + > > > +# Attach DGP public1 to GW-1 and chassis-3 (overlay connectivity) > > > +m_as ovn-gw-1 ovs-vsctl set open . > > external-ids:ovn-bridge-mappings=public1:br-ex > > > +m_as ovn-chassis-3 ovs-vsctl set open . > > external-ids:ovn-bridge-mappings=public1:br-ex > > > + > > > +# Attach DGP public2 to GW-2 and chassis-4 (overlay connectivity) > > > +m_as ovn-gw-2 ovs-vsctl set open . > > external-ids:ovn-bridge-mappings=public2:br-ex > > > +m_as ovn-chassis-4 ovs-vsctl set open . > > external-ids:ovn-bridge-mappings=public2:br-ex > > > + > > > +# Create the external LR0 port to the DGP public1 > > > +check multinode_nbctl lsp-add public public-port1 > > > +check multinode_nbctl lsp-set-addresses public-port1 "40:54:00:00:00:03 > > 20.0.0.3 2000::3" > > > + > > > +check multinode_nbctl lrp-add lr0 lr0-public-p1 00:00:00:00:ff:02 > > 20.0.0.1/24 2000::a/64 > > > +check multinode_nbctl lsp-add public public-lr0-p1 > > > +check multinode_nbctl lsp-set-type public-lr0-p1 router > > > +check multinode_nbctl lsp-set-addresses public-lr0-p1 router > > > +check multinode_nbctl lsp-set-options public-lr0-p1 > > router-port=lr0-public-p1 > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p1 ovn-gw-1 10 > > > + > > > +# Create a VM on ovn-chassis-3 in the same public1 overlay > > > +m_as ovn-chassis-3 /data/create_fake_vm.sh public-port1 publicp1 > > 40:54:00:00:00:03 1400 20.0.0.3 24 20.0.0.1 2000::4/64 2000::a > > > + > > > +m_wait_for_ports_up public-port1 > > > + > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [ping -q -c 3 -i 0.3 -w 2 > > 20.0.0.1 | FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > +# Create the external LR0 port to the DGP public2 > > > +check multinode_nbctl lsp-add public public-port2 > > > +check multinode_nbctl lsp-set-addresses public-port2 "60:54:00:00:00:03 > > 30.0.0.3 3000::3" > > > + > > > +check multinode_nbctl lrp-add lr0 lr0-public-p2 00:00:00:00:ff:03 > > 30.0.0.1/24 3000::a/64 > > > +check multinode_nbctl lsp-add public public-lr0-p2 > > > +check multinode_nbctl lsp-set-type public-lr0-p2 router > > > +check multinode_nbctl lsp-set-addresses public-lr0-p2 router > > > +check multinode_nbctl lsp-set-options public-lr0-p2 > > router-port=lr0-public-p2 > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p2 ovn-gw-2 10 > > > + > > > +# Create a VM on ovn-chassis-4 in the same public2 overlay > > > +m_as ovn-chassis-4 /data/create_fake_vm.sh public-port2 publicp2 > > 60:54:00:00:00:03 1400 30.0.0.3 24 30.0.0.1 3000::4/64 3000::a > > > + > > > +m_wait_for_ports_up public-port2 > > > + > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [ping -q -c 3 -i 0.3 -w 2 > > 30.0.0.1 | FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > +# Add a default route for multiple DGPs - using ECMP > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 20.0.0.3 > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 30.0.0.3 > > > + > > > +# Add SNAT rules using gateway-port > > > +check multinode_nbctl --gateway-port lr0-public-p1 lr-nat-add lr0 snat > > 20.0.0.1 10.0.0.0/24 > > > +check multinode_nbctl --gateway-port lr0-public-p2 lr-nat-add lr0 snat > > 30.0.0.1 10.0.0.0/24 > > > + > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 > > 20.0.0.3 | FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 > > 30.0.0.3 | FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > +# create LB > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:80" "10.0.0.3:80, > > 10.0.0.4:80" > > > +check multinode_nbctl lr-lb-add lr0 lb0 > > > +check multinode_nbctl ls-lb-add sw0 lb0 > > > + > > > +# Set use_stateless_nat to true > > > +check multinode_nbctl set load_balancer lb0 > > options:use_stateless_nat=true > > > + > > > +# Start backend http services > > > +M_NS_DAEMONIZE([ovn-chassis-1], [sw0p1], [$PYTHON -m http.server --bind > > 10.0.0.3 80 >/dev/null 2>&1], [http1.pid]) > > > +M_NS_DAEMONIZE([ovn-chassis-2], [sw0p2], [$PYTHON -m http.server --bind > > 10.0.0.4 80 >/dev/null 2>&1], [http2.pid]) > > > + > > > +# wait for http server be ready > > > +sleep 2 > > > + > > > +# Flush conntrack entries for easier output parsing of next test. > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v 172.16.0.100:80 > > --retry 3 --max-time 1 --local-port 59002 2> curl.out']) > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [sh -c 'cat curl.out | > > grep -i -e connect | grep -v 'Server:''], \ > > > +[0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +* Closing connection > > > +]) > > > + > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v 172.16.0.100:80 > > --retry 3 --max-time 1 --local-port 59003 2> curl.out']) > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [sh -c 'cat curl.out | > > grep -i -e connect | grep -v 'Server:''], \ > > > +[0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +* Closing connection > > > +]) > > > + > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v 172.16.0.100:80 > > --retry 3 --max-time 1 --local-port 59001']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | > > M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v 172.16.0.100:80 > > --retry 3 --max-time 1 --local-port 59000']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | > > M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# create a big file on web servers for download > > > +M_NS_EXEC([ovn-chassis-1], [sw0p1], [dd bs=512 count=200000 > > if=/dev/urandom of=download_file]) > > > +M_NS_EXEC([ovn-chassis-2], [sw0p2], [dd bs=512 count=200000 > > if=/dev/urandom of=download_file]) > > > + > > > +# Flush conntrack entries for easier output parsing of next test. > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59004 > > 2>curl.out']) > > > + > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > + > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat > > curl.out | \ > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +* Closing connection > > > +]) > > > + > > > +# Check if we have only one backend for the same connection - orig + > > dest ports > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Check if gw-2 is empty to ensure that the traffic only come from/to > > the originator chassis via DGP public1 > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > > +0 > > > +]) > > > + > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" -c) > > > + > > > +if [[ $backend_check -gt 0 ]]; then > > > +# Backend resides on ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59004,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +else > > > +# Backend resides on ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59004,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +fi > > > + > > > +# Flush conntrack entries for easier output parsing of next test. > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +# Check the flows again for a new source port > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59005 > > 2>curl.out']) > > > + > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > + > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat > > curl.out | \ > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +* Closing connection > > > +]) > > > + > > > +# Check if we have only one backend for the same connection - orig + > > dest ports > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Check if gw-2 is empty to ensure that the traffic only come from/to > > the originator chassis via DGP public1 > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > > +0 > > > +]) > > > + > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" -c) > > > + > > > +if [[ $backend_check -gt 0 ]]; then > > > +# Backend resides on ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59005,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +else > > > +# Backend resides on ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59005,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +fi > > > + > > > +# Flush conntrack entries for easier output parsing of next test. > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +# Start a new test using the second DGP as origin (public2) > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59006 > > 2>curl.out']) > > > + > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > + > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat > > curl.out | \ > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +* Closing connection > > > +]) > > > + > > > +# Check if we have only one backend for the same connection - orig + > > dest ports > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Check if gw-1 is empty to ensure that the traffic only come from/to > > the originator chassis via DGP public2 > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > > +0 > > > +]) > > > + > > > +# Check the backend IP from ct entries on gw-2 (DGP public2) > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" -c) > > > + > > > +if [[ $backend_check -gt 0 ]]; then > > > +# Backend resides on ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59006,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +else > > > +# Backend resides on ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59006,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +fi > > > + > > > +# Flush conntrack entries for easier output parsing of next test. > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +# Check the flows again for a new source port using the second DGP as > > origin (public2) > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59007 > > 2>curl.out']) > > > + > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > + > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat > > curl.out | \ > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +* Closing connection > > > +]) > > > + > > > +# Check if we have only one backend for the same connection - orig + > > dest ports > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Check if gw-1 is empty to ensure that the traffic only come from/to > > the originator chassis via DGP public2 > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > > +0 > > > +]) > > > + > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" -c) > > > + > > > +if [[ $backend_check -gt 0 ]]; then > > > +# Backend resides on ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59007,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +else > > > +# Backend resides on ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59007,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +fi > > > + > > > +# Check multiple requests coming from DGP's public1 and public2 > > > + > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +200 OK > > > +* Closing connection > > > +]) > > > + > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +200 OK > > > +* Closing connection > > > +]) > > > + > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +200 OK > > > +* Closing connection > > > +]) > > > + > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +200 OK > > > +* Closing connection > > > +]) > > > + > > > +# Remove the LB and change the VIP port - different from the backend > > ports > > > +check multinode_nbctl lb-del lb0 > > > + > > > +# create LB again > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:9000" "10.0.0.3:80, > > 10.0.0.4:80" > > > +check multinode_nbctl lr-lb-add lr0 lb0 > > > +check multinode_nbctl ls-lb-add sw0 lb0 > > > + > > > +# Set use_stateless_nat to true > > > +check multinode_nbctl set load_balancer lb0 > > options:use_stateless_nat=true > > > + > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +# Check end-to-end request using a new port for VIP > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port 59008 > > 2>curl.out']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | > > M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > > +200 OK > > > +* Closing connection > > > +]) > > > + > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +# Check end-to-end request using a new port for VIP > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port 59008 > > 2>curl.out']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | > > M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > > +200 OK > > > +* Closing connection > > > +]) > > > + > > > +AT_CLEANUP > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at > > > index dcc3dbbc3..9e7a2f225 100644 > > > --- a/tests/ovn-northd.at > > > +++ b/tests/ovn-northd.at > > > @@ -13864,3 +13864,323 @@ check_no_redirect > > > > > > AT_CLEANUP > > > ]) > > > + > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + NAT > > Stateless)]) > > > +ovn_start > > > + > > > +check ovn-nbctl ls-add public > > > +check ovn-nbctl lr-add lr1 > > > + > > > +# lr1 DGP ts1 > > > +check ovn-nbctl ls-add ts1 > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 172.16.10.1/24 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > > + > > > +# lr1 DGP ts2 > > > +check ovn-nbctl ls-add ts2 > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 172.16.20.1/24 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > > + > > > +# lr1 DGP public > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 173.16.0.1/16 > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > + > > > +check ovn-nbctl ls-add s1 > > > +# s1 - lr1 > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 172.16.0.1" > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > + > > > +# s1 - backend vm1 > > > +check ovn-nbctl lsp-add s1 vm1 > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 172.16.0.101" > > > + > > > +# s1 - backend vm2 > > > +check ovn-nbctl lsp-add s1 vm2 > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 172.16.0.102" > > > + > > > +# s1 - backend vm3 > > > +check ovn-nbctl lsp-add s1 vm3 > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 172.16.0.103" > > > + > > > +# Add the lr1 DGP ts1 to the public switch > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > nat-addresses=router > > > + > > > +# Add the lr1 DGP ts2 to the public switch > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > nat-addresses=router > > > + > > > +# Add the lr1 DGP public to the public switch > > > +check ovn-nbctl lsp-add public public_lr1 > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > nat-addresses=router > > > + > > > +# Create the Load Balancer lb1 > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > > "172.16.0.103,172.16.0.102,172.16.0.101" > > > + > > > +# Set use_stateless_nat to true > > > +check ovn-nbctl --wait=sb set load_balancer lb1 > > options:use_stateless_nat=true > > > + > > > +# Associate load balancer to s1 > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > +check ovn-nbctl --wait=sb sync > > > + > > > +ovn-sbctl dump-flows s1 > s1flows > > > +AT_CAPTURE_FILE([s1flows]) > > > + > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep > > "30.0.0.1"], [0], [dnl > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > > +]) > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > "30.0.0.1"], [0], [dnl > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > ip4.dst == 30.0.0.1), > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > +]) > > > + > > > +# Associate load balancer to lr1 with DGP > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > +check ovn-nbctl --wait=sb sync > > > + > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > +AT_CAPTURE_FILE([lr1flows]) > > > + > > > +# Check stateless NAT rules for load balancer with multiple DGP > > > +# 1. Check if the backend IPs are in the ipX.dst action > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > "30.0.0.1"], [0], [dnl > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > is_chassis_resident("cr-lr1-ts1")), > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > is_chassis_resident("cr-lr1-ts2")), > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > is_chassis_resident("cr-lr1_public")), > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > +]) > > > + > > > +# 2. Check if the DGP ports are in the match with action next > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], [dnl > > > + table=??(lr_out_undnat ), priority=0 , match=(1), > > action=(next;) > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") && > > is_chassis_resident("cr-lr1_public") && tcp), action=(next;) > > > +]) > > > + > > > +# 3. Check if the VIP IP is in the ipX.src action > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl > > > + table=??(lr_out_snat ), priority=0 , match=(1), > > action=(next;) > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > > action=(next;) > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(ip4.src=30.0.0.1; next;) > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(ip4.src=30.0.0.1; next;) > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") && > > is_chassis_resident("cr-lr1_public") && tcp), action=(ip4.src=30.0.0.1; > > next;) > > > +]) > > > + > > > +AT_CLEANUP > > > +]) > > > + > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + NAT > > Stateless) - IPv6]) > > > +ovn_start > > > + > > > +check ovn-nbctl ls-add public > > > +check ovn-nbctl lr-add lr1 > > > + > > > +# lr1 DGP ts1 > > > +check ovn-nbctl ls-add ts1 > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > > 2001:db8:aaaa:1::1/64 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > > + > > > +# lr1 DGP ts2 > > > +check ovn-nbctl ls-add ts2 > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > > 2001:db8:aaaa:2::1/64 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > > + > > > +# lr1 DGP public > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > > 2001:db8:bbbb::1/64 > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 > > 2001:db8:aaaa:3::1/64 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > + > > > +check ovn-nbctl ls-add s1 > > > +# s1 - lr1 > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > > 2001:db8:aaaa:3::1" > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > + > > > +# s1 - backend vm1 > > > +check ovn-nbctl lsp-add s1 vm1 > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > > 2001:db8:aaaa:3::101" > > > + > > > +# s1 - backend vm2 > > > +check ovn-nbctl lsp-add s1 vm2 > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > > 2001:db8:aaaa:3::102" > > > + > > > +# s1 - backend vm3 > > > +check ovn-nbctl lsp-add s1 vm3 > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > > 2001:db8:aaaa:3::103" > > > + > > > +# Add the lr1 DGP ts1 to the public switch > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > nat-addresses=router > > > + > > > +# Add the lr1 DGP ts2 to the public switch > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > nat-addresses=router > > > + > > > +# Add the lr1 DGP public to the public switch > > > +check ovn-nbctl lsp-add public public_lr1 > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > nat-addresses=router > > > + > > > +# Create the Load Balancer lb1 > > > +check ovn-nbctl --wait=sb lb-add lb1 "2001:db8:cccc::1" > > "2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101" > > > + > > > +# Set use_stateless_nat to true > > > +check ovn-nbctl --wait=sb set load_balancer lb1 > > options:use_stateless_nat=true > > > + > > > +# Associate load balancer to s1 > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > +check ovn-nbctl --wait=sb sync > > > + > > > +ovn-sbctl dump-flows s1 > s1flows > > > +AT_CAPTURE_FILE([s1flows]) > > > + > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep > > "2001:db8:cccc::1"], [0], [dnl > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 > > && ip6.dst == 2001:db8:cccc::1), action=(xxreg1 = 2001:db8:cccc::1; > > ct_lb_mark;) > > > +]) > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > "2001:db8:cccc::1"], [0], [dnl > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > ip6.dst == 2001:db8:cccc::1), > > action=(ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > +]) > > > + > > > +# Associate load balancer to lr1 with DGP > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > +check ovn-nbctl --wait=sb sync > > > + > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > +AT_CAPTURE_FILE([lr1flows]) > > > + > > > +# Check stateless NAT rules for load balancer with multiple DGP > > > +# 1. Check if the backend IPs are in the ipX.dst action > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > "2001:db8:cccc::1"], [0], [dnl > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > is_chassis_resident("cr-lr1-ts1")), > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > is_chassis_resident("cr-lr1-ts2")), > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > is_chassis_resident("cr-lr1_public")), > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > +]) > > > + > > > +# 2. Check if the DGP ports are in the match with action next > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], [dnl > > > + table=??(lr_out_undnat ), priority=0 , match=(1), > > action=(next;) > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport == > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport == > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || outport == > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > > action=(next;) > > > +]) > > > + > > > +# 3. Check if the VIP IP is in the ipX.src action > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl > > > + table=??(lr_out_snat ), priority=0 , match=(1), > > action=(next;) > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > > action=(next;) > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport == > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), > > action=(ip6.src=2001:db8:cccc::1; next;) > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport == > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), > > action=(ip6.src=2001:db8:cccc::1; next;) > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || outport == > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > > action=(ip6.src=2001:db8:cccc::1; next;) > > > +]) > > > + > > > +AT_CLEANUP > > > +]) > > > + > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (DGP)]) > > > +ovn_start > > > + > > > +check ovn-nbctl ls-add public > > > +check ovn-nbctl lr-add lr1 > > > + > > > +# lr1 DGP ts1 > > > +check ovn-nbctl ls-add ts1 > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 172.16.10.1/24 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-1 > > > + > > > +# lr1 DGP ts2 > > > +check ovn-nbctl ls-add ts2 > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 172.16.20.1/24 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-1 > > > + > > > +# lr1 DGP public > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 173.16.0.1/16 > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > + > > > +check ovn-nbctl ls-add s1 > > > +# s1 - lr1 > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 172.16.0.1" > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > + > > > +# s1 - backend vm1 > > > +check ovn-nbctl lsp-add s1 vm1 > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 172.16.0.101" > > > + > > > +# s1 - backend vm2 > > > +check ovn-nbctl lsp-add s1 vm2 > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 172.16.0.102" > > > + > > > +# s1 - backend vm3 > > > +check ovn-nbctl lsp-add s1 vm3 > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 172.16.0.103" > > > + > > > +# Add the lr1 DGP ts1 to the public switch > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > nat-addresses=router > > > + > > > +# Add the lr1 DGP ts2 to the public switch > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > nat-addresses=router > > > + > > > +# Add the lr1 DGP public to the public switch > > > +check ovn-nbctl lsp-add public public_lr1 > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > nat-addresses=router > > > + > > > +# Create the Load Balancer lb1 > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > > "172.16.0.103,172.16.0.102,172.16.0.101" > > > + > > > +# Associate load balancer to s1 > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > +check ovn-nbctl --wait=sb sync > > > + > > > +ovn-sbctl dump-flows s1 > s1flows > > > +AT_CAPTURE_FILE([s1flows]) > > > + > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep > > "30.0.0.1"], [0], [dnl > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > > +]) > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > "30.0.0.1"], [0], [dnl > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > ip4.dst == 30.0.0.1), > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > +]) > > > + > > > +# Associate load balancer to lr1 with DGP > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > +check ovn-nbctl --wait=sb sync > > > + > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > +AT_CAPTURE_FILE([lr1flows]) > > > + > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > "30.0.0.1"], [0], [dnl > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > is_chassis_resident("cr-lr1-ts1")), > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > is_chassis_resident("cr-lr1-ts2")), > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > is_chassis_resident("cr-lr1_public")), > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > +]) > > > + > > > +AT_CLEANUP > > > +]) > > > -- > > > 2.34.1 > > > > > > > > > -- > > > > > > > > > > > > > > > _'Esta mensagem é direcionada apenas para os endereços constantes no > > > cabeçalho inicial. Se você não está listado nos endereços constantes no > > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa > > > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas > > estão > > > imediatamente anuladas e proibidas'._ > > > > > > > > > * **'Apesar do Magazine Luiza tomar > > > todas as precauções razoáveis para assegurar que nenhum vírus esteja > > > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade > > por > > > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos'.* > > > > > > > > > > > > _______________________________________________ > > > dev mailing list > > > [email protected] > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > > > > -- > > > > > _‘Esta mensagem é direcionada apenas para os endereços constantes no > cabeçalho inicial. Se você não está listado nos endereços constantes no > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão > imediatamente anuladas e proibidas’._ > > > * **‘Apesar do Magazine Luiza tomar > todas as precauções razoáveis para assegurar que nenhum vírus esteja > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.* > > > > _______________________________________________ > dev mailing list > [email protected] > https://mail.openvswitch.org/mailman/listinfo/ovs-dev _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
