Thanks Numan! Em seg., 7 de out. de 2024 às 11:34, Numan Siddique <[email protected]> escreveu:
> On Mon, Sep 30, 2024 at 1:57 PM Roberto Bartzen Acosta via dev > <[email protected]> wrote: > > > > Hi Numan, > > > > Em sex., 27 de set. de 2024 às 12:47, Numan Siddique <[email protected]> > > escreveu: > > > > > On Thu, Sep 26, 2024 at 10:55 AM Roberto Bartzen Acosta via dev > > > <[email protected]> wrote: > > > > > > > > Hi Numan, > > > > > > > > Thanks for your feedback and review. > > > > > > > > Em qua., 25 de set. de 2024 às 19:50, Numan Siddique <[email protected] > > > > > > escreveu: > > > > > > > > > On Thu, Sep 19, 2024 at 6:12 PM Roberto Bartzen Acosta via dev > > > > > <[email protected]> wrote: > > > > > > > > > > > > This commit fixes the build_distr_lrouter_nat_flows_for_lb > function > > > to > > > > > > include a DNAT flow entry for each DGP in use. Since we have > added > > > > > support > > > > > > to create multiple gateway ports per logical router, it's > necessary > > > to > > > > > > include in the LR NAT rules pipeline a specific entry for each > > > attached > > > > > DGP. > > > > > > Otherwise, the inbound traffic will only be redirected when the > > > incoming > > > > > LRP > > > > > > matches the chassis_resident field. > > > > > > > > > > > > Additionally, this patch includes the ability to use > load-balancer > > > with > > > > > DGPs > > > > > > attached to multiple chassis. We can have each of the DGPs > associated > > > > > with a > > > > > > different chassis, and in this case the DNAT rules added by > default > > > will > > > > > not > > > > > > be enough to guarantee outgoing traffic. > > > > > > > > > > > > To solve the multiple chassis for DGPs problem, this patch > include a > > > new > > > > > > config options to be configured in the load-balancer. If the > > > > > use_stateless_nat > > > > > > is set to true, the logical router that references this > load-balancer > > > > > will use > > > > > > Stateless NAT rules when the logical router has multiple DGPs. > After > > > > > applying > > > > > > this patch and setting the use_stateless_nat option, the inbound > > > and/or > > > > > > outbound traffic can pass through any chassis where the DGP > resides > > > > > without > > > > > > having problems with CT state. > > > > > > > > > > > > Reported-at: > > > https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2054322 > > > > > > Fixes: 15348b7b806f ("ovn-northd: Multiple distributed gateway > port > > > > > support.") > > > > > > > > > > > > Signed-off-by: Roberto Bartzen Acosta < > [email protected]> > > > > > > > > > > Hi Roberto, > > > > > > > > > > Thanks for the patch. I tested this patch using the test example > in > > > > > multinode.at. > > > > > > > > > > The test case adds the below load balancer > > > > > > > > > > [root@ovn-central ~]# ovn-nbctl lb-list > > > > > UUID LB PROTO > > > > > VIP IPs > > > > > f3e29869-3bb5-4df0-960a-171106f5913a lb0 tcp > > > > > 172.16.0.100:9000 10.0.0.3:80,10.0.0.4:80 > > > > > > > > > > And the below logical flows are generated by this patch > > > > > > > > > > -------- > > > > > [root@ovn-central ~]# ovn-sbctl dump-flows lr0 | grep 172.16.0.100 > > > > > table=6 (lr_in_defrag ), priority=100 , match=(ip && > ip4.dst > > > > > == 172.16.0.100), action=(ct_dnat;) > > > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > > > > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == > 9000 && > > > > > is_chassis_resident("cr-lr0-public-p1")), > > > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= > > > 10.0.0.3:80, > > > > > 10.0.0.4:80);) > > > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > > > > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == > 9000 && > > > > > is_chassis_resident("cr-lr0-public-p2")), > > > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= > > > 10.0.0.3:80, > > > > > 10.0.0.4:80);) > > > > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > > > > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > > > > > tcp.src == 80)) && (inport == "lr0-public-p1" || outport == > > > > > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1") && > tcp), > > > > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > > > > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > > > > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > > > > > tcp.src == 80)) && (inport == "lr0-public-p2" || outport == > > > > > "lr0-public-p2") && is_chassis_resident("cr-lr0-public-p2") && > tcp), > > > > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > > > > > -------------- > > > > > > > > > > > > > > > I fail to understand the reason for modifying the ip4.dst before > > > > > calling ct_lb_mark. Can you please explain why ? Because the > > > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;) will first modify the > > > > > ip4.dst to 10.0.0.3 and > > > > > then to 10.0.0.4 and then the ct_lb_mark will actually do the > > > > > conntrack with NAT to either 10.0.0.3 or 10.0.0.4. > > > > > > > > > > Is it because you want the conntrack entry to not have > 172.16.0.100 ? > > > > > > > > > > > > > > > > The only reason I included this ip4.dst action in the DNAT rule is > > > because > > > > it's required to accept packets coming from a chassis that doesn't > have > > > > previously created conntrack entries. The main feature introduced in > this > > > > patch is to allow the administrator to have multiple DGPs attached to > > > > different chassis (is_chassis_resident...). So, my implementation was > > > based > > > > on the normal behavior when using stateless NAT for external > addresses, > > > > where we need to add the ipx.dst in lr_in_dnat for traffic to be > received > > > > on the chassis (put the DGP port as chassis_resident match, as is the > > > case > > > > with stateless NAT [1] with DGP[2]). > > > > > > > > The question is, if we only have the ct_lb_mark, packets that pass > > > through > > > > the chassis and are already part of an active flow in another chassis > > > (same > > > > IPs and Ports) will be dropped because there is no correspondence in > the > > > > backend. So only packets with the NEW flag will be accepted and sent > to > > > the > > > > backend (at least for TCP traffic). If we only have the ip4.dst > action, > > > > this will always perform the dnat for the same backend, without > > > balancing. > > > > Therefore, the combination of the two actions allows the packet to > always > > > > be received (regardless of whether conntrack is active for it), and > > > > ct_lb_mark will take care of balancing for different backends. > > > > > > > > If we had conntrack sync between different chassis this would not be > > > > necessary, as the ct_lb_mark action could always be executed without > > > > dropping packets due to lack of correspondence in the conntrack > table. > > > > > > > > [1] > > > > > > > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15737 > > > > [2] > > > > > > > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15726 > > > > > > > > > > > > > > I'm sorry, but it's not 100% clear to me. I know that you've already > > > explained to Mark in the older version of this patch. > > > Can you please explain with an example ? > > > > > > Let's take the below topology you've added in the mutlinode test as > example > > > > > > # Network topology > > > # > > > # publicp1 (ovn-chassis-3) (20.0.0.3/24) > > > # | > > > # overlay > > > # | > > > # DGP public1 (ovn-gw-1) (20.0.0.1/24) > > > # | > > > # | > > > # | > > > # lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) 10.0.0.3/24 > > > # | | > > > # | + --- sw0p2 (ovn-chassis-2) 10.0.0.4/24 > > > # | > > > # DGP public2 (ovn-gw-2) (30.0.0.1/24) > > > # | > > > # overlay > > > # | > > > # publicp2 (ovn-chassis-4) (30.0.0.3/24) > > > > > > > > > load balancer is configured on lr0 -> ovn-nbctl lb-add lb0 > > > "172.16.0.100:80" "10.0.0.3:80,10.0.0.4:80" > > > and it is attached to both lr0 and sw0. > > > > > > Scenario 1: > > > > > > publicp1 with IP 20.0.0.3 sends TCP traffic to VIP 172.16.0.100. I > > > think this is what will happen > > > > > > - The packet (ip4.src = 20.0.0.3 , ip4.dst = 172.16.0.100, tcp.dst > > > = 80) from ovn-chassis-3 will be sent out via the localnet bridge > > > (br-ex) > > > and the packet is received on ovn-gw-1 (as DGP public1 is resident > > > on it) via the localnet bridge and the > > > packet first enters public logical switch pipeline and then the > > > lr0 router pipeline. > > > > > > - In the router's lr_in_dnat state, the packet will be load > > > balanced to one of the backends using ct_lb_mark. Lets say 10.0.0.3 > > > is chosen > > > - The packet from router pipeline lr0 enters sw0 switch pipeline and > > > then the packet is tunneled to ovn-chassis-1 and delivered to sw0p1. > > > - The reply packet (ip4.src = 10.0.0.3, ip4.dst = 20.0.0.3, tcp.src = > > > 80) will enter the router pipeline and since 20.0.0.0/24 is handled by > > > DGP public1, > > > the packet is tunnelled to ovn-gw-1. > > > - In the router pipeline of ovn-gw-1, the packet is undnatted from > > > ip4.src 10.0.0.3 to 172.16.0.100 and the packet is sent out via the > > > localnet bridge. > > > - ovn-chassis-3 receives the packet via the localnet bridge and into > > > br-int and finally to publicp1. > > > > > > > > > In this scenario the load balancing is handled and conntrack entries > > > are created in ovn-gw-1. And there is no need to add flows in > > > "lr_out_snat" for stateless NAT > > > or set ip4.dst to one or all of backend IPs before "ct_lb_mark" in > > > lr_in_dnat stage. > > > > > > Scenario 2: > > > publicp2 with IP 30.0.0.3 sends TCP traffic to VIP 172.16.0.100. > > > > > > This is similar to scenario 1. Except that load balancing happens in > > > ovn-gw-2. > > > since DGP public2 is on this chassis. > > > > > > > > > Scenario 3: > > > > > > An external entity with IP 20.0.0.50 sends TCP traffic to VIP > > > 172.16.0.100. > > > > > > - This scenario is similar to the first one. The packet from this > > > external entity is received on ovn-gw-1 via the localnet bridge. > > > Rest all is the same. > > > > > > Scenario 4: > > > > > > sw0p2 on ovn-chassis-2 sends TCP traffic to VIP 172.16.0.100. > > > > > > - Since sw0 is attached with load balancer lb0, load balancing > > > happens in the source chassis - ovn-chassis-2 itself and depending on > > > the backend chosen, > > > the packet is tunnelled to ovn-chassis-1 (if 10.0.0.3 is chosen) > > > or delivered directly to sw0p2 (if i0.0.0.4 is chosen). > > > > > > > > > Scenario 5: > > > > > > An external entity with IP 40.0.0.40 sends TCP traffic to VIP > > > 172.16.0.100 and there are 2 ECMP routes configured > > > 172.16.0.100 via 20.0.0.1 > > > 172.16.0.100 via 30.0.0.1 > > > > > > In this case if the packet uses the route via 20.0.0.1 it will be > > > received on ovn-gw-1 via the localnet bridge br-ex. > > > > > > - With your patch, ip4.dst is modified to the last backend in the > > > list and then ct_lb_mark chosen as one of the backends > > > - And then it is tunnelled to the destination chassis. > > > > > > > > > Is this the scenario 5 you're trying to address with the stateless > > > NAT ? How would the reply packet work ? > > > The reply packet from backend 10.0.0.3 can use either of the paths ? > > > i.e ovn-gw-1 or ovn-gw-2 ? > > > > > > > > > Yes, this scenario would be the main goal of this patch. > > Let me explain with an example related to the OVN interconnect (one of > the > > use cases of this patch). In the context of ovn-ic the backend 10.0.0.3 > > could reply packets using either of the paths (ovn-gw-1 or ovn-gw-2). > > > > > > > > LB VIP 172.16.0.100 > > > > | > > vm (40.0.0.40) - LR-EXT - lrp1 - chassis1 - Transit switch TS1 - > ovn-gw-1 > > - lr0-public-p1 - lr0 - sw0 - VM 10.0.0.3 (ovn-chassis-1) > > - lrp2 - chassis2 - Transit > switch > > TS2 - ovn-gw-2 - lr0-public-p2 - - VM 10.0.0.4 > > (ovn-chassis-2) > > > > > > Let's take an example of the ovn-ic in the above topology: > > > > Assumptions: > > A) logical router LR-EXT: route to 172.16.0.100/32 via ECMP > (lrp1/chassis1 > > and lrp2/chassis2) > > B) logical router lr0: route to 40.0.0.0/24 via ECMP > > (lr0-public-p1/ovn-gw-1 and lr0-public-p2/ovn-gw-2) > > > > > > 1 - VM 40.0.0.40 send a request to the LB 172.16.0.100:80 > > 2 - LR-EXT chooses one of the possible outgoing routes via ECMP - e.g. > lrp1 > > / chassis1 / TS1 > > 3 - The lr0 receives the ingoing traffic through the port lr0-public-p1 / > > ovn-gw-1 because the TS1 is used to start the traffic > > 4 - with this patch we added lr_in_dnat rules for all DGPs (with or > without > > the stateless NAT config flag) > > > > with the stateless NAT config flag True: > > ...is_chassis_resident("cr-lr0-public-p1")), > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= > 10.0.0.3:80 > > ,10.0.0.4:80);) > > ...is_chassis_resident("cr-lr0-public-p2")), > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= > 10.0.0.3:80 > > ,10.0.0.4:80);) > > 5 - The ip4.dst will perform the action cumulatively, basically the last > > ip4.dst action applied will be to change dst to 10.0.0.4. > > 5.1 - The ct_lb_mark will check the conntrack and validate if the traffic > > refers to a new one (TCP SYNC), and then, it executes the final action > > provided by ct_lb_mark and select one of the available backends. e.g > > 10.0.0.3 > > > > without the stateless NAT config flag: > > ...is_chassis_resident("cr-lr0-public-p1")), > action=(ct_lb_mark(backends= > > 10.0.0.3:80,10.0.0.4:80);) > > ...is_chassis_resident("cr-lr0-public-p2")), > action=(ct_lb_mark(backends= > > 10.0.0.3:80,10.0.0.4:80);) > > > > 5 - same as the step 5.1 (ct_lb_mark check only). > > > > 6 - VM 10.0.0.3 receives the new traffic and sends a reply (TCP SYN+ACK) > > > > 7 - lr0 receives the response packet (from VM 10.0.0.3) in the routing > > pipeline and route via one of the available paths (ECMP). > > For example: lr0-public-p2 / ovn-gw-2 / TS2 > > So, we have a different outgoing path of the original one. > > > > 8 - ovn-gw-2 receives (SYN+ACK). We have no conntrack entries on ovn-gw-2 > > to match and perform the LB SNAT action! The ct_lb_mark previously > creates > > the conntrack entry on chassis ovn-gw-1 (when receives TCP SYN). > > > > with the stateless NAT config flag True: > > The outgoing packet fail to match the lb conntrack but the packet is > > SNATed by the lr_out_snat rule with a priority lower then cl_lb_mark, > > executing after cl_lb_mark in the pipeline. > > > > 9 - The SYN+ACK packet crosses the TS2 and is delivered to VM 40.0.0.40 > via > > lrp2 (SRC = 172.16.0.100:80) > > > > lr_out_snat will match and SNATed: > > match=(ip4 && ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == > > 10.0.0.4 && tcp.src == 80)) && (inport == "lr0-public-p1" || outport == > > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1") && tcp), > > action=(ip4.src=172.16.0.100; tcp.src=80; next;) > > > > without the stateless NAT config flag: > > The outgoing will be returned to the VM 40.0.0.4 with the LR SNAT for > > public-p2 port as ip4.src! So, basically this will break the TCP > > handshake!!! > > > > 9 - The SYN+ACK packet cross the TS2 and is delivered to the VM 40.0.0.40 > > via lrp2 (SRC = 30.0.0.1:80) > > > > Moving forward on the happy path (stateless NAT is true): > > > > 10 - VM 40.0.0.40 sends an ACK to complete the TCP handshake. > > > > 11 - 2 - LR-EXT chooses one of the possible outgoing routes via ECMP > > Let's assume that LR-EXT correctly execute the ecmp algorithm (per flow > > basis) and forwards to the same initial path because we're using the same > > flow (src/dst IPs and src/dts TCP ports still the same). > > e.g. still using the lrp1 / chassis1 / TS1 > > > > 12 - The lr0 receive the ingoing traffic through the port lr0-public-p1 / > > ovn-gw-1 / TS1 > > > > With this patch we have 2 possible behaviours: > > > > 12.1 - Without Stateless NAT config flag > > > > table=1 (lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src > == > > 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && tcp.src == 80)) && > > (inport == "lr0-public-p1" || outport == > > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1")), > > action=(ct_dnat_in_czone;) > > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && !ct.rel > > && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > > is_chassis_resident("cr-lr0-public-p1")), action=(ct_lb_mark(backends= > > 10.0.0.3:80,10.0.0.4:80);) > > > > openflow: br-int (ovn-gw-1) > > cookie=0x9381c20e, duration=2361.675s, table=15, n_packets=2, > n_bytes=148, > > idle_age=1564, > > > priority=120,ct_state=+new-rel+trk,tcp,metadata=0x4,nw_dst=172.16.0.100,tp_dst=80 > > actions=group:3 > > > > ovs-appctl dpctl/dump-conntrack (ovn-gw-1) > > > tcp,orig=(src=40.0.0.40,dst=172.16.0.100,sport=35274,dport=80),reply=(src=10.0.0.3,dst=40.0.0.40,sport=80,dport=35274),zone=8,mark=2,protoinfo=(state=TIME_WAIT) > > > > At this point, the ACK packet will be discarded because we didn't perform > > the SYN+ACK return on this chassis (ovn-gw -1). So, without the > previously > > established conntrack to match the ACK packet we broke the TCP handshake. > > > > Remember the flow: > > -> SYN: ovn-gw-1 > > <- SYN+ACK: ovn-gw-2 > > -> ACK: ovn-gw-1 > > > > 12.2 - With stateless NAT config flag is True > > > > table=1 (lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src > == > > 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && tcp.src == 80)) && > > (inport == "lr0-public-p1" || outport == > > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1")), > action=(next;) > > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && !ct.rel > > && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > > is_chassis_resident("cr-lr0-public-p1")), > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= > 10.0.0.3:80 > > ,10.0.0.4:80);) > > > > openflow: br-int (ovn-gw-1) > > cookie=0xab99f240, duration=1086.022s, table=15, n_packets=610, > > n_bytes=48979, idle_age=825, > > > priority=120,ct_state=+new-rel+trk,tcp,metadata=0x2,nw_dst=172.16.0.100,tp_dst=80 > > actions=mod_nw_dst:10.0.0.3,mod_nw_dst:10.0.0.4,group:1 > > > > ovs-appctl dpctl/dump-conntrack (ovn-gw-1) > > > tcp,orig=(src=40.0.0.40,dst=10.0.0.3,sport=54632,dport=80),reply=(src=10.0.0.3,dst=40.0.0.40,sport=80,dport=54632),zone=7,mark=2,protoinfo=(state=SYN_SENT) > > > > This is the reason to use stateless NAT rules. Unlike the case where we > > don't have the SYN_SENT, now the ACK packet will be accepted and > forwarded > > after performing the mod_nw_dst action. > > > > How does this work? We already have the previous flow created by the > first > > packet (SYN), so regardless of the IP.dst modified in the action, the > > packet will be forwarded to the same backend that corresponds to the > > conntrack match that is in SYN_SENT state. That's why I created the > action > > with the two modifiers (ip4.dst + ct_lb_mark). Using only the ct_lb_mark > > action creates a strong dependency on the connection state match in the > > conntrack entry (which we don't have in these cases). However, only using > > ip4.dst doesn't create traffic balancing as it will always send to the > same > > backend (last in the ip4.dst). > > > > > > I hope this has helped clarify the design decisions when > creating/modifying > > flows (lr_in_dnat/lr_out_snat/lr_out_undnat). > > > > > > Thanks for the detailed explanation. Its clear to me now. > > I'd suggest the following > > 1. In lr_in_dnat stage, choose either the first backend or the last > backend to modify the ip4.dst before ct_lb_mark. > I've no strong preference. Either would do. > > 2. Please document or add comments in northd.c (and in ovn-nb.xml)so > that we don't lose the context later. > > 3. Please run the ovn-fake-multinode tests in your github CI repo and > make sure it passes. I think you can trigger the test in your repo > in the Actions tab. > > 4. In the setup you described above, is it possible to add the > (below) route in the vm (40.0.0.40) deployment so that 40.0.0.0.40 > can > talk directly to 10.0.0.3 and 10.0.0.4 instead of the LB VIP and > send the tcp traffic to 10.0.0.3 and see if it works ? > If not, then please document the same i.e with the option > stateless_nat in the LB, the backends cannot be directly > communicated. > > logical router LR-EXT: route to 10.0.0.0/24 via ECMP (lrp1/chassis1 > and lrp2/chassis2) > > I'll do your suggestions. Just a quick question: how can I customize/parametrize the ovn-fake-multinode elements to run on CI? I mean to enable 2 more ovn-chassis needed to run the related tests. I've enabled it locally in the ./ovn_cluster.sh file but how do I change this for the fake-multinode CI actions? Is it per job or is this global? Regarding your suggestion 4, I imagine it makes more sense to integrate/create a new test with the ovn-interconnect setup + LB + multiple DGPs, which allows us to have multiple paths on routers on both sides (addressing the case discussed above) Thanks, Roberto > Thanks > Numan > > > > > > > > > > > > Thanks > > > Numan > > > > > > > > > > > > > > > > Also I don't understand why this patch adds the logical flows in > > > > > "lr_out_snat" stage ? > > > > > > > > > > > > > The flow for lr_out_snat is necessary for the correct functioning of > > > > stateless NAT for the same reason explained previously. I mean, if > the > > > > outgoing packet is redirected to a chassis that doesn't have an > active > > > > conntrack entry, it will not be NATed by ct_lb action because it > doesn't > > > > refer to a valid flow (use case with ecmp). > > > > > > > > So it is necessary to create a stateless SNAT rule (similar to this > [3]) > > > > with a lower priority than the other router pipeline entries, in this > > > case, > > > > if the packet is not SNATed by ct_lb (conntrack missed) it will be > SNATed > > > > by stateless NAT rule. > > > > > > > > [3] > > > > > > > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15884 > > > > > > > > > > > > > > > > > > > > > > Using the system multinode test as an example, the below fails > > > > > (which is a regression) > > > > > > > > > > --- > > > > > root@ovn-chassis-3 ~]# ip netns exec publicp1 nc -vz 10.0.0.3 80 > > > > > ---- > > > > > > > > > > In the above test, publicp1 with IP 20.0.0.3 when it tries to > connect > > > > > to one if the backends directly (without the LB VIP), it fails. > > > > > It fails because of the logical flows in "lr_out_snat". > > > > > > > > > > > > > > > Looks to me the solution proposed here is incomplete. > > > > > > > > > > Also please note that in our CI we run the multinode tests > > > > > periodically once a day using the v0.1 of the ovn-fake-multinode > > > > > and the tests you added will fail. This needs to be fixed and > until > > > > > we move to the latest version of ovn-fake-multinode. > > > > > > > > > > > > > I imagine that the test you are doing is using the same port as the > LB > > > > backend (TCP 80 in this case). So, the stateless lr_out_snat flow > will > > > > force the output to be SNATed because this port is in use by the > backend. > > > > Traffic to/from other ports will work without problems and will > follow > > > the > > > > normal programmed flows (e.g. ICMP). > > > > > > > > This is necessary to ensure the egress traffic because the DGPs are > > > > distributed across multiple chassis. Also, this setup is being > validated > > > in > > > > the test ovn-fake-multinode testcase (ICMP from the backends chassis > use > > > > the router's default SNAT and not the LB's). I didn't understand the > > > > regression you mentioned because this was programmed to be stateless > and > > > > it's traffic that uses the same ports as the LB backend, could you > > > explain > > > > better? > > > > > > > > Thanks, > > > > Roberto > > > > > > > > > > > > > Thanks > > > > > Numan > > > > > > > > > > > > > > > > --- > > > > > > northd/en-lr-stateful.c | 12 - > > > > > > northd/northd.c | 116 ++++++-- > > > > > > ovn-nb.xml | 10 + > > > > > > tests/multinode-macros.at | 40 +++ > > > > > > tests/multinode.at | 556 > > > ++++++++++++++++++++++++++++++++++++++ > > > > > > tests/ovn-northd.at | 320 ++++++++++++++++++++++ > > > > > > 6 files changed, 1017 insertions(+), 37 deletions(-) > > > > > > > > > > > > diff --git a/northd/en-lr-stateful.c b/northd/en-lr-stateful.c > > > > > > index baf1bd2f8..f09691af6 100644 > > > > > > --- a/northd/en-lr-stateful.c > > > > > > +++ b/northd/en-lr-stateful.c > > > > > > @@ -516,18 +516,6 @@ lr_stateful_record_create(struct > > > lr_stateful_table > > > > > *table, > > > > > > > > > > > > table->array[od->index] = lr_stateful_rec; > > > > > > > > > > > > - /* Load balancers are not supported (yet) if a logical > router > > > has > > > > > multiple > > > > > > - * distributed gateway port. Log a warning. */ > > > > > > - if (lr_stateful_rec->has_lb_vip && > > > lr_has_multiple_gw_ports(od)) { > > > > > > - static struct vlog_rate_limit rl = > VLOG_RATE_LIMIT_INIT(1, > > > 1); > > > > > > - VLOG_WARN_RL(&rl, "Load-balancers are configured on > logical > > > " > > > > > > - "router %s, which has %"PRIuSIZE" > distributed " > > > > > > - "gateway ports. Load-balancer is not > supported > > > " > > > > > > - "yet when there is more than one > distributed " > > > > > > - "gateway port on the router.", > > > > > > - od->nbr->name, od->n_l3dgw_ports); > > > > > > - } > > > > > > - > > > > > > return lr_stateful_rec; > > > > > > } > > > > > > > > > > > > diff --git a/northd/northd.c b/northd/northd.c > > > > > > index a267cd5f8..bbe97acf8 100644 > > > > > > --- a/northd/northd.c > > > > > > +++ b/northd/northd.c > > > > > > @@ -11807,31 +11807,30 @@ static void > > > > > > build_distr_lrouter_nat_flows_for_lb(struct > lrouter_nat_lb_flows_ctx > > > > > *ctx, > > > > > > enum > lrouter_nat_lb_flow_type > > > type, > > > > > > struct ovn_datapath *od, > > > > > > - struct lflow_ref > *lflow_ref) > > > > > > + struct lflow_ref > *lflow_ref, > > > > > > + struct ovn_port *dgp, > > > > > > + bool stateless_nat) > > > > > > { > > > > > > - struct ovn_port *dgp = od->l3dgw_ports[0]; > > > > > > - > > > > > > - const char *undnat_action; > > > > > > - > > > > > > - switch (type) { > > > > > > - case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > > > > > - undnat_action = "flags.force_snat_for_lb = 1; next;"; > > > > > > - break; > > > > > > - case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > > > > > - undnat_action = "flags.skip_snat_for_lb = 1; next;"; > > > > > > - break; > > > > > > - case LROUTER_NAT_LB_FLOW_NORMAL: > > > > > > - case LROUTER_NAT_LB_FLOW_MAX: > > > > > > - undnat_action = lrouter_use_common_zone(od) > > > > > > - ? "ct_dnat_in_czone;" > > > > > > - : "ct_dnat;"; > > > > > > - break; > > > > > > - } > > > > > > + struct ds dnat_action = DS_EMPTY_INITIALIZER; > > > > > > > > > > > > /* Store the match lengths, so we can reuse the ds buffer. > */ > > > > > > size_t new_match_len = ctx->new_match->length; > > > > > > size_t undnat_match_len = ctx->undnat_match->length; > > > > > > > > > > > > + /* dnat_action: Add the LB backend IPs as a destination > action > > > of > > > > > the > > > > > > + * lr_in_dnat NAT rule with cumulative effect > > > because > > > > > any > > > > > > + * backend dst IP used in the action list will > > > > > redirect the > > > > > > + * packet to the ct_lb pipeline. > > > > > > + */ > > > > > > + if (stateless_nat) { > > > > > > + for (size_t i = 0; i < ctx->lb_vip->n_backends; i++) { > > > > > > + struct ovn_lb_backend *backend = > > > &ctx->lb_vip->backends[i]; > > > > > > + bool ipv6 = !IN6_IS_ADDR_V4MAPPED(&backend->ip); > > > > > > + ds_put_format(&dnat_action, "%s.dst=%s;", ipv6 ? > "ip6" : > > > > > "ip4", > > > > > > + backend->ip_str); > > > > > > + } > > > > > > + } > > > > > > + ds_put_format(&dnat_action, "%s", ctx->new_action[type]); > > > > > > > > > > > > const char *meter = NULL; > > > > > > > > > > > > @@ -11841,20 +11840,46 @@ > build_distr_lrouter_nat_flows_for_lb(struct > > > > > lrouter_nat_lb_flows_ctx *ctx, > > > > > > > > > > > > if (ctx->lb_vip->n_backends || > !ctx->lb_vip->empty_backend_rej) > > > { > > > > > > ds_put_format(ctx->new_match, " && > is_chassis_resident(%s)", > > > > > > - od->l3dgw_ports[0]->cr_port->json_key); > > > > > > + dgp->cr_port->json_key); > > > > > > } > > > > > > > > > > > > ovn_lflow_add_with_hint__(ctx->lflows, od, S_ROUTER_IN_DNAT, > > > > > ctx->prio, > > > > > > - ds_cstr(ctx->new_match), > > > > > ctx->new_action[type], > > > > > > + ds_cstr(ctx->new_match), > > > > > ds_cstr(&dnat_action), > > > > > > NULL, meter, > &ctx->lb->nlb->header_, > > > > > > lflow_ref); > > > > > > > > > > > > ds_truncate(ctx->new_match, new_match_len); > > > > > > > > > > > > + ds_destroy(&dnat_action); > > > > > > if (!ctx->lb_vip->n_backends) { > > > > > > return; > > > > > > } > > > > > > > > > > > > + struct ds undnat_action = DS_EMPTY_INITIALIZER; > > > > > > + struct ds snat_action = DS_EMPTY_INITIALIZER; > > > > > > + > > > > > > + switch (type) { > > > > > > + case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > > > > > + ds_put_format(&undnat_action, "flags.force_snat_for_lb > = 1; > > > > > next;"); > > > > > > + break; > > > > > > + case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > > > > > + ds_put_format(&undnat_action, "flags.skip_snat_for_lb = > 1; > > > > > next;"); > > > > > > + break; > > > > > > + case LROUTER_NAT_LB_FLOW_NORMAL: > > > > > > + case LROUTER_NAT_LB_FLOW_MAX: > > > > > > + ds_put_format(&undnat_action, "%s", > > > > > > + lrouter_use_common_zone(od) ? > > > "ct_dnat_in_czone;" > > > > > > + : "ct_dnat;"); > > > > > > + break; > > > > > > + } > > > > > > + > > > > > > + /* undnat_action: Remove the ct action from the > lr_out_undenat > > > NAT > > > > > rule. > > > > > > + */ > > > > > > + if (stateless_nat) { > > > > > > + ds_clear(&undnat_action); > > > > > > + ds_put_format(&undnat_action, "next;"); > > > > > > + } > > > > > > + > > > > > > /* We need to centralize the LB traffic to properly perform > > > > > > * the undnat stage. > > > > > > */ > > > > > > @@ -11873,11 +11898,41 @@ > build_distr_lrouter_nat_flows_for_lb(struct > > > > > lrouter_nat_lb_flows_ctx *ctx, > > > > > > ds_put_format(ctx->undnat_match, ") && (inport == %s || > outport > > > == > > > > > %s)" > > > > > > " && is_chassis_resident(%s)", dgp->json_key, > > > > > dgp->json_key, > > > > > > dgp->cr_port->json_key); > > > > > > + /* Use the LB protocol as matching criteria for out undnat > and > > > snat > > > > > when > > > > > > + * creating LBs with stateless NAT. */ > > > > > > + if (stateless_nat) { > > > > > > + ds_put_format(ctx->undnat_match, " && %s", > ctx->lb->proto); > > > > > > + } > > > > > > ovn_lflow_add_with_hint(ctx->lflows, od, > S_ROUTER_OUT_UNDNAT, > > > 120, > > > > > > - ds_cstr(ctx->undnat_match), > > > undnat_action, > > > > > > - &ctx->lb->nlb->header_, > > > > > > + ds_cstr(ctx->undnat_match), > > > > > > + ds_cstr(&undnat_action), > > > > > &ctx->lb->nlb->header_, > > > > > > lflow_ref); > > > > > > + > > > > > > + /* snat_action: Add a new lr_out_snat rule with the LB VIP > as > > > > > source IP > > > > > > + * action to perform the NAT stateless pipeline > > > > > completely. > > > > > > + */ > > > > > > + if (stateless_nat) { > > > > > > + if (ctx->lb_vip->port_str) { > > > > > > + ds_put_format(&snat_action, "%s.src=%s; %s.src=%s; > > > next;", > > > > > > + ctx->lb_vip->address_family == > AF_INET6 ? > > > > > > + "ip6" : "ip4", > > > > > > + ctx->lb_vip->vip_str, ctx->lb->proto, > > > > > > + ctx->lb_vip->port_str); > > > > > > + } else { > > > > > > + ds_put_format(&snat_action, "%s.src=%s; next;", > > > > > > + ctx->lb_vip->address_family == > AF_INET6 ? > > > > > > + "ip6" : "ip4", > > > > > > + ctx->lb_vip->vip_str); > > > > > > + } > > > > > > + ovn_lflow_add_with_hint(ctx->lflows, od, > S_ROUTER_OUT_SNAT, > > > 160, > > > > > > + ds_cstr(ctx->undnat_match), > > > > > > + ds_cstr(&snat_action), > > > > > &ctx->lb->nlb->header_, > > > > > > + lflow_ref); > > > > > > + } > > > > > > + > > > > > > ds_truncate(ctx->undnat_match, undnat_match_len); > > > > > > + ds_destroy(&undnat_action); > > > > > > + ds_destroy(&snat_action); > > > > > > } > > > > > > > > > > > > static void > > > > > > @@ -12022,6 +12077,8 @@ build_lrouter_nat_flows_for_lb( > > > > > > * lflow generation for them. > > > > > > */ > > > > > > size_t index; > > > > > > + bool use_stateless_nat = smap_get_bool(&lb->nlb->options, > > > > > > + "use_stateless_nat", > > > false); > > > > > > BITMAP_FOR_EACH_1 (index, bitmap_len, lb_dps->nb_lr_map) { > > > > > > struct ovn_datapath *od = lr_datapaths->array[index]; > > > > > > enum lrouter_nat_lb_flow_type type; > > > > > > @@ -12043,8 +12100,17 @@ build_lrouter_nat_flows_for_lb( > > > > > > if (!od->n_l3dgw_ports) { > > > > > > bitmap_set1(gw_dp_bitmap[type], index); > > > > > > } else { > > > > > > - build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, > > > > > > - > lb_dps->lflow_ref); > > > > > > + /* Create stateless LB NAT rules when using multiple > > > DGPs > > > > > and > > > > > > + * use_stateless_nat is true. > > > > > > + */ > > > > > > + bool stateless_nat = (od->n_l3dgw_ports > 1) > > > > > > + ? use_stateless_nat : false; > > > > > > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) { > > > > > > + struct ovn_port *dgp = od->l3dgw_ports[i]; > > > > > > + build_distr_lrouter_nat_flows_for_lb(&ctx, > type, od, > > > > > > + > > > lb_dps->lflow_ref, > > > > > dgp, > > > > > > + > stateless_nat); > > > > > > + } > > > > > > } > > > > > > > > > > > > if (lb->affinity_timeout) { > > > > > > diff --git a/ovn-nb.xml b/ovn-nb.xml > > > > > > index 2836f58f5..ad03c6214 100644 > > > > > > --- a/ovn-nb.xml > > > > > > +++ b/ovn-nb.xml > > > > > > @@ -2302,6 +2302,16 @@ or > > > > > > local anymore by the ovn-controller. This option is set > to > > > > > > <code>false</code> by default. > > > > > > </column> > > > > > > + > > > > > > + <column name="options" key="use_stateless_nat" > > > > > > + type='{"type": "boolean"}'> > > > > > > + If the load balancer is configured with > > > > > <code>use_stateless_nat</code> > > > > > > + option to <code>true</code>, the logical router that > > > references > > > > > this > > > > > > + load balancer will use Stateless NAT rules when the > logical > > > > > router > > > > > > + has multiple distributed gateway ports(DGP). Otherwise, > the > > > > > outbound > > > > > > + traffic may be dropped in scenarios where we have > different > > > > > chassis > > > > > > + for each DGP. This option is set to <code>false</code> > by > > > > > default. > > > > > > + </column> > > > > > > </group> > > > > > > </table> > > > > > > > > > > > > diff --git a/tests/multinode-macros.at b/tests/ > multinode-macros.at > > > > > > index 757917626..2f69433fc 100644 > > > > > > --- a/tests/multinode-macros.at > > > > > > +++ b/tests/multinode-macros.at > > > > > > @@ -40,6 +40,27 @@ m4_define([M_START_TCPDUMP], > > > > > > ] > > > > > > ) > > > > > > > > > > > > +# M_EXEC([fake_node], [command]) > > > > > > +# > > > > > > +# Execute 'command' in 'fakenode' > > > > > > +m4_define([M_EXEC], > > > > > > + [podman exec $1 $2]) > > > > > > + > > > > > > +# M_CHECK_EXEC([fake_node], [command], other_params...) > > > > > > +# > > > > > > +# Wrapper for AT_CHECK that executes 'command' inside > > > 'fake_node''s'. > > > > > > +# 'other_params' as passed as they are to AT_CHECK. > > > > > > +m4_define([M_CHECK_EXEC], > > > > > > + [ AT_CHECK([M_EXEC([$1], [$2])], > > > m4_shift(m4_shift(m4_shift($@)))) ] > > > > > > +) > > > > > > + > > > > > > +# M_FORMAT_CT([ip-addr]) > > > > > > +# > > > > > > +# Strip content from the piped input which would differ from > test to > > > > > test > > > > > > +# and limit the output to the rows containing 'ip-addr'. > > > > > > +# > > > > > > +m4_define([M_FORMAT_CT], > > > > > > + [[grep -F "dst=$1," | sed -e 's/id=[0-9]*/id=<cleared>/g' -e > > > > > 's/state=[0-9_A-Z]*/state=<cleared>/g' | sort | uniq | sed -e > > > > > 's/zone=[[0-9]]*/zone=<cleared>/' -e > 's/mark=[[0-9]]*/mark=<cleared>/' > > > ]]) > > > > > > > > > > > > OVS_START_SHELL_HELPERS > > > > > > > > > > > > @@ -76,6 +97,25 @@ multinode_nbctl () { > > > > > > m_as ovn-central ovn-nbctl "$@" > > > > > > } > > > > > > > > > > > > +check_fake_multinode_setup_by_nodes() { > > > > > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > > > > > + for c in $1 > > > > > > + do > > > > > > + AT_CHECK([m_as $c ovn-appctl -t ovn-controller version], > > > [0], > > > > > [ignore]) > > > > > > + done > > > > > > +} > > > > > > + > > > > > > +cleanup_multinode_resources_by_nodes() { > > > > > > + m_as ovn-central rm -f /etc/ovn/ovnnb_db.db > > > > > > + m_as ovn-central /usr/share/ovn/scripts/ovn-ctl > restart_northd > > > > > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > > > > > + for c in $1 > > > > > > + do > > > > > > + m_as $c ovs-vsctl del-br br-int > > > > > > + m_as $c ip --all netns delete > > > > > > + done > > > > > > +} > > > > > > + > > > > > > # m_count_rows TABLE [CONDITION...] > > > > > > # > > > > > > # Prints the number of rows in TABLE (that satisfy CONDITION). > > > > > > diff --git a/tests/multinode.at b/tests/multinode.at > > > > > > index a0eb8fc67..b1beb4d97 100644 > > > > > > --- a/tests/multinode.at > > > > > > +++ b/tests/multinode.at > > > > > > @@ -1591,3 +1591,559 @@ AT_CHECK([cat ch1_eth2.tcpdump], [0], > [dnl > > > > > > ]) > > > > > > > > > > > > AT_CLEANUP > > > > > > + > > > > > > +AT_SETUP([ovn multinode load-balancer with multiple DGPs and > > > multiple > > > > > chassis]) > > > > > > + > > > > > > +# Check that ovn-fake-multinode setup is up and running - > requires > > > > > additional nodes > > > > > > +check_fake_multinode_setup_by_nodes 'ovn-chassis-1 ovn-chassis-2 > > > > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > > > > > + > > > > > > +# Delete the multinode NB and OVS resources before starting the > > > test. > > > > > > +cleanup_multinode_resources_by_nodes 'ovn-chassis-1 > ovn-chassis-2 > > > > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > > > > > + > > > > > > +# Network topology > > > > > > +# > > > > > > +# publicp1 (ovn-chassis-3) (20.0.0.3/24) > > > > > > +# | > > > > > > +# overlay > > > > > > +# | > > > > > > +# DGP public1 (ovn-gw-1) (20.0.0.1/24) > > > > > > +# | > > > > > > +# | > > > > > > +# | > > > > > > +# lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) > > > 10.0.0.3/24 > > > > > > +# | | > > > > > > +# | + --- sw0p2 (ovn-chassis-2) > > > 10.0.0.4/24 > > > > > > +# | > > > > > > +# DGP public2 (ovn-gw-2) (30.0.0.1/24) > > > > > > +# | > > > > > > +# overlay > > > > > > +# | > > > > > > +# publicp2 (ovn-chassis-4) (30.0.0.3/24) > > > > > > + > > > > > > +# Delete already used ovs-ports > > > > > > +m_as ovn-chassis-1 ovs-vsctl del-port br-int sw0p1-p > > > > > > +m_as ovn-chassis-2 ovs-vsctl del-port br-int sw0p2-p > > > > > > +m_as ovn-chassis-1 ip link del sw0p1-p > > > > > > +m_as ovn-chassis-2 ip link del sw0p2-p > > > > > > +m_as ovn-chassis-3 ovs-vsctl del-port br-int publicp1-p > > > > > > +m_as ovn-chassis-4 ovs-vsctl del-port br-int publicp2-p > > > > > > +m_as ovn-chassis-3 ip link del publicp1-p > > > > > > +m_as ovn-chassis-4 ip link del publicp2-p > > > > > > + > > > > > > +# Create East-West switch for LB backends > > > > > > +check multinode_nbctl ls-add sw0 > > > > > > +check multinode_nbctl lsp-add sw0 sw0-port1 > > > > > > +check multinode_nbctl lsp-set-addresses sw0-port1 > "50:54:00:00:00:03 > > > > > 10.0.0.3 1000::3" > > > > > > +check multinode_nbctl lsp-add sw0 sw0-port2 > > > > > > +check multinode_nbctl lsp-set-addresses sw0-port2 > "50:54:00:00:00:04 > > > > > 10.0.0.4 1000::4" > > > > > > + > > > > > > +m_as ovn-chassis-1 /data/create_fake_vm.sh sw0-port1 sw0p1 > > > > > 50:54:00:00:00:03 1400 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a > > > > > > +m_as ovn-chassis-2 /data/create_fake_vm.sh sw0-port2 sw0p2 > > > > > 50:54:00:00:00:04 1400 10.0.0.4 24 10.0.0.1 1000::4/64 1000::a > > > > > > + > > > > > > +m_wait_for_ports_up > > > > > > + > > > > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 > -w 2 > > > > > 10.0.0.4 | FORMAT_PING], \ > > > > > > +[0], [dnl > > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > > +]) > > > > > > + > > > > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 > -w 2 > > > > > 10.0.0.3 | FORMAT_PING], \ > > > > > > +[0], [dnl > > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > > +]) > > > > > > + > > > > > > +# Create a logical router and attach to sw0 > > > > > > +check multinode_nbctl lr-add lr0 > > > > > > +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 > > > 10.0.0.1/24 > > > > > 1000::a/64 > > > > > > +check multinode_nbctl lsp-add sw0 sw0-lr0 > > > > > > +check multinode_nbctl lsp-set-type sw0-lr0 router > > > > > > +check multinode_nbctl lsp-set-addresses sw0-lr0 router > > > > > > +check multinode_nbctl lsp-set-options sw0-lr0 > router-port=lr0-sw0 > > > > > > + > > > > > > +# create external connection for N/S traffic using multiple DGPs > > > > > > +check multinode_nbctl ls-add public > > > > > > + > > > > > > +# DGP public1 > > > > > > +check multinode_nbctl lsp-add public ln-public-1 > > > > > > +check multinode_nbctl lsp-set-type ln-public-1 localnet > > > > > > +check multinode_nbctl lsp-set-addresses ln-public-1 unknown > > > > > > +check multinode_nbctl lsp-set-options ln-public-1 > > > network_name=public1 > > > > > > + > > > > > > +# DGP public2 > > > > > > +# create exteranl connection for N/S traffic > > > > > > +check multinode_nbctl lsp-add public ln-public-2 > > > > > > +check multinode_nbctl lsp-set-type ln-public-2 localnet > > > > > > +check multinode_nbctl lsp-set-addresses ln-public-2 unknown > > > > > > +check multinode_nbctl lsp-set-options ln-public-2 > > > network_name=public2 > > > > > > + > > > > > > +# Attach DGP public1 to GW-1 and chassis-3 (overlay > connectivity) > > > > > > +m_as ovn-gw-1 ovs-vsctl set open . > > > > > external-ids:ovn-bridge-mappings=public1:br-ex > > > > > > +m_as ovn-chassis-3 ovs-vsctl set open . > > > > > external-ids:ovn-bridge-mappings=public1:br-ex > > > > > > + > > > > > > +# Attach DGP public2 to GW-2 and chassis-4 (overlay > connectivity) > > > > > > +m_as ovn-gw-2 ovs-vsctl set open . > > > > > external-ids:ovn-bridge-mappings=public2:br-ex > > > > > > +m_as ovn-chassis-4 ovs-vsctl set open . > > > > > external-ids:ovn-bridge-mappings=public2:br-ex > > > > > > + > > > > > > +# Create the external LR0 port to the DGP public1 > > > > > > +check multinode_nbctl lsp-add public public-port1 > > > > > > +check multinode_nbctl lsp-set-addresses public-port1 > > > "40:54:00:00:00:03 > > > > > 20.0.0.3 2000::3" > > > > > > + > > > > > > +check multinode_nbctl lrp-add lr0 lr0-public-p1 > 00:00:00:00:ff:02 > > > > > 20.0.0.1/24 2000::a/64 > > > > > > +check multinode_nbctl lsp-add public public-lr0-p1 > > > > > > +check multinode_nbctl lsp-set-type public-lr0-p1 router > > > > > > +check multinode_nbctl lsp-set-addresses public-lr0-p1 router > > > > > > +check multinode_nbctl lsp-set-options public-lr0-p1 > > > > > router-port=lr0-public-p1 > > > > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p1 > > > ovn-gw-1 10 > > > > > > + > > > > > > +# Create a VM on ovn-chassis-3 in the same public1 overlay > > > > > > +m_as ovn-chassis-3 /data/create_fake_vm.sh public-port1 publicp1 > > > > > 40:54:00:00:00:03 1400 20.0.0.3 24 20.0.0.1 2000::4/64 2000::a > > > > > > + > > > > > > +m_wait_for_ports_up public-port1 > > > > > > + > > > > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [ping -q -c 3 -i > 0.3 > > > -w 2 > > > > > 20.0.0.1 | FORMAT_PING], \ > > > > > > +[0], [dnl > > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > > +]) > > > > > > + > > > > > > +# Create the external LR0 port to the DGP public2 > > > > > > +check multinode_nbctl lsp-add public public-port2 > > > > > > +check multinode_nbctl lsp-set-addresses public-port2 > > > "60:54:00:00:00:03 > > > > > 30.0.0.3 3000::3" > > > > > > + > > > > > > +check multinode_nbctl lrp-add lr0 lr0-public-p2 > 00:00:00:00:ff:03 > > > > > 30.0.0.1/24 3000::a/64 > > > > > > +check multinode_nbctl lsp-add public public-lr0-p2 > > > > > > +check multinode_nbctl lsp-set-type public-lr0-p2 router > > > > > > +check multinode_nbctl lsp-set-addresses public-lr0-p2 router > > > > > > +check multinode_nbctl lsp-set-options public-lr0-p2 > > > > > router-port=lr0-public-p2 > > > > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p2 > > > ovn-gw-2 10 > > > > > > + > > > > > > +# Create a VM on ovn-chassis-4 in the same public2 overlay > > > > > > +m_as ovn-chassis-4 /data/create_fake_vm.sh public-port2 publicp2 > > > > > 60:54:00:00:00:03 1400 30.0.0.3 24 30.0.0.1 3000::4/64 3000::a > > > > > > + > > > > > > +m_wait_for_ports_up public-port2 > > > > > > + > > > > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [ping -q -c 3 -i > 0.3 > > > -w 2 > > > > > 30.0.0.1 | FORMAT_PING], \ > > > > > > +[0], [dnl > > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > > +]) > > > > > > + > > > > > > +# Add a default route for multiple DGPs - using ECMP > > > > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 > > > 20.0.0.3 > > > > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 > > > 30.0.0.3 > > > > > > + > > > > > > +# Add SNAT rules using gateway-port > > > > > > +check multinode_nbctl --gateway-port lr0-public-p1 lr-nat-add > lr0 > > > snat > > > > > 20.0.0.1 10.0.0.0/24 > > > > > > +check multinode_nbctl --gateway-port lr0-public-p2 lr-nat-add > lr0 > > > snat > > > > > 30.0.0.1 10.0.0.0/24 > > > > > > + > > > > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 > -w 2 > > > > > 20.0.0.3 | FORMAT_PING], \ > > > > > > +[0], [dnl > > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > > +]) > > > > > > + > > > > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 > -w 2 > > > > > 30.0.0.3 | FORMAT_PING], \ > > > > > > +[0], [dnl > > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > > +]) > > > > > > + > > > > > > +# create LB > > > > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:80" "10.0.0.3:80 > , > > > > > 10.0.0.4:80" > > > > > > +check multinode_nbctl lr-lb-add lr0 lb0 > > > > > > +check multinode_nbctl ls-lb-add sw0 lb0 > > > > > > + > > > > > > +# Set use_stateless_nat to true > > > > > > +check multinode_nbctl set load_balancer lb0 > > > > > options:use_stateless_nat=true > > > > > > + > > > > > > +# Start backend http services > > > > > > +M_NS_DAEMONIZE([ovn-chassis-1], [sw0p1], [$PYTHON -m http.server > > > --bind > > > > > 10.0.0.3 80 >/dev/null 2>&1], [http1.pid]) > > > > > > +M_NS_DAEMONIZE([ovn-chassis-2], [sw0p2], [$PYTHON -m http.server > > > --bind > > > > > 10.0.0.4 80 >/dev/null 2>&1], [http2.pid]) > > > > > > + > > > > > > +# wait for http server be ready > > > > > > +sleep 2 > > > > > > + > > > > > > +# Flush conntrack entries for easier output parsing of next > test. > > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > > + > > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v > > > 172.16.0.100:80 > > > > > --retry 3 --max-time 1 --local-port 59002 2> curl.out']) > > > > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [sh -c 'cat > curl.out | > > > > > grep -i -e connect | grep -v 'Server:''], \ > > > > > > +[0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v > > > 172.16.0.100:80 > > > > > --retry 3 --max-time 1 --local-port 59003 2> curl.out']) > > > > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [sh -c 'cat > curl.out | > > > > > grep -i -e connect | grep -v 'Server:''], \ > > > > > > +[0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > > + > > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v > > > 172.16.0.100:80 > > > > > --retry 3 --max-time 1 --local-port 59001']) > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl > dpctl/dump-conntrack | > > > > > M_FORMAT_CT(20.0.0.3) | \ > > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > > [0], > > > > > [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v > > > 172.16.0.100:80 > > > > > --retry 3 --max-time 1 --local-port 59000']) > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl > dpctl/dump-conntrack | > > > > > M_FORMAT_CT(30.0.0.3) | \ > > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > > [0], > > > > > [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# create a big file on web servers for download > > > > > > +M_NS_EXEC([ovn-chassis-1], [sw0p1], [dd bs=512 count=200000 > > > > > if=/dev/urandom of=download_file]) > > > > > > +M_NS_EXEC([ovn-chassis-2], [sw0p2], [dd bs=512 count=200000 > > > > > if=/dev/urandom of=download_file]) > > > > > > + > > > > > > +# Flush conntrack entries for easier output parsing of next > test. > > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > > + > > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > > > 59004 > > > > > 2>curl.out']) > > > > > > + > > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl > dpctl/dump-conntrack | > > > sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl > dpctl/dump-conntrack | > > > sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > + > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 > cat > > > > > curl.out | \ > > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +# Check if we have only one backend for the same connection - > orig + > > > > > dest ports > > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > > [0], > > > > > [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Check if gw-2 is empty to ensure that the traffic only come > > > from/to > > > > > the originator chassis via DGP public1 > > > > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > + > > > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep > "dport=80" > > > -c) > > > > > > + > > > > > > +if [[ $backend_check -gt 0 ]]; then > > > > > > +# Backend resides on ovn-chassis-1 > > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59004,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep > "dport=80" > > > -c], > > > > > [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep > "dport=80" > > > > > -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +else > > > > > > +# Backend resides on ovn-chassis-2 > > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59004,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep > "dport=80" > > > -c], > > > > > [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep > "dport=80" > > > > > -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +fi > > > > > > + > > > > > > +# Flush conntrack entries for easier output parsing of next > test. > > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > > + > > > > > > +# Check the flows again for a new source port > > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > > > 59005 > > > > > 2>curl.out']) > > > > > > + > > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl > dpctl/dump-conntrack | > > > sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl > dpctl/dump-conntrack | > > > sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > + > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 > cat > > > > > curl.out | \ > > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +# Check if we have only one backend for the same connection - > orig + > > > > > dest ports > > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > > [0], > > > > > [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Check if gw-2 is empty to ensure that the traffic only come > > > from/to > > > > > the originator chassis via DGP public1 > > > > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > + > > > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep > "dport=80" > > > -c) > > > > > > + > > > > > > +if [[ $backend_check -gt 0 ]]; then > > > > > > +# Backend resides on ovn-chassis-1 > > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59005,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep > "dport=80" > > > -c], > > > > > [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep > "dport=80" > > > > > -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +else > > > > > > +# Backend resides on ovn-chassis-2 > > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59005,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep > "dport=80" > > > -c], > > > > > [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep > "dport=80" > > > > > -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +fi > > > > > > + > > > > > > +# Flush conntrack entries for easier output parsing of next > test. > > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > > + > > > > > > +# Start a new test using the second DGP as origin (public2) > > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > > > 59006 > > > > > 2>curl.out']) > > > > > > + > > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl > dpctl/dump-conntrack | > > > sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl > dpctl/dump-conntrack | > > > sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > + > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 > cat > > > > > curl.out | \ > > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +# Check if we have only one backend for the same connection - > orig + > > > > > dest ports > > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > > [0], > > > > > [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Check if gw-1 is empty to ensure that the traffic only come > > > from/to > > > > > the originator chassis via DGP public2 > > > > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > + > > > > > > +# Check the backend IP from ct entries on gw-2 (DGP public2) > > > > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep > "dport=80" > > > -c) > > > > > > + > > > > > > +if [[ $backend_check -gt 0 ]]; then > > > > > > +# Backend resides on ovn-chassis-1 > > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59006,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep > "dport=80" > > > -c], > > > > > [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep > "dport=80" > > > > > -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +else > > > > > > +# Backend resides on ovn-chassis-2 > > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59006,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep > "dport=80" > > > -c], > > > > > [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep > "dport=80" > > > > > -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +fi > > > > > > + > > > > > > +# Flush conntrack entries for easier output parsing of next > test. > > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > > + > > > > > > +# Check the flows again for a new source port using the second > DGP > > > as > > > > > origin (public2) > > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > > > 59007 > > > > > 2>curl.out']) > > > > > > + > > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl > dpctl/dump-conntrack | > > > sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl > dpctl/dump-conntrack | > > > sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > + > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 > cat > > > > > curl.out | \ > > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +# Check if we have only one backend for the same connection - > orig + > > > > > dest ports > > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > > [0], > > > > > [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Check if gw-1 is empty to ensure that the traffic only come > > > from/to > > > > > the originator chassis via DGP public2 > > > > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > + > > > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep > "dport=80" > > > -c) > > > > > > + > > > > > > +if [[ $backend_check -gt 0 ]]; then > > > > > > +# Backend resides on ovn-chassis-1 > > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59007,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep > "dport=80" > > > -c], > > > > > [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep > "dport=80" > > > > > -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +else > > > > > > +# Backend resides on ovn-chassis-2 > > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59007,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep > "dport=80" > > > -c], > > > > > [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep > "dport=80" > > > > > -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +fi > > > > > > + > > > > > > +# Check multiple requests coming from DGP's public1 and public2 > > > > > > + > > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" > | > > > grep > > > > > -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +200 OK > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" > | > > > grep > > > > > -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +200 OK > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" > | > > > grep > > > > > -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +200 OK > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" > | > > > grep > > > > > -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +200 OK > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +# Remove the LB and change the VIP port - different from the > backend > > > > > ports > > > > > > +check multinode_nbctl lb-del lb0 > > > > > > + > > > > > > +# create LB again > > > > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:9000" " > 10.0.0.3:80, > > > > > 10.0.0.4:80" > > > > > > +check multinode_nbctl lr-lb-add lr0 lb0 > > > > > > +check multinode_nbctl ls-lb-add sw0 lb0 > > > > > > + > > > > > > +# Set use_stateless_nat to true > > > > > > +check multinode_nbctl set load_balancer lb0 > > > > > options:use_stateless_nat=true > > > > > > + > > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > > + > > > > > > +# Check end-to-end request using a new port for VIP > > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 > --local-port > > > 59008 > > > > > 2>curl.out']) > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl > dpctl/dump-conntrack | > > > > > M_FORMAT_CT(20.0.0.3) | \ > > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > > [0], > > > > > [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" > | > > > grep > > > > > -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > > > > > +200 OK > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > > + > > > > > > +# Check end-to-end request using a new port for VIP > > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 > --local-port > > > 59008 > > > > > 2>curl.out']) > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl > dpctl/dump-conntrack | > > > > > M_FORMAT_CT(30.0.0.3) | \ > > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > > [0], > > > > > [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" > | > > > grep > > > > > -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > > > > > +200 OK > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +AT_CLEANUP > > > > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at > > > > > > index dcc3dbbc3..9e7a2f225 100644 > > > > > > --- a/tests/ovn-northd.at > > > > > > +++ b/tests/ovn-northd.at > > > > > > @@ -13864,3 +13864,323 @@ check_no_redirect > > > > > > > > > > > > AT_CLEANUP > > > > > > ]) > > > > > > + > > > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + > DGP + > > > NAT > > > > > Stateless)]) > > > > > > +ovn_start > > > > > > + > > > > > > +check ovn-nbctl ls-add public > > > > > > +check ovn-nbctl lr-add lr1 > > > > > > + > > > > > > +# lr1 DGP ts1 > > > > > > +check ovn-nbctl ls-add ts1 > > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > > > 172.16.10.1/24 > > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > > > > > + > > > > > > +# lr1 DGP ts2 > > > > > > +check ovn-nbctl ls-add ts2 > > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > > > 172.16.20.1/24 > > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > > > > > + > > > > > > +# lr1 DGP public > > > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > > > 173.16.0.1/16 > > > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 > 172.16.0.1/24 > > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > > > > + > > > > > > +check ovn-nbctl ls-add s1 > > > > > > +# s1 - lr1 > > > > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > > > 172.16.0.1" > > > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > > > > + > > > > > > +# s1 - backend vm1 > > > > > > +check ovn-nbctl lsp-add s1 vm1 > > > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > > > 172.16.0.101" > > > > > > + > > > > > > +# s1 - backend vm2 > > > > > > +check ovn-nbctl lsp-add s1 vm2 > > > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > > > 172.16.0.102" > > > > > > + > > > > > > +# s1 - backend vm3 > > > > > > +check ovn-nbctl lsp-add s1 vm3 > > > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > > > 172.16.0.103" > > > > > > + > > > > > > +# Add the lr1 DGP ts1 to the public switch > > > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 > router-port=lr1-ts1 > > > > > nat-addresses=router > > > > > > + > > > > > > +# Add the lr1 DGP ts2 to the public switch > > > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 > router-port=lr1-ts2 > > > > > nat-addresses=router > > > > > > + > > > > > > +# Add the lr1 DGP public to the public switch > > > > > > +check ovn-nbctl lsp-add public public_lr1 > > > > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > > > > +check ovn-nbctl lsp-set-options public_lr1 > router-port=lr1_public > > > > > nat-addresses=router > > > > > > + > > > > > > +# Create the Load Balancer lb1 > > > > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > > > > > "172.16.0.103,172.16.0.102,172.16.0.101" > > > > > > + > > > > > > +# Set use_stateless_nat to true > > > > > > +check ovn-nbctl --wait=sb set load_balancer lb1 > > > > > options:use_stateless_nat=true > > > > > > + > > > > > > +# Associate load balancer to s1 > > > > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > > > > +check ovn-nbctl --wait=sb sync > > > > > > + > > > > > > +ovn-sbctl dump-flows s1 > s1flows > > > > > > +AT_CAPTURE_FILE([s1flows]) > > > > > > + > > > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | > > > grep > > > > > "30.0.0.1"], [0], [dnl > > > > > > + table=??(ls_in_pre_stateful ), priority=120 , > match=(reg0[[2]] > > > == 1 > > > > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > > > > > +]) > > > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > > > > "30.0.0.1"], [0], [dnl > > > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new > && > > > > > ip4.dst == 30.0.0.1), > > > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > > +]) > > > > > > + > > > > > > +# Associate load balancer to lr1 with DGP > > > > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > > > > +check ovn-nbctl --wait=sb sync > > > > > > + > > > > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > > > > +AT_CAPTURE_FILE([lr1flows]) > > > > > > + > > > > > > +# Check stateless NAT rules for load balancer with multiple DGP > > > > > > +# 1. Check if the backend IPs are in the ipX.dst action > > > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > > > > "30.0.0.1"], [0], [dnl > > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new > && > > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > > is_chassis_resident("cr-lr1-ts1")), > > > > > > > > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new > && > > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > > is_chassis_resident("cr-lr1-ts2")), > > > > > > > > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new > && > > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > > is_chassis_resident("cr-lr1_public")), > > > > > > > > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > > +]) > > > > > > + > > > > > > +# 2. Check if the DGP ports are in the match with action next > > > > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], > [0], > > > [dnl > > > > > > + table=??(lr_out_undnat ), priority=0 , match=(1), > > > > > action=(next;) > > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || > (ip4.src == > > > > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > > > > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) > > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || > (ip4.src == > > > > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > > > > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) > > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || > (ip4.src == > > > > > 172.16.0.101)) && (inport == "lr1_public" || outport == > "lr1_public") > > > && > > > > > is_chassis_resident("cr-lr1_public") && tcp), action=(next;) > > > > > > +]) > > > > > > + > > > > > > +# 3. Check if the VIP IP is in the ipX.src action > > > > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], > [dnl > > > > > > + table=??(lr_out_snat ), priority=0 , match=(1), > > > > > action=(next;) > > > > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > > > > > action=(next;) > > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || > (ip4.src == > > > > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > > > > > is_chassis_resident("cr-lr1-ts1") && tcp), > action=(ip4.src=30.0.0.1; > > > next;) > > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || > (ip4.src == > > > > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > > > > > is_chassis_resident("cr-lr1-ts2") && tcp), > action=(ip4.src=30.0.0.1; > > > next;) > > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || > (ip4.src == > > > > > 172.16.0.101)) && (inport == "lr1_public" || outport == > "lr1_public") > > > && > > > > > is_chassis_resident("cr-lr1_public") && tcp), > action=(ip4.src=30.0.0.1; > > > > > next;) > > > > > > +]) > > > > > > + > > > > > > +AT_CLEANUP > > > > > > +]) > > > > > > + > > > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + > DGP + > > > NAT > > > > > Stateless) - IPv6]) > > > > > > +ovn_start > > > > > > + > > > > > > +check ovn-nbctl ls-add public > > > > > > +check ovn-nbctl lr-add lr1 > > > > > > + > > > > > > +# lr1 DGP ts1 > > > > > > +check ovn-nbctl ls-add ts1 > > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > > > > > 2001:db8:aaaa:1::1/64 > > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > > > > > + > > > > > > +# lr1 DGP ts2 > > > > > > +check ovn-nbctl ls-add ts2 > > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > > > > > 2001:db8:aaaa:2::1/64 > > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > > > > > + > > > > > > +# lr1 DGP public > > > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > > > > > 2001:db8:bbbb::1/64 > > > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 > > > > > 2001:db8:aaaa:3::1/64 > > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > > > > + > > > > > > +check ovn-nbctl ls-add s1 > > > > > > +# s1 - lr1 > > > > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > > > > > 2001:db8:aaaa:3::1" > > > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > > > > + > > > > > > +# s1 - backend vm1 > > > > > > +check ovn-nbctl lsp-add s1 vm1 > > > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > > > > > 2001:db8:aaaa:3::101" > > > > > > + > > > > > > +# s1 - backend vm2 > > > > > > +check ovn-nbctl lsp-add s1 vm2 > > > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > > > > > 2001:db8:aaaa:3::102" > > > > > > + > > > > > > +# s1 - backend vm3 > > > > > > +check ovn-nbctl lsp-add s1 vm3 > > > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > > > > > 2001:db8:aaaa:3::103" > > > > > > + > > > > > > +# Add the lr1 DGP ts1 to the public switch > > > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 > router-port=lr1-ts1 > > > > > nat-addresses=router > > > > > > + > > > > > > +# Add the lr1 DGP ts2 to the public switch > > > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 > router-port=lr1-ts2 > > > > > nat-addresses=router > > > > > > + > > > > > > +# Add the lr1 DGP public to the public switch > > > > > > +check ovn-nbctl lsp-add public public_lr1 > > > > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > > > > +check ovn-nbctl lsp-set-options public_lr1 > router-port=lr1_public > > > > > nat-addresses=router > > > > > > + > > > > > > +# Create the Load Balancer lb1 > > > > > > +check ovn-nbctl --wait=sb lb-add lb1 "2001:db8:cccc::1" > > > > > "2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101" > > > > > > + > > > > > > +# Set use_stateless_nat to true > > > > > > +check ovn-nbctl --wait=sb set load_balancer lb1 > > > > > options:use_stateless_nat=true > > > > > > + > > > > > > +# Associate load balancer to s1 > > > > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > > > > +check ovn-nbctl --wait=sb sync > > > > > > + > > > > > > +ovn-sbctl dump-flows s1 > s1flows > > > > > > +AT_CAPTURE_FILE([s1flows]) > > > > > > + > > > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | > > > grep > > > > > "2001:db8:cccc::1"], [0], [dnl > > > > > > + table=??(ls_in_pre_stateful ), priority=120 , > match=(reg0[[2]] > > > == 1 > > > > > && ip6.dst == 2001:db8:cccc::1), action=(xxreg1 = 2001:db8:cccc::1; > > > > > ct_lb_mark;) > > > > > > +]) > > > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > > > > "2001:db8:cccc::1"], [0], [dnl > > > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new > && > > > > > ip6.dst == 2001:db8:cccc::1), > > > > > > > > > action=(ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > > > +]) > > > > > > + > > > > > > +# Associate load balancer to lr1 with DGP > > > > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > > > > +check ovn-nbctl --wait=sb sync > > > > > > + > > > > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > > > > +AT_CAPTURE_FILE([lr1flows]) > > > > > > + > > > > > > +# Check stateless NAT rules for load balancer with multiple DGP > > > > > > +# 1. Check if the backend IPs are in the ipX.dst action > > > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > > > > "2001:db8:cccc::1"], [0], [dnl > > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new > && > > > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > > > > is_chassis_resident("cr-lr1-ts1")), > > > > > > > > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new > && > > > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > > > > is_chassis_resident("cr-lr1-ts2")), > > > > > > > > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new > && > > > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > > > > is_chassis_resident("cr-lr1_public")), > > > > > > > > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > > > +]) > > > > > > + > > > > > > +# 2. Check if the DGP ports are in the match with action next > > > > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], > [0], > > > [dnl > > > > > > + table=??(lr_out_undnat ), priority=0 , match=(1), > > > > > action=(next;) > > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > > > 2001:db8:aaaa:3::102) || > > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || > outport > > > == > > > > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), > action=(next;) > > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > > > 2001:db8:aaaa:3::102) || > > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || > outport > > > == > > > > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), > action=(next;) > > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > > > 2001:db8:aaaa:3::102) || > > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || > > > outport == > > > > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > > > > > action=(next;) > > > > > > +]) > > > > > > + > > > > > > +# 3. Check if the VIP IP is in the ipX.src action > > > > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], > [dnl > > > > > > + table=??(lr_out_snat ), priority=0 , match=(1), > > > > > action=(next;) > > > > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > > > > > action=(next;) > > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > > > 2001:db8:aaaa:3::102) || > > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || > outport > > > == > > > > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), > > > > > action=(ip6.src=2001:db8:cccc::1; next;) > > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > > > 2001:db8:aaaa:3::102) || > > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || > outport > > > == > > > > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), > > > > > action=(ip6.src=2001:db8:cccc::1; next;) > > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > > > 2001:db8:aaaa:3::102) || > > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || > > > outport == > > > > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > > > > > action=(ip6.src=2001:db8:cccc::1; next;) > > > > > > +]) > > > > > > + > > > > > > +AT_CLEANUP > > > > > > +]) > > > > > > + > > > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (DGP)]) > > > > > > +ovn_start > > > > > > + > > > > > > +check ovn-nbctl ls-add public > > > > > > +check ovn-nbctl lr-add lr1 > > > > > > + > > > > > > +# lr1 DGP ts1 > > > > > > +check ovn-nbctl ls-add ts1 > > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > > > 172.16.10.1/24 > > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-1 > > > > > > + > > > > > > +# lr1 DGP ts2 > > > > > > +check ovn-nbctl ls-add ts2 > > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > > > 172.16.20.1/24 > > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-1 > > > > > > + > > > > > > +# lr1 DGP public > > > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > > > 173.16.0.1/16 > > > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 > 172.16.0.1/24 > > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > > > > + > > > > > > +check ovn-nbctl ls-add s1 > > > > > > +# s1 - lr1 > > > > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > > > 172.16.0.1" > > > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > > > > + > > > > > > +# s1 - backend vm1 > > > > > > +check ovn-nbctl lsp-add s1 vm1 > > > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > > > 172.16.0.101" > > > > > > + > > > > > > +# s1 - backend vm2 > > > > > > +check ovn-nbctl lsp-add s1 vm2 > > > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > > > 172.16.0.102" > > > > > > + > > > > > > +# s1 - backend vm3 > > > > > > +check ovn-nbctl lsp-add s1 vm3 > > > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > > > 172.16.0.103" > > > > > > + > > > > > > +# Add the lr1 DGP ts1 to the public switch > > > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 > router-port=lr1-ts1 > > > > > nat-addresses=router > > > > > > + > > > > > > +# Add the lr1 DGP ts2 to the public switch > > > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 > router-port=lr1-ts2 > > > > > nat-addresses=router > > > > > > + > > > > > > +# Add the lr1 DGP public to the public switch > > > > > > +check ovn-nbctl lsp-add public public_lr1 > > > > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > > > > +check ovn-nbctl lsp-set-options public_lr1 > router-port=lr1_public > > > > > nat-addresses=router > > > > > > + > > > > > > +# Create the Load Balancer lb1 > > > > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > > > > > "172.16.0.103,172.16.0.102,172.16.0.101" > > > > > > + > > > > > > +# Associate load balancer to s1 > > > > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > > > > +check ovn-nbctl --wait=sb sync > > > > > > + > > > > > > +ovn-sbctl dump-flows s1 > s1flows > > > > > > +AT_CAPTURE_FILE([s1flows]) > > > > > > + > > > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | > > > grep > > > > > "30.0.0.1"], [0], [dnl > > > > > > + table=??(ls_in_pre_stateful ), priority=120 , > match=(reg0[[2]] > > > == 1 > > > > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > > > > > +]) > > > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > > > > "30.0.0.1"], [0], [dnl > > > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new > && > > > > > ip4.dst == 30.0.0.1), > > > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > > +]) > > > > > > + > > > > > > +# Associate load balancer to lr1 with DGP > > > > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > > > > +check ovn-nbctl --wait=sb sync > > > > > > + > > > > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > > > > +AT_CAPTURE_FILE([lr1flows]) > > > > > > + > > > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > > > > "30.0.0.1"], [0], [dnl > > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new > && > > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > > is_chassis_resident("cr-lr1-ts1")), > > > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new > && > > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > > is_chassis_resident("cr-lr1-ts2")), > > > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new > && > > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > > is_chassis_resident("cr-lr1_public")), > > > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > > +]) > > > > > > + > > > > > > +AT_CLEANUP > > > > > > +]) > > > > > > -- > > > > > > 2.34.1 > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _'Esta mensagem é direcionada apenas para os endereços > constantes no > > > > > > cabeçalho inicial. Se você não está listado nos endereços > constantes > > > no > > > > > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo > > > dessa > > > > > > mensagem e cuja cópia, encaminhamento e/ou execução das ações > citadas > > > > > estão > > > > > > imediatamente anuladas e proibidas'._ > > > > > > > > > > > > > > > > > > * **'Apesar do Magazine Luiza tomar > > > > > > todas as precauções razoáveis para assegurar que nenhum vírus > esteja > > > > > > presente nesse e-mail, a empresa não poderá aceitar a > > > responsabilidade > > > > > por > > > > > > quaisquer perdas ou danos causados por esse e-mail ou por seus > > > anexos'.* > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > dev mailing list > > > > > > [email protected] > > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > _‘Esta mensagem é direcionada apenas para os endereços constantes no > > > > cabeçalho inicial. Se você não está listado nos endereços constantes > no > > > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo > dessa > > > > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas > > > estão > > > > imediatamente anuladas e proibidas’._ > > > > > > > > > > > > * **‘Apesar do Magazine Luiza tomar > > > > todas as precauções razoáveis para assegurar que nenhum vírus esteja > > > > presente nesse e-mail, a empresa não poderá aceitar a > responsabilidade > > > por > > > > quaisquer perdas ou danos causados por esse e-mail ou por seus > anexos’.* > > > > > > > > > > > > > > > > _______________________________________________ > > > > dev mailing list > > > > [email protected] > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > > > > -- > > > > > > > > > > _‘Esta mensagem é direcionada apenas para os endereços constantes no > > cabeçalho inicial. Se você não está listado nos endereços constantes no > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa > > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas > estão > > imediatamente anuladas e proibidas’._ > > > > > > * **‘Apesar do Magazine Luiza tomar > > todas as precauções razoáveis para assegurar que nenhum vírus esteja > > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade > por > > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.* > > > > > > > > _______________________________________________ > > dev mailing list > > [email protected] > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > -- _‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’._ * **‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.* _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
