On Mon, Oct 7, 2024 at 11:01 AM Roberto Bartzen Acosta <[email protected]> wrote: > > Thanks Numan! > > Em seg., 7 de out. de 2024 às 11:34, Numan Siddique <[email protected]> > escreveu: >> >> On Mon, Sep 30, 2024 at 1:57 PM Roberto Bartzen Acosta via dev >> <[email protected]> wrote: >> > >> > Hi Numan, >> > >> > Em sex., 27 de set. de 2024 às 12:47, Numan Siddique <[email protected]> >> > escreveu: >> > >> > > On Thu, Sep 26, 2024 at 10:55 AM Roberto Bartzen Acosta via dev >> > > <[email protected]> wrote: >> > > > >> > > > Hi Numan, >> > > > >> > > > Thanks for your feedback and review. >> > > > >> > > > Em qua., 25 de set. de 2024 às 19:50, Numan Siddique <[email protected]> >> > > > escreveu: >> > > > >> > > > > On Thu, Sep 19, 2024 at 6:12 PM Roberto Bartzen Acosta via dev >> > > > > <[email protected]> wrote: >> > > > > > >> > > > > > This commit fixes the build_distr_lrouter_nat_flows_for_lb function >> > > to >> > > > > > include a DNAT flow entry for each DGP in use. Since we have added >> > > > > support >> > > > > > to create multiple gateway ports per logical router, it's necessary >> > > to >> > > > > > include in the LR NAT rules pipeline a specific entry for each >> > > attached >> > > > > DGP. >> > > > > > Otherwise, the inbound traffic will only be redirected when the >> > > incoming >> > > > > LRP >> > > > > > matches the chassis_resident field. >> > > > > > >> > > > > > Additionally, this patch includes the ability to use load-balancer >> > > with >> > > > > DGPs >> > > > > > attached to multiple chassis. We can have each of the DGPs >> > > > > > associated >> > > > > with a >> > > > > > different chassis, and in this case the DNAT rules added by default >> > > will >> > > > > not >> > > > > > be enough to guarantee outgoing traffic. >> > > > > > >> > > > > > To solve the multiple chassis for DGPs problem, this patch include >> > > > > > a >> > > new >> > > > > > config options to be configured in the load-balancer. If the >> > > > > use_stateless_nat >> > > > > > is set to true, the logical router that references this >> > > > > > load-balancer >> > > > > will use >> > > > > > Stateless NAT rules when the logical router has multiple DGPs. >> > > > > > After >> > > > > applying >> > > > > > this patch and setting the use_stateless_nat option, the inbound >> > > and/or >> > > > > > outbound traffic can pass through any chassis where the DGP resides >> > > > > without >> > > > > > having problems with CT state. >> > > > > > >> > > > > > Reported-at: >> > > https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2054322 >> > > > > > Fixes: 15348b7b806f ("ovn-northd: Multiple distributed gateway port >> > > > > support.") >> > > > > > >> > > > > > Signed-off-by: Roberto Bartzen Acosta >> > > > > > <[email protected]> >> > > > > >> > > > > Hi Roberto, >> > > > > >> > > > > Thanks for the patch. I tested this patch using the test example in >> > > > > multinode.at. >> > > > > >> > > > > The test case adds the below load balancer >> > > > > >> > > > > [root@ovn-central ~]# ovn-nbctl lb-list >> > > > > UUID LB PROTO >> > > > > VIP IPs >> > > > > f3e29869-3bb5-4df0-960a-171106f5913a lb0 tcp >> > > > > 172.16.0.100:9000 10.0.0.3:80,10.0.0.4:80 >> > > > > >> > > > > And the below logical flows are generated by this patch >> > > > > >> > > > > -------- >> > > > > [root@ovn-central ~]# ovn-sbctl dump-flows lr0 | grep 172.16.0.100 >> > > > > table=6 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst >> > > > > == 172.16.0.100), action=(ct_dnat;) >> > > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && >> > > > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 >> > > > > && >> > > > > is_chassis_resident("cr-lr0-public-p1")), >> > > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= >> > > 10.0.0.3:80, >> > > > > 10.0.0.4:80);) >> > > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && >> > > > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 >> > > > > && >> > > > > is_chassis_resident("cr-lr0-public-p2")), >> > > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= >> > > 10.0.0.3:80, >> > > > > 10.0.0.4:80);) >> > > > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && >> > > > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && >> > > > > tcp.src == 80)) && (inport == "lr0-public-p1" || outport == >> > > > > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1") && tcp), >> > > > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) >> > > > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && >> > > > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && >> > > > > tcp.src == 80)) && (inport == "lr0-public-p2" || outport == >> > > > > "lr0-public-p2") && is_chassis_resident("cr-lr0-public-p2") && tcp), >> > > > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) >> > > > > -------------- >> > > > > >> > > > > >> > > > > I fail to understand the reason for modifying the ip4.dst before >> > > > > calling ct_lb_mark. Can you please explain why ? Because the >> > > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;) will first modify the >> > > > > ip4.dst to 10.0.0.3 and >> > > > > then to 10.0.0.4 and then the ct_lb_mark will actually do the >> > > > > conntrack with NAT to either 10.0.0.3 or 10.0.0.4. >> > > > > >> > > > > Is it because you want the conntrack entry to not have 172.16.0.100 ? >> > > > > >> > > > >> > > >> > > > The only reason I included this ip4.dst action in the DNAT rule is >> > > because >> > > > it's required to accept packets coming from a chassis that doesn't have >> > > > previously created conntrack entries. The main feature introduced in >> > > > this >> > > > patch is to allow the administrator to have multiple DGPs attached to >> > > > different chassis (is_chassis_resident...). So, my implementation was >> > > based >> > > > on the normal behavior when using stateless NAT for external addresses, >> > > > where we need to add the ipx.dst in lr_in_dnat for traffic to be >> > > > received >> > > > on the chassis (put the DGP port as chassis_resident match, as is the >> > > case >> > > > with stateless NAT [1] with DGP[2]). >> > > > >> > > > The question is, if we only have the ct_lb_mark, packets that pass >> > > through >> > > > the chassis and are already part of an active flow in another chassis >> > > (same >> > > > IPs and Ports) will be dropped because there is no correspondence in >> > > > the >> > > > backend. So only packets with the NEW flag will be accepted and sent to >> > > the >> > > > backend (at least for TCP traffic). If we only have the ip4.dst action, >> > > > this will always perform the dnat for the same backend, without >> > > balancing. >> > > > Therefore, the combination of the two actions allows the packet to >> > > > always >> > > > be received (regardless of whether conntrack is active for it), and >> > > > ct_lb_mark will take care of balancing for different backends. >> > > > >> > > > If we had conntrack sync between different chassis this would not be >> > > > necessary, as the ct_lb_mark action could always be executed without >> > > > dropping packets due to lack of correspondence in the conntrack table. >> > > > >> > > > [1] >> > > > >> > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15737 >> > > > [2] >> > > > >> > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15726 >> > > > >> > > > >> > > >> > > I'm sorry, but it's not 100% clear to me. I know that you've already >> > > explained to Mark in the older version of this patch. >> > > Can you please explain with an example ? >> > > >> > > Let's take the below topology you've added in the mutlinode test as >> > > example >> > > >> > > # Network topology >> > > # >> > > # publicp1 (ovn-chassis-3) (20.0.0.3/24) >> > > # | >> > > # overlay >> > > # | >> > > # DGP public1 (ovn-gw-1) (20.0.0.1/24) >> > > # | >> > > # | >> > > # | >> > > # lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) 10.0.0.3/24 >> > > # | | >> > > # | + --- sw0p2 (ovn-chassis-2) 10.0.0.4/24 >> > > # | >> > > # DGP public2 (ovn-gw-2) (30.0.0.1/24) >> > > # | >> > > # overlay >> > > # | >> > > # publicp2 (ovn-chassis-4) (30.0.0.3/24) >> > > >> > > >> > > load balancer is configured on lr0 -> ovn-nbctl lb-add lb0 >> > > "172.16.0.100:80" "10.0.0.3:80,10.0.0.4:80" >> > > and it is attached to both lr0 and sw0. >> > > >> > > Scenario 1: >> > > >> > > publicp1 with IP 20.0.0.3 sends TCP traffic to VIP 172.16.0.100. I >> > > think this is what will happen >> > > >> > > - The packet (ip4.src = 20.0.0.3 , ip4.dst = 172.16.0.100, tcp.dst >> > > = 80) from ovn-chassis-3 will be sent out via the localnet bridge >> > > (br-ex) >> > > and the packet is received on ovn-gw-1 (as DGP public1 is resident >> > > on it) via the localnet bridge and the >> > > packet first enters public logical switch pipeline and then the >> > > lr0 router pipeline. >> > > >> > > - In the router's lr_in_dnat state, the packet will be load >> > > balanced to one of the backends using ct_lb_mark. Lets say 10.0.0.3 >> > > is chosen >> > > - The packet from router pipeline lr0 enters sw0 switch pipeline and >> > > then the packet is tunneled to ovn-chassis-1 and delivered to sw0p1. >> > > - The reply packet (ip4.src = 10.0.0.3, ip4.dst = 20.0.0.3, tcp.src = >> > > 80) will enter the router pipeline and since 20.0.0.0/24 is handled by >> > > DGP public1, >> > > the packet is tunnelled to ovn-gw-1. >> > > - In the router pipeline of ovn-gw-1, the packet is undnatted from >> > > ip4.src 10.0.0.3 to 172.16.0.100 and the packet is sent out via the >> > > localnet bridge. >> > > - ovn-chassis-3 receives the packet via the localnet bridge and into >> > > br-int and finally to publicp1. >> > > >> > > >> > > In this scenario the load balancing is handled and conntrack entries >> > > are created in ovn-gw-1. And there is no need to add flows in >> > > "lr_out_snat" for stateless NAT >> > > or set ip4.dst to one or all of backend IPs before "ct_lb_mark" in >> > > lr_in_dnat stage. >> > > >> > > Scenario 2: >> > > publicp2 with IP 30.0.0.3 sends TCP traffic to VIP 172.16.0.100. >> > > >> > > This is similar to scenario 1. Except that load balancing happens in >> > > ovn-gw-2. >> > > since DGP public2 is on this chassis. >> > > >> > > >> > > Scenario 3: >> > > >> > > An external entity with IP 20.0.0.50 sends TCP traffic to VIP >> > > 172.16.0.100. >> > > >> > > - This scenario is similar to the first one. The packet from this >> > > external entity is received on ovn-gw-1 via the localnet bridge. >> > > Rest all is the same. >> > > >> > > Scenario 4: >> > > >> > > sw0p2 on ovn-chassis-2 sends TCP traffic to VIP 172.16.0.100. >> > > >> > > - Since sw0 is attached with load balancer lb0, load balancing >> > > happens in the source chassis - ovn-chassis-2 itself and depending on >> > > the backend chosen, >> > > the packet is tunnelled to ovn-chassis-1 (if 10.0.0.3 is chosen) >> > > or delivered directly to sw0p2 (if i0.0.0.4 is chosen). >> > > >> > > >> > > Scenario 5: >> > > >> > > An external entity with IP 40.0.0.40 sends TCP traffic to VIP >> > > 172.16.0.100 and there are 2 ECMP routes configured >> > > 172.16.0.100 via 20.0.0.1 >> > > 172.16.0.100 via 30.0.0.1 >> > > >> > > In this case if the packet uses the route via 20.0.0.1 it will be >> > > received on ovn-gw-1 via the localnet bridge br-ex. >> > > >> > > - With your patch, ip4.dst is modified to the last backend in the >> > > list and then ct_lb_mark chosen as one of the backends >> > > - And then it is tunnelled to the destination chassis. >> > > >> > > >> > > Is this the scenario 5 you're trying to address with the stateless >> > > NAT ? How would the reply packet work ? >> > > The reply packet from backend 10.0.0.3 can use either of the paths ? >> > > i.e ovn-gw-1 or ovn-gw-2 ? >> > > >> > >> > >> > Yes, this scenario would be the main goal of this patch. >> > Let me explain with an example related to the OVN interconnect (one of the >> > use cases of this patch). In the context of ovn-ic the backend 10.0.0.3 >> > could reply packets using either of the paths (ovn-gw-1 or ovn-gw-2). >> > >> > >> > >> > LB VIP 172.16.0.100 >> > >> > | >> > vm (40.0.0.40) - LR-EXT - lrp1 - chassis1 - Transit switch TS1 - ovn-gw-1 >> > - lr0-public-p1 - lr0 - sw0 - VM 10.0.0.3 (ovn-chassis-1) >> > - lrp2 - chassis2 - Transit switch >> > TS2 - ovn-gw-2 - lr0-public-p2 - - VM 10.0.0.4 >> > (ovn-chassis-2) >> > >> > >> > Let's take an example of the ovn-ic in the above topology: >> > >> > Assumptions: >> > A) logical router LR-EXT: route to 172.16.0.100/32 via ECMP (lrp1/chassis1 >> > and lrp2/chassis2) >> > B) logical router lr0: route to 40.0.0.0/24 via ECMP >> > (lr0-public-p1/ovn-gw-1 and lr0-public-p2/ovn-gw-2) >> > >> > >> > 1 - VM 40.0.0.40 send a request to the LB 172.16.0.100:80 >> > 2 - LR-EXT chooses one of the possible outgoing routes via ECMP - e.g. lrp1 >> > / chassis1 / TS1 >> > 3 - The lr0 receives the ingoing traffic through the port lr0-public-p1 / >> > ovn-gw-1 because the TS1 is used to start the traffic >> > 4 - with this patch we added lr_in_dnat rules for all DGPs (with or without >> > the stateless NAT config flag) >> > >> > with the stateless NAT config flag True: >> > ...is_chassis_resident("cr-lr0-public-p1")), >> > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80 >> > ,10.0.0.4:80);) >> > ...is_chassis_resident("cr-lr0-public-p2")), >> > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80 >> > ,10.0.0.4:80);) >> > 5 - The ip4.dst will perform the action cumulatively, basically the last >> > ip4.dst action applied will be to change dst to 10.0.0.4. >> > 5.1 - The ct_lb_mark will check the conntrack and validate if the traffic >> > refers to a new one (TCP SYNC), and then, it executes the final action >> > provided by ct_lb_mark and select one of the available backends. e.g >> > 10.0.0.3 >> > >> > without the stateless NAT config flag: >> > ...is_chassis_resident("cr-lr0-public-p1")), >> > action=(ct_lb_mark(backends= >> > 10.0.0.3:80,10.0.0.4:80);) >> > ...is_chassis_resident("cr-lr0-public-p2")), >> > action=(ct_lb_mark(backends= >> > 10.0.0.3:80,10.0.0.4:80);) >> > >> > 5 - same as the step 5.1 (ct_lb_mark check only). >> > >> > 6 - VM 10.0.0.3 receives the new traffic and sends a reply (TCP SYN+ACK) >> > >> > 7 - lr0 receives the response packet (from VM 10.0.0.3) in the routing >> > pipeline and route via one of the available paths (ECMP). >> > For example: lr0-public-p2 / ovn-gw-2 / TS2 >> > So, we have a different outgoing path of the original one. >> > >> > 8 - ovn-gw-2 receives (SYN+ACK). We have no conntrack entries on ovn-gw-2 >> > to match and perform the LB SNAT action! The ct_lb_mark previously creates >> > the conntrack entry on chassis ovn-gw-1 (when receives TCP SYN). >> > >> > with the stateless NAT config flag True: >> > The outgoing packet fail to match the lb conntrack but the packet is >> > SNATed by the lr_out_snat rule with a priority lower then cl_lb_mark, >> > executing after cl_lb_mark in the pipeline. >> > >> > 9 - The SYN+ACK packet crosses the TS2 and is delivered to VM 40.0.0.40 via >> > lrp2 (SRC = 172.16.0.100:80) >> > >> > lr_out_snat will match and SNATed: >> > match=(ip4 && ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == >> > 10.0.0.4 && tcp.src == 80)) && (inport == "lr0-public-p1" || outport == >> > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1") && tcp), >> > action=(ip4.src=172.16.0.100; tcp.src=80; next;) >> > >> > without the stateless NAT config flag: >> > The outgoing will be returned to the VM 40.0.0.4 with the LR SNAT for >> > public-p2 port as ip4.src! So, basically this will break the TCP >> > handshake!!! >> > >> > 9 - The SYN+ACK packet cross the TS2 and is delivered to the VM 40.0.0.40 >> > via lrp2 (SRC = 30.0.0.1:80) >> > >> > Moving forward on the happy path (stateless NAT is true): >> > >> > 10 - VM 40.0.0.40 sends an ACK to complete the TCP handshake. >> > >> > 11 - 2 - LR-EXT chooses one of the possible outgoing routes via ECMP >> > Let's assume that LR-EXT correctly execute the ecmp algorithm (per flow >> > basis) and forwards to the same initial path because we're using the same >> > flow (src/dst IPs and src/dts TCP ports still the same). >> > e.g. still using the lrp1 / chassis1 / TS1 >> > >> > 12 - The lr0 receive the ingoing traffic through the port lr0-public-p1 / >> > ovn-gw-1 / TS1 >> > >> > With this patch we have 2 possible behaviours: >> > >> > 12.1 - Without Stateless NAT config flag >> > >> > table=1 (lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src == >> > 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && tcp.src == 80)) && >> > (inport == "lr0-public-p1" || outport == >> > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1")), >> > action=(ct_dnat_in_czone;) >> > >> > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && !ct.rel >> > && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && >> > is_chassis_resident("cr-lr0-public-p1")), action=(ct_lb_mark(backends= >> > 10.0.0.3:80,10.0.0.4:80);) >> > >> > openflow: br-int (ovn-gw-1) >> > cookie=0x9381c20e, duration=2361.675s, table=15, n_packets=2, n_bytes=148, >> > idle_age=1564, >> > priority=120,ct_state=+new-rel+trk,tcp,metadata=0x4,nw_dst=172.16.0.100,tp_dst=80 >> > actions=group:3 >> > >> > ovs-appctl dpctl/dump-conntrack (ovn-gw-1) >> > tcp,orig=(src=40.0.0.40,dst=172.16.0.100,sport=35274,dport=80),reply=(src=10.0.0.3,dst=40.0.0.40,sport=80,dport=35274),zone=8,mark=2,protoinfo=(state=TIME_WAIT) >> > >> > At this point, the ACK packet will be discarded because we didn't perform >> > the SYN+ACK return on this chassis (ovn-gw -1). So, without the previously >> > established conntrack to match the ACK packet we broke the TCP handshake. >> > >> > Remember the flow: >> > -> SYN: ovn-gw-1 >> > <- SYN+ACK: ovn-gw-2 >> > -> ACK: ovn-gw-1 >> > >> > 12.2 - With stateless NAT config flag is True >> > >> > table=1 (lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src == >> > 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && tcp.src == 80)) && >> > (inport == "lr0-public-p1" || outport == >> > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1")), >> > action=(next;) >> > >> > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && !ct.rel >> > && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && >> > is_chassis_resident("cr-lr0-public-p1")), >> > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80 >> > ,10.0.0.4:80);) >> > >> > openflow: br-int (ovn-gw-1) >> > cookie=0xab99f240, duration=1086.022s, table=15, n_packets=610, >> > n_bytes=48979, idle_age=825, >> > priority=120,ct_state=+new-rel+trk,tcp,metadata=0x2,nw_dst=172.16.0.100,tp_dst=80 >> > actions=mod_nw_dst:10.0.0.3,mod_nw_dst:10.0.0.4,group:1 >> > >> > ovs-appctl dpctl/dump-conntrack (ovn-gw-1) >> > tcp,orig=(src=40.0.0.40,dst=10.0.0.3,sport=54632,dport=80),reply=(src=10.0.0.3,dst=40.0.0.40,sport=80,dport=54632),zone=7,mark=2,protoinfo=(state=SYN_SENT) >> > >> > This is the reason to use stateless NAT rules. Unlike the case where we >> > don't have the SYN_SENT, now the ACK packet will be accepted and forwarded >> > after performing the mod_nw_dst action. >> > >> > How does this work? We already have the previous flow created by the first >> > packet (SYN), so regardless of the IP.dst modified in the action, the >> > packet will be forwarded to the same backend that corresponds to the >> > conntrack match that is in SYN_SENT state. That's why I created the action >> > with the two modifiers (ip4.dst + ct_lb_mark). Using only the ct_lb_mark >> > action creates a strong dependency on the connection state match in the >> > conntrack entry (which we don't have in these cases). However, only using >> > ip4.dst doesn't create traffic balancing as it will always send to the same >> > backend (last in the ip4.dst). >> > >> > >> > I hope this has helped clarify the design decisions when creating/modifying >> > flows (lr_in_dnat/lr_out_snat/lr_out_undnat). >> > >> > >> >> Thanks for the detailed explanation. Its clear to me now. >> >> I'd suggest the following >> >> 1. In lr_in_dnat stage, choose either the first backend or the last >> backend to modify the ip4.dst before ct_lb_mark. >> I've no strong preference. Either would do. >> >> 2. Please document or add comments in northd.c (and in ovn-nb.xml)so >> that we don't lose the context later. >> >> 3. Please run the ovn-fake-multinode tests in your github CI repo and >> make sure it passes. I think you can trigger the test in your repo >> in the Actions tab. >> >> 4. In the setup you described above, is it possible to add the >> (below) route in the vm (40.0.0.40) deployment so that 40.0.0.0.40 >> can >> talk directly to 10.0.0.3 and 10.0.0.4 instead of the LB VIP and >> send the tcp traffic to 10.0.0.3 and see if it works ? >> If not, then please document the same i.e with the option >> stateless_nat in the LB, the backends cannot be directly >> communicated. >> >> logical router LR-EXT: route to 10.0.0.0/24 via ECMP (lrp1/chassis1 >> and lrp2/chassis2) >> > > I'll do your suggestions. Just a quick question: how can I > customize/parametrize the ovn-fake-multinode elements to run on CI? I mean to > enable 2 more ovn-chassis needed to run the related tests. I've enabled it > locally in the ./ovn_cluster.sh file but how do I change this for the > fake-multinode CI actions? Is it per job or is this global?
Its global. Take a look here - https://github.com/ovn-org/ovn/blob/main/.github/workflows/ovn-fake-multinode-tests.yml#L152 You can customize here to enbable 2 additional chassis. > > Regarding your suggestion 4, I imagine it makes more sense to > integrate/create a new test with the ovn-interconnect setup + LB + multiple > DGPs, which allows us to have multiple paths on routers on both sides > (addressing the case discussed above) Sounds good to me. Numan > > Thanks, > Roberto > >> >> Thanks >> Numan >> >> >> > >> > >> > > >> > > Thanks >> > > Numan >> > > >> > > >> > > > > >> > > > > Also I don't understand why this patch adds the logical flows in >> > > > > "lr_out_snat" stage ? >> > > > > >> > > > >> > > > The flow for lr_out_snat is necessary for the correct functioning of >> > > > stateless NAT for the same reason explained previously. I mean, if the >> > > > outgoing packet is redirected to a chassis that doesn't have an active >> > > > conntrack entry, it will not be NATed by ct_lb action because it >> > > > doesn't >> > > > refer to a valid flow (use case with ecmp). >> > > > >> > > > So it is necessary to create a stateless SNAT rule (similar to this >> > > > [3]) >> > > > with a lower priority than the other router pipeline entries, in this >> > > case, >> > > > if the packet is not SNATed by ct_lb (conntrack missed) it will be >> > > > SNATed >> > > > by stateless NAT rule. >> > > > >> > > > [3] >> > > > >> > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15884 >> > > > >> > > > >> > > > >> > > > > >> > > > > Using the system multinode test as an example, the below fails >> > > > > (which is a regression) >> > > > > >> > > > > --- >> > > > > root@ovn-chassis-3 ~]# ip netns exec publicp1 nc -vz 10.0.0.3 80 >> > > > > ---- >> > > > > >> > > > > In the above test, publicp1 with IP 20.0.0.3 when it tries to >> > > > > connect >> > > > > to one if the backends directly (without the LB VIP), it fails. >> > > > > It fails because of the logical flows in "lr_out_snat". >> > > > > >> > > > > >> > > > > Looks to me the solution proposed here is incomplete. >> > > > > >> > > > > Also please note that in our CI we run the multinode tests >> > > > > periodically once a day using the v0.1 of the ovn-fake-multinode >> > > > > and the tests you added will fail. This needs to be fixed and until >> > > > > we move to the latest version of ovn-fake-multinode. >> > > > > >> > > > >> > > > I imagine that the test you are doing is using the same port as the LB >> > > > backend (TCP 80 in this case). So, the stateless lr_out_snat flow will >> > > > force the output to be SNATed because this port is in use by the >> > > > backend. >> > > > Traffic to/from other ports will work without problems and will follow >> > > the >> > > > normal programmed flows (e.g. ICMP). >> > > > >> > > > This is necessary to ensure the egress traffic because the DGPs are >> > > > distributed across multiple chassis. Also, this setup is being >> > > > validated >> > > in >> > > > the test ovn-fake-multinode testcase (ICMP from the backends chassis >> > > > use >> > > > the router's default SNAT and not the LB's). I didn't understand the >> > > > regression you mentioned because this was programmed to be stateless >> > > > and >> > > > it's traffic that uses the same ports as the LB backend, could you >> > > explain >> > > > better? >> > > > >> > > > Thanks, >> > > > Roberto >> > > > >> > > > >> > > > > Thanks >> > > > > Numan >> > > > > >> > > > > >> > > > > > --- >> > > > > > northd/en-lr-stateful.c | 12 - >> > > > > > northd/northd.c | 116 ++++++-- >> > > > > > ovn-nb.xml | 10 + >> > > > > > tests/multinode-macros.at | 40 +++ >> > > > > > tests/multinode.at | 556 >> > > ++++++++++++++++++++++++++++++++++++++ >> > > > > > tests/ovn-northd.at | 320 ++++++++++++++++++++++ >> > > > > > 6 files changed, 1017 insertions(+), 37 deletions(-) >> > > > > > >> > > > > > diff --git a/northd/en-lr-stateful.c b/northd/en-lr-stateful.c >> > > > > > index baf1bd2f8..f09691af6 100644 >> > > > > > --- a/northd/en-lr-stateful.c >> > > > > > +++ b/northd/en-lr-stateful.c >> > > > > > @@ -516,18 +516,6 @@ lr_stateful_record_create(struct >> > > lr_stateful_table >> > > > > *table, >> > > > > > >> > > > > > table->array[od->index] = lr_stateful_rec; >> > > > > > >> > > > > > - /* Load balancers are not supported (yet) if a logical router >> > > has >> > > > > multiple >> > > > > > - * distributed gateway port. Log a warning. */ >> > > > > > - if (lr_stateful_rec->has_lb_vip && >> > > lr_has_multiple_gw_ports(od)) { >> > > > > > - static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, >> > > 1); >> > > > > > - VLOG_WARN_RL(&rl, "Load-balancers are configured on >> > > > > > logical >> > > " >> > > > > > - "router %s, which has %"PRIuSIZE" >> > > > > > distributed " >> > > > > > - "gateway ports. Load-balancer is not >> > > > > > supported >> > > " >> > > > > > - "yet when there is more than one distributed >> > > > > > " >> > > > > > - "gateway port on the router.", >> > > > > > - od->nbr->name, od->n_l3dgw_ports); >> > > > > > - } >> > > > > > - >> > > > > > return lr_stateful_rec; >> > > > > > } >> > > > > > >> > > > > > diff --git a/northd/northd.c b/northd/northd.c >> > > > > > index a267cd5f8..bbe97acf8 100644 >> > > > > > --- a/northd/northd.c >> > > > > > +++ b/northd/northd.c >> > > > > > @@ -11807,31 +11807,30 @@ static void >> > > > > > build_distr_lrouter_nat_flows_for_lb(struct >> > > > > > lrouter_nat_lb_flows_ctx >> > > > > *ctx, >> > > > > > enum lrouter_nat_lb_flow_type >> > > type, >> > > > > > struct ovn_datapath *od, >> > > > > > - struct lflow_ref *lflow_ref) >> > > > > > + struct lflow_ref *lflow_ref, >> > > > > > + struct ovn_port *dgp, >> > > > > > + bool stateless_nat) >> > > > > > { >> > > > > > - struct ovn_port *dgp = od->l3dgw_ports[0]; >> > > > > > - >> > > > > > - const char *undnat_action; >> > > > > > - >> > > > > > - switch (type) { >> > > > > > - case LROUTER_NAT_LB_FLOW_FORCE_SNAT: >> > > > > > - undnat_action = "flags.force_snat_for_lb = 1; next;"; >> > > > > > - break; >> > > > > > - case LROUTER_NAT_LB_FLOW_SKIP_SNAT: >> > > > > > - undnat_action = "flags.skip_snat_for_lb = 1; next;"; >> > > > > > - break; >> > > > > > - case LROUTER_NAT_LB_FLOW_NORMAL: >> > > > > > - case LROUTER_NAT_LB_FLOW_MAX: >> > > > > > - undnat_action = lrouter_use_common_zone(od) >> > > > > > - ? "ct_dnat_in_czone;" >> > > > > > - : "ct_dnat;"; >> > > > > > - break; >> > > > > > - } >> > > > > > + struct ds dnat_action = DS_EMPTY_INITIALIZER; >> > > > > > >> > > > > > /* Store the match lengths, so we can reuse the ds buffer. */ >> > > > > > size_t new_match_len = ctx->new_match->length; >> > > > > > size_t undnat_match_len = ctx->undnat_match->length; >> > > > > > >> > > > > > + /* dnat_action: Add the LB backend IPs as a destination action >> > > of >> > > > > the >> > > > > > + * lr_in_dnat NAT rule with cumulative effect >> > > because >> > > > > any >> > > > > > + * backend dst IP used in the action list will >> > > > > redirect the >> > > > > > + * packet to the ct_lb pipeline. >> > > > > > + */ >> > > > > > + if (stateless_nat) { >> > > > > > + for (size_t i = 0; i < ctx->lb_vip->n_backends; i++) { >> > > > > > + struct ovn_lb_backend *backend = >> > > &ctx->lb_vip->backends[i]; >> > > > > > + bool ipv6 = !IN6_IS_ADDR_V4MAPPED(&backend->ip); >> > > > > > + ds_put_format(&dnat_action, "%s.dst=%s;", ipv6 ? >> > > > > > "ip6" : >> > > > > "ip4", >> > > > > > + backend->ip_str); >> > > > > > + } >> > > > > > + } >> > > > > > + ds_put_format(&dnat_action, "%s", ctx->new_action[type]); >> > > > > > >> > > > > > const char *meter = NULL; >> > > > > > >> > > > > > @@ -11841,20 +11840,46 @@ >> > > > > > build_distr_lrouter_nat_flows_for_lb(struct >> > > > > lrouter_nat_lb_flows_ctx *ctx, >> > > > > > >> > > > > > if (ctx->lb_vip->n_backends || >> > > > > > !ctx->lb_vip->empty_backend_rej) >> > > { >> > > > > > ds_put_format(ctx->new_match, " && >> > > > > > is_chassis_resident(%s)", >> > > > > > - od->l3dgw_ports[0]->cr_port->json_key); >> > > > > > + dgp->cr_port->json_key); >> > > > > > } >> > > > > > >> > > > > > ovn_lflow_add_with_hint__(ctx->lflows, od, S_ROUTER_IN_DNAT, >> > > > > ctx->prio, >> > > > > > - ds_cstr(ctx->new_match), >> > > > > ctx->new_action[type], >> > > > > > + ds_cstr(ctx->new_match), >> > > > > ds_cstr(&dnat_action), >> > > > > > NULL, meter, &ctx->lb->nlb->header_, >> > > > > > lflow_ref); >> > > > > > >> > > > > > ds_truncate(ctx->new_match, new_match_len); >> > > > > > >> > > > > > + ds_destroy(&dnat_action); >> > > > > > if (!ctx->lb_vip->n_backends) { >> > > > > > return; >> > > > > > } >> > > > > > >> > > > > > + struct ds undnat_action = DS_EMPTY_INITIALIZER; >> > > > > > + struct ds snat_action = DS_EMPTY_INITIALIZER; >> > > > > > + >> > > > > > + switch (type) { >> > > > > > + case LROUTER_NAT_LB_FLOW_FORCE_SNAT: >> > > > > > + ds_put_format(&undnat_action, "flags.force_snat_for_lb = >> > > > > > 1; >> > > > > next;"); >> > > > > > + break; >> > > > > > + case LROUTER_NAT_LB_FLOW_SKIP_SNAT: >> > > > > > + ds_put_format(&undnat_action, "flags.skip_snat_for_lb = 1; >> > > > > next;"); >> > > > > > + break; >> > > > > > + case LROUTER_NAT_LB_FLOW_NORMAL: >> > > > > > + case LROUTER_NAT_LB_FLOW_MAX: >> > > > > > + ds_put_format(&undnat_action, "%s", >> > > > > > + lrouter_use_common_zone(od) ? >> > > "ct_dnat_in_czone;" >> > > > > > + : "ct_dnat;"); >> > > > > > + break; >> > > > > > + } >> > > > > > + >> > > > > > + /* undnat_action: Remove the ct action from the lr_out_undenat >> > > NAT >> > > > > rule. >> > > > > > + */ >> > > > > > + if (stateless_nat) { >> > > > > > + ds_clear(&undnat_action); >> > > > > > + ds_put_format(&undnat_action, "next;"); >> > > > > > + } >> > > > > > + >> > > > > > /* We need to centralize the LB traffic to properly perform >> > > > > > * the undnat stage. >> > > > > > */ >> > > > > > @@ -11873,11 +11898,41 @@ >> > > > > > build_distr_lrouter_nat_flows_for_lb(struct >> > > > > lrouter_nat_lb_flows_ctx *ctx, >> > > > > > ds_put_format(ctx->undnat_match, ") && (inport == %s || >> > > > > > outport >> > > == >> > > > > %s)" >> > > > > > " && is_chassis_resident(%s)", dgp->json_key, >> > > > > dgp->json_key, >> > > > > > dgp->cr_port->json_key); >> > > > > > + /* Use the LB protocol as matching criteria for out undnat and >> > > snat >> > > > > when >> > > > > > + * creating LBs with stateless NAT. */ >> > > > > > + if (stateless_nat) { >> > > > > > + ds_put_format(ctx->undnat_match, " && %s", >> > > > > > ctx->lb->proto); >> > > > > > + } >> > > > > > ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_UNDNAT, >> > > 120, >> > > > > > - ds_cstr(ctx->undnat_match), >> > > undnat_action, >> > > > > > - &ctx->lb->nlb->header_, >> > > > > > + ds_cstr(ctx->undnat_match), >> > > > > > + ds_cstr(&undnat_action), >> > > > > &ctx->lb->nlb->header_, >> > > > > > lflow_ref); >> > > > > > + >> > > > > > + /* snat_action: Add a new lr_out_snat rule with the LB VIP as >> > > > > source IP >> > > > > > + * action to perform the NAT stateless pipeline >> > > > > completely. >> > > > > > + */ >> > > > > > + if (stateless_nat) { >> > > > > > + if (ctx->lb_vip->port_str) { >> > > > > > + ds_put_format(&snat_action, "%s.src=%s; %s.src=%s; >> > > next;", >> > > > > > + ctx->lb_vip->address_family == AF_INET6 >> > > > > > ? >> > > > > > + "ip6" : "ip4", >> > > > > > + ctx->lb_vip->vip_str, ctx->lb->proto, >> > > > > > + ctx->lb_vip->port_str); >> > > > > > + } else { >> > > > > > + ds_put_format(&snat_action, "%s.src=%s; next;", >> > > > > > + ctx->lb_vip->address_family == AF_INET6 >> > > > > > ? >> > > > > > + "ip6" : "ip4", >> > > > > > + ctx->lb_vip->vip_str); >> > > > > > + } >> > > > > > + ovn_lflow_add_with_hint(ctx->lflows, od, >> > > > > > S_ROUTER_OUT_SNAT, >> > > 160, >> > > > > > + ds_cstr(ctx->undnat_match), >> > > > > > + ds_cstr(&snat_action), >> > > > > &ctx->lb->nlb->header_, >> > > > > > + lflow_ref); >> > > > > > + } >> > > > > > + >> > > > > > ds_truncate(ctx->undnat_match, undnat_match_len); >> > > > > > + ds_destroy(&undnat_action); >> > > > > > + ds_destroy(&snat_action); >> > > > > > } >> > > > > > >> > > > > > static void >> > > > > > @@ -12022,6 +12077,8 @@ build_lrouter_nat_flows_for_lb( >> > > > > > * lflow generation for them. >> > > > > > */ >> > > > > > size_t index; >> > > > > > + bool use_stateless_nat = smap_get_bool(&lb->nlb->options, >> > > > > > + "use_stateless_nat", >> > > false); >> > > > > > BITMAP_FOR_EACH_1 (index, bitmap_len, lb_dps->nb_lr_map) { >> > > > > > struct ovn_datapath *od = lr_datapaths->array[index]; >> > > > > > enum lrouter_nat_lb_flow_type type; >> > > > > > @@ -12043,8 +12100,17 @@ build_lrouter_nat_flows_for_lb( >> > > > > > if (!od->n_l3dgw_ports) { >> > > > > > bitmap_set1(gw_dp_bitmap[type], index); >> > > > > > } else { >> > > > > > - build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, >> > > > > > - >> > > > > > lb_dps->lflow_ref); >> > > > > > + /* Create stateless LB NAT rules when using multiple >> > > DGPs >> > > > > and >> > > > > > + * use_stateless_nat is true. >> > > > > > + */ >> > > > > > + bool stateless_nat = (od->n_l3dgw_ports > 1) >> > > > > > + ? use_stateless_nat : false; >> > > > > > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) { >> > > > > > + struct ovn_port *dgp = od->l3dgw_ports[i]; >> > > > > > + build_distr_lrouter_nat_flows_for_lb(&ctx, type, >> > > > > > od, >> > > > > > + >> > > lb_dps->lflow_ref, >> > > > > dgp, >> > > > > > + >> > > > > > stateless_nat); >> > > > > > + } >> > > > > > } >> > > > > > >> > > > > > if (lb->affinity_timeout) { >> > > > > > diff --git a/ovn-nb.xml b/ovn-nb.xml >> > > > > > index 2836f58f5..ad03c6214 100644 >> > > > > > --- a/ovn-nb.xml >> > > > > > +++ b/ovn-nb.xml >> > > > > > @@ -2302,6 +2302,16 @@ or >> > > > > > local anymore by the ovn-controller. This option is set to >> > > > > > <code>false</code> by default. >> > > > > > </column> >> > > > > > + >> > > > > > + <column name="options" key="use_stateless_nat" >> > > > > > + type='{"type": "boolean"}'> >> > > > > > + If the load balancer is configured with >> > > > > <code>use_stateless_nat</code> >> > > > > > + option to <code>true</code>, the logical router that >> > > references >> > > > > this >> > > > > > + load balancer will use Stateless NAT rules when the >> > > > > > logical >> > > > > router >> > > > > > + has multiple distributed gateway ports(DGP). Otherwise, >> > > > > > the >> > > > > outbound >> > > > > > + traffic may be dropped in scenarios where we have >> > > > > > different >> > > > > chassis >> > > > > > + for each DGP. This option is set to <code>false</code> by >> > > > > default. >> > > > > > + </column> >> > > > > > </group> >> > > > > > </table> >> > > > > > >> > > > > > diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at >> > > > > > index 757917626..2f69433fc 100644 >> > > > > > --- a/tests/multinode-macros.at >> > > > > > +++ b/tests/multinode-macros.at >> > > > > > @@ -40,6 +40,27 @@ m4_define([M_START_TCPDUMP], >> > > > > > ] >> > > > > > ) >> > > > > > >> > > > > > +# M_EXEC([fake_node], [command]) >> > > > > > +# >> > > > > > +# Execute 'command' in 'fakenode' >> > > > > > +m4_define([M_EXEC], >> > > > > > + [podman exec $1 $2]) >> > > > > > + >> > > > > > +# M_CHECK_EXEC([fake_node], [command], other_params...) >> > > > > > +# >> > > > > > +# Wrapper for AT_CHECK that executes 'command' inside >> > > 'fake_node''s'. >> > > > > > +# 'other_params' as passed as they are to AT_CHECK. >> > > > > > +m4_define([M_CHECK_EXEC], >> > > > > > + [ AT_CHECK([M_EXEC([$1], [$2])], >> > > m4_shift(m4_shift(m4_shift($@)))) ] >> > > > > > +) >> > > > > > + >> > > > > > +# M_FORMAT_CT([ip-addr]) >> > > > > > +# >> > > > > > +# Strip content from the piped input which would differ from test >> > > > > > to >> > > > > test >> > > > > > +# and limit the output to the rows containing 'ip-addr'. >> > > > > > +# >> > > > > > +m4_define([M_FORMAT_CT], >> > > > > > + [[grep -F "dst=$1," | sed -e 's/id=[0-9]*/id=<cleared>/g' -e >> > > > > 's/state=[0-9_A-Z]*/state=<cleared>/g' | sort | uniq | sed -e >> > > > > 's/zone=[[0-9]]*/zone=<cleared>/' -e >> > > > > 's/mark=[[0-9]]*/mark=<cleared>/' >> > > ]]) >> > > > > > >> > > > > > OVS_START_SHELL_HELPERS >> > > > > > >> > > > > > @@ -76,6 +97,25 @@ multinode_nbctl () { >> > > > > > m_as ovn-central ovn-nbctl "$@" >> > > > > > } >> > > > > > >> > > > > > +check_fake_multinode_setup_by_nodes() { >> > > > > > + check m_as ovn-central ovn-nbctl --wait=sb sync >> > > > > > + for c in $1 >> > > > > > + do >> > > > > > + AT_CHECK([m_as $c ovn-appctl -t ovn-controller version], >> > > [0], >> > > > > [ignore]) >> > > > > > + done >> > > > > > +} >> > > > > > + >> > > > > > +cleanup_multinode_resources_by_nodes() { >> > > > > > + m_as ovn-central rm -f /etc/ovn/ovnnb_db.db >> > > > > > + m_as ovn-central /usr/share/ovn/scripts/ovn-ctl restart_northd >> > > > > > + check m_as ovn-central ovn-nbctl --wait=sb sync >> > > > > > + for c in $1 >> > > > > > + do >> > > > > > + m_as $c ovs-vsctl del-br br-int >> > > > > > + m_as $c ip --all netns delete >> > > > > > + done >> > > > > > +} >> > > > > > + >> > > > > > # m_count_rows TABLE [CONDITION...] >> > > > > > # >> > > > > > # Prints the number of rows in TABLE (that satisfy CONDITION). >> > > > > > diff --git a/tests/multinode.at b/tests/multinode.at >> > > > > > index a0eb8fc67..b1beb4d97 100644 >> > > > > > --- a/tests/multinode.at >> > > > > > +++ b/tests/multinode.at >> > > > > > @@ -1591,3 +1591,559 @@ AT_CHECK([cat ch1_eth2.tcpdump], [0], [dnl >> > > > > > ]) >> > > > > > >> > > > > > AT_CLEANUP >> > > > > > + >> > > > > > +AT_SETUP([ovn multinode load-balancer with multiple DGPs and >> > > multiple >> > > > > chassis]) >> > > > > > + >> > > > > > +# Check that ovn-fake-multinode setup is up and running - requires >> > > > > additional nodes >> > > > > > +check_fake_multinode_setup_by_nodes 'ovn-chassis-1 ovn-chassis-2 >> > > > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' >> > > > > > + >> > > > > > +# Delete the multinode NB and OVS resources before starting the >> > > test. >> > > > > > +cleanup_multinode_resources_by_nodes 'ovn-chassis-1 ovn-chassis-2 >> > > > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' >> > > > > > + >> > > > > > +# Network topology >> > > > > > +# >> > > > > > +# publicp1 (ovn-chassis-3) (20.0.0.3/24) >> > > > > > +# | >> > > > > > +# overlay >> > > > > > +# | >> > > > > > +# DGP public1 (ovn-gw-1) (20.0.0.1/24) >> > > > > > +# | >> > > > > > +# | >> > > > > > +# | >> > > > > > +# lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) >> > > 10.0.0.3/24 >> > > > > > +# | | >> > > > > > +# | + --- sw0p2 (ovn-chassis-2) >> > > 10.0.0.4/24 >> > > > > > +# | >> > > > > > +# DGP public2 (ovn-gw-2) (30.0.0.1/24) >> > > > > > +# | >> > > > > > +# overlay >> > > > > > +# | >> > > > > > +# publicp2 (ovn-chassis-4) (30.0.0.3/24) >> > > > > > + >> > > > > > +# Delete already used ovs-ports >> > > > > > +m_as ovn-chassis-1 ovs-vsctl del-port br-int sw0p1-p >> > > > > > +m_as ovn-chassis-2 ovs-vsctl del-port br-int sw0p2-p >> > > > > > +m_as ovn-chassis-1 ip link del sw0p1-p >> > > > > > +m_as ovn-chassis-2 ip link del sw0p2-p >> > > > > > +m_as ovn-chassis-3 ovs-vsctl del-port br-int publicp1-p >> > > > > > +m_as ovn-chassis-4 ovs-vsctl del-port br-int publicp2-p >> > > > > > +m_as ovn-chassis-3 ip link del publicp1-p >> > > > > > +m_as ovn-chassis-4 ip link del publicp2-p >> > > > > > + >> > > > > > +# Create East-West switch for LB backends >> > > > > > +check multinode_nbctl ls-add sw0 >> > > > > > +check multinode_nbctl lsp-add sw0 sw0-port1 >> > > > > > +check multinode_nbctl lsp-set-addresses sw0-port1 >> > > > > > "50:54:00:00:00:03 >> > > > > 10.0.0.3 1000::3" >> > > > > > +check multinode_nbctl lsp-add sw0 sw0-port2 >> > > > > > +check multinode_nbctl lsp-set-addresses sw0-port2 >> > > > > > "50:54:00:00:00:04 >> > > > > 10.0.0.4 1000::4" >> > > > > > + >> > > > > > +m_as ovn-chassis-1 /data/create_fake_vm.sh sw0-port1 sw0p1 >> > > > > 50:54:00:00:00:03 1400 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a >> > > > > > +m_as ovn-chassis-2 /data/create_fake_vm.sh sw0-port2 sw0p2 >> > > > > 50:54:00:00:00:04 1400 10.0.0.4 24 10.0.0.1 1000::4/64 1000::a >> > > > > > + >> > > > > > +m_wait_for_ports_up >> > > > > > + >> > > > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w >> > > > > > 2 >> > > > > 10.0.0.4 | FORMAT_PING], \ >> > > > > > +[0], [dnl >> > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms >> > > > > > +]) >> > > > > > + >> > > > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w >> > > > > > 2 >> > > > > 10.0.0.3 | FORMAT_PING], \ >> > > > > > +[0], [dnl >> > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms >> > > > > > +]) >> > > > > > + >> > > > > > +# Create a logical router and attach to sw0 >> > > > > > +check multinode_nbctl lr-add lr0 >> > > > > > +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 >> > > 10.0.0.1/24 >> > > > > 1000::a/64 >> > > > > > +check multinode_nbctl lsp-add sw0 sw0-lr0 >> > > > > > +check multinode_nbctl lsp-set-type sw0-lr0 router >> > > > > > +check multinode_nbctl lsp-set-addresses sw0-lr0 router >> > > > > > +check multinode_nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0 >> > > > > > + >> > > > > > +# create external connection for N/S traffic using multiple DGPs >> > > > > > +check multinode_nbctl ls-add public >> > > > > > + >> > > > > > +# DGP public1 >> > > > > > +check multinode_nbctl lsp-add public ln-public-1 >> > > > > > +check multinode_nbctl lsp-set-type ln-public-1 localnet >> > > > > > +check multinode_nbctl lsp-set-addresses ln-public-1 unknown >> > > > > > +check multinode_nbctl lsp-set-options ln-public-1 >> > > network_name=public1 >> > > > > > + >> > > > > > +# DGP public2 >> > > > > > +# create exteranl connection for N/S traffic >> > > > > > +check multinode_nbctl lsp-add public ln-public-2 >> > > > > > +check multinode_nbctl lsp-set-type ln-public-2 localnet >> > > > > > +check multinode_nbctl lsp-set-addresses ln-public-2 unknown >> > > > > > +check multinode_nbctl lsp-set-options ln-public-2 >> > > network_name=public2 >> > > > > > + >> > > > > > +# Attach DGP public1 to GW-1 and chassis-3 (overlay connectivity) >> > > > > > +m_as ovn-gw-1 ovs-vsctl set open . >> > > > > external-ids:ovn-bridge-mappings=public1:br-ex >> > > > > > +m_as ovn-chassis-3 ovs-vsctl set open . >> > > > > external-ids:ovn-bridge-mappings=public1:br-ex >> > > > > > + >> > > > > > +# Attach DGP public2 to GW-2 and chassis-4 (overlay connectivity) >> > > > > > +m_as ovn-gw-2 ovs-vsctl set open . >> > > > > external-ids:ovn-bridge-mappings=public2:br-ex >> > > > > > +m_as ovn-chassis-4 ovs-vsctl set open . >> > > > > external-ids:ovn-bridge-mappings=public2:br-ex >> > > > > > + >> > > > > > +# Create the external LR0 port to the DGP public1 >> > > > > > +check multinode_nbctl lsp-add public public-port1 >> > > > > > +check multinode_nbctl lsp-set-addresses public-port1 >> > > "40:54:00:00:00:03 >> > > > > 20.0.0.3 2000::3" >> > > > > > + >> > > > > > +check multinode_nbctl lrp-add lr0 lr0-public-p1 00:00:00:00:ff:02 >> > > > > 20.0.0.1/24 2000::a/64 >> > > > > > +check multinode_nbctl lsp-add public public-lr0-p1 >> > > > > > +check multinode_nbctl lsp-set-type public-lr0-p1 router >> > > > > > +check multinode_nbctl lsp-set-addresses public-lr0-p1 router >> > > > > > +check multinode_nbctl lsp-set-options public-lr0-p1 >> > > > > router-port=lr0-public-p1 >> > > > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p1 >> > > ovn-gw-1 10 >> > > > > > + >> > > > > > +# Create a VM on ovn-chassis-3 in the same public1 overlay >> > > > > > +m_as ovn-chassis-3 /data/create_fake_vm.sh public-port1 publicp1 >> > > > > 40:54:00:00:00:03 1400 20.0.0.3 24 20.0.0.1 2000::4/64 2000::a >> > > > > > + >> > > > > > +m_wait_for_ports_up public-port1 >> > > > > > + >> > > > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [ping -q -c 3 -i 0.3 >> > > -w 2 >> > > > > 20.0.0.1 | FORMAT_PING], \ >> > > > > > +[0], [dnl >> > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms >> > > > > > +]) >> > > > > > + >> > > > > > +# Create the external LR0 port to the DGP public2 >> > > > > > +check multinode_nbctl lsp-add public public-port2 >> > > > > > +check multinode_nbctl lsp-set-addresses public-port2 >> > > "60:54:00:00:00:03 >> > > > > 30.0.0.3 3000::3" >> > > > > > + >> > > > > > +check multinode_nbctl lrp-add lr0 lr0-public-p2 00:00:00:00:ff:03 >> > > > > 30.0.0.1/24 3000::a/64 >> > > > > > +check multinode_nbctl lsp-add public public-lr0-p2 >> > > > > > +check multinode_nbctl lsp-set-type public-lr0-p2 router >> > > > > > +check multinode_nbctl lsp-set-addresses public-lr0-p2 router >> > > > > > +check multinode_nbctl lsp-set-options public-lr0-p2 >> > > > > router-port=lr0-public-p2 >> > > > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p2 >> > > ovn-gw-2 10 >> > > > > > + >> > > > > > +# Create a VM on ovn-chassis-4 in the same public2 overlay >> > > > > > +m_as ovn-chassis-4 /data/create_fake_vm.sh public-port2 publicp2 >> > > > > 60:54:00:00:00:03 1400 30.0.0.3 24 30.0.0.1 3000::4/64 3000::a >> > > > > > + >> > > > > > +m_wait_for_ports_up public-port2 >> > > > > > + >> > > > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [ping -q -c 3 -i 0.3 >> > > -w 2 >> > > > > 30.0.0.1 | FORMAT_PING], \ >> > > > > > +[0], [dnl >> > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms >> > > > > > +]) >> > > > > > + >> > > > > > +# Add a default route for multiple DGPs - using ECMP >> > > > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 >> > > 20.0.0.3 >> > > > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 >> > > 30.0.0.3 >> > > > > > + >> > > > > > +# Add SNAT rules using gateway-port >> > > > > > +check multinode_nbctl --gateway-port lr0-public-p1 lr-nat-add lr0 >> > > snat >> > > > > 20.0.0.1 10.0.0.0/24 >> > > > > > +check multinode_nbctl --gateway-port lr0-public-p2 lr-nat-add lr0 >> > > snat >> > > > > 30.0.0.1 10.0.0.0/24 >> > > > > > + >> > > > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w >> > > > > > 2 >> > > > > 20.0.0.3 | FORMAT_PING], \ >> > > > > > +[0], [dnl >> > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms >> > > > > > +]) >> > > > > > + >> > > > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w >> > > > > > 2 >> > > > > 30.0.0.3 | FORMAT_PING], \ >> > > > > > +[0], [dnl >> > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms >> > > > > > +]) >> > > > > > + >> > > > > > +# create LB >> > > > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:80" "10.0.0.3:80, >> > > > > 10.0.0.4:80" >> > > > > > +check multinode_nbctl lr-lb-add lr0 lb0 >> > > > > > +check multinode_nbctl ls-lb-add sw0 lb0 >> > > > > > + >> > > > > > +# Set use_stateless_nat to true >> > > > > > +check multinode_nbctl set load_balancer lb0 >> > > > > options:use_stateless_nat=true >> > > > > > + >> > > > > > +# Start backend http services >> > > > > > +M_NS_DAEMONIZE([ovn-chassis-1], [sw0p1], [$PYTHON -m http.server >> > > --bind >> > > > > 10.0.0.3 80 >/dev/null 2>&1], [http1.pid]) >> > > > > > +M_NS_DAEMONIZE([ovn-chassis-2], [sw0p2], [$PYTHON -m http.server >> > > --bind >> > > > > 10.0.0.4 80 >/dev/null 2>&1], [http2.pid]) >> > > > > > + >> > > > > > +# wait for http server be ready >> > > > > > +sleep 2 >> > > > > > + >> > > > > > +# Flush conntrack entries for easier output parsing of next test. >> > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack >> > > > > > + >> > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v >> > > 172.16.0.100:80 >> > > > > --retry 3 --max-time 1 --local-port 59002 2> curl.out']) >> > > > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [sh -c 'cat curl.out >> > > > > > | >> > > > > grep -i -e connect | grep -v 'Server:''], \ >> > > > > > +[0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v >> > > 172.16.0.100:80 >> > > > > --retry 3 --max-time 1 --local-port 59003 2> curl.out']) >> > > > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [sh -c 'cat curl.out >> > > > > > | >> > > > > grep -i -e connect | grep -v 'Server:''], \ >> > > > > > +[0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack >> > > > > > + >> > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v >> > > 172.16.0.100:80 >> > > > > --retry 3 --max-time 1 --local-port 59001']) >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl >> > > > > > dpctl/dump-conntrack | >> > > > > M_FORMAT_CT(20.0.0.3) | \ >> > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], >> > > [0], >> > > > > [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v >> > > 172.16.0.100:80 >> > > > > --retry 3 --max-time 1 --local-port 59000']) >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl >> > > > > > dpctl/dump-conntrack | >> > > > > M_FORMAT_CT(30.0.0.3) | \ >> > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], >> > > [0], >> > > > > [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# create a big file on web servers for download >> > > > > > +M_NS_EXEC([ovn-chassis-1], [sw0p1], [dd bs=512 count=200000 >> > > > > if=/dev/urandom of=download_file]) >> > > > > > +M_NS_EXEC([ovn-chassis-2], [sw0p2], [dd bs=512 count=200000 >> > > > > if=/dev/urandom of=download_file]) >> > > > > > + >> > > > > > +# Flush conntrack entries for easier output parsing of next test. >> > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack >> > > > > > + >> > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O >> > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port >> > > 59004 >> > > > > 2>curl.out']) >> > > > > > + >> > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | >> > > sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | >> > > sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > + >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat >> > > > > curl.out | \ >> > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +# Check if we have only one backend for the same connection - >> > > > > > orig + >> > > > > dest ports >> > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ >> > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], >> > > [0], >> > > > > [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Check if gw-2 is empty to ensure that the traffic only come >> > > from/to >> > > > > the originator chassis via DGP public1 >> > > > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > + >> > > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) >> > > > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep >> > > > > > "dport=80" >> > > -c) >> > > > > > + >> > > > > > +if [[ $backend_check -gt 0 ]]; then >> > > > > > +# Backend resides on ovn-chassis-1 >> > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ >> > > > > > +grep tcp], [0], [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59004,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Ensure that the traffic only come from ovn-chassis-1 >> > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" >> > > -c], >> > > > > [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep >> > > > > > "dport=80" >> > > > > -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +else >> > > > > > +# Backend resides on ovn-chassis-2 >> > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ >> > > > > > +grep tcp], [0], [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59004,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Ensure that the traffic only come from ovn-chassis-2 >> > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" >> > > -c], >> > > > > [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep >> > > > > > "dport=80" >> > > > > -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +fi >> > > > > > + >> > > > > > +# Flush conntrack entries for easier output parsing of next test. >> > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack >> > > > > > + >> > > > > > +# Check the flows again for a new source port >> > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O >> > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port >> > > 59005 >> > > > > 2>curl.out']) >> > > > > > + >> > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | >> > > sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | >> > > sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > + >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat >> > > > > curl.out | \ >> > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +# Check if we have only one backend for the same connection - >> > > > > > orig + >> > > > > dest ports >> > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ >> > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], >> > > [0], >> > > > > [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Check if gw-2 is empty to ensure that the traffic only come >> > > from/to >> > > > > the originator chassis via DGP public1 >> > > > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > + >> > > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) >> > > > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep >> > > > > > "dport=80" >> > > -c) >> > > > > > + >> > > > > > +if [[ $backend_check -gt 0 ]]; then >> > > > > > +# Backend resides on ovn-chassis-1 >> > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ >> > > > > > +grep tcp], [0], [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59005,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Ensure that the traffic only come from ovn-chassis-1 >> > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" >> > > -c], >> > > > > [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep >> > > > > > "dport=80" >> > > > > -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +else >> > > > > > +# Backend resides on ovn-chassis-2 >> > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ >> > > > > > +grep tcp], [0], [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59005,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Ensure that the traffic only come from ovn-chassis-2 >> > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" >> > > -c], >> > > > > [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep >> > > > > > "dport=80" >> > > > > -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +fi >> > > > > > + >> > > > > > +# Flush conntrack entries for easier output parsing of next test. >> > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack >> > > > > > + >> > > > > > +# Start a new test using the second DGP as origin (public2) >> > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O >> > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port >> > > 59006 >> > > > > 2>curl.out']) >> > > > > > + >> > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | >> > > sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | >> > > sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > + >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat >> > > > > curl.out | \ >> > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +# Check if we have only one backend for the same connection - >> > > > > > orig + >> > > > > dest ports >> > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ >> > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], >> > > [0], >> > > > > [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Check if gw-1 is empty to ensure that the traffic only come >> > > from/to >> > > > > the originator chassis via DGP public2 >> > > > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > + >> > > > > > +# Check the backend IP from ct entries on gw-2 (DGP public2) >> > > > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep >> > > > > > "dport=80" >> > > -c) >> > > > > > + >> > > > > > +if [[ $backend_check -gt 0 ]]; then >> > > > > > +# Backend resides on ovn-chassis-1 >> > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ >> > > > > > +grep tcp], [0], [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59006,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Ensure that the traffic only come from ovn-chassis-1 >> > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" >> > > -c], >> > > > > [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep >> > > > > > "dport=80" >> > > > > -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +else >> > > > > > +# Backend resides on ovn-chassis-2 >> > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ >> > > > > > +grep tcp], [0], [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59006,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Ensure that the traffic only come from ovn-chassis-2 >> > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" >> > > -c], >> > > > > [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep >> > > > > > "dport=80" >> > > > > -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +fi >> > > > > > + >> > > > > > +# Flush conntrack entries for easier output parsing of next test. >> > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack >> > > > > > + >> > > > > > +# Check the flows again for a new source port using the second DGP >> > > as >> > > > > origin (public2) >> > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O >> > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port >> > > 59007 >> > > > > 2>curl.out']) >> > > > > > + >> > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | >> > > sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | >> > > sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > + >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat >> > > > > curl.out | \ >> > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +# Check if we have only one backend for the same connection - >> > > > > > orig + >> > > > > dest ports >> > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ >> > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], >> > > [0], >> > > > > [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Check if gw-1 is empty to ensure that the traffic only come >> > > from/to >> > > > > the originator chassis via DGP public2 >> > > > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > + >> > > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) >> > > > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep >> > > > > > "dport=80" >> > > -c) >> > > > > > + >> > > > > > +if [[ $backend_check -gt 0 ]]; then >> > > > > > +# Backend resides on ovn-chassis-1 >> > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ >> > > > > > +grep tcp], [0], [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59007,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Ensure that the traffic only come from ovn-chassis-1 >> > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" >> > > -c], >> > > > > [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep >> > > > > > "dport=80" >> > > > > -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +else >> > > > > > +# Backend resides on ovn-chassis-2 >> > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ >> > > > > > +grep tcp], [0], [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59007,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Ensure that the traffic only come from ovn-chassis-2 >> > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" >> > > -c], >> > > > > [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep >> > > > > > "dport=80" >> > > > > -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +fi >> > > > > > + >> > > > > > +# Check multiple requests coming from DGP's public1 and public2 >> > > > > > + >> > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O >> > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ >> > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | >> > > grep >> > > > > -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +200 OK >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O >> > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ >> > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | >> > > grep >> > > > > -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +200 OK >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O >> > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ >> > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | >> > > grep >> > > > > -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +200 OK >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O >> > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ >> > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | >> > > grep >> > > > > -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +200 OK >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +# Remove the LB and change the VIP port - different from the >> > > > > > backend >> > > > > ports >> > > > > > +check multinode_nbctl lb-del lb0 >> > > > > > + >> > > > > > +# create LB again >> > > > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:9000" "10.0.0.3:80, >> > > > > 10.0.0.4:80" >> > > > > > +check multinode_nbctl lr-lb-add lr0 lb0 >> > > > > > +check multinode_nbctl ls-lb-add sw0 lb0 >> > > > > > + >> > > > > > +# Set use_stateless_nat to true >> > > > > > +check multinode_nbctl set load_balancer lb0 >> > > > > options:use_stateless_nat=true >> > > > > > + >> > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack >> > > > > > + >> > > > > > +# Check end-to-end request using a new port for VIP >> > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O >> > > > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port >> > > 59008 >> > > > > 2>curl.out']) >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl >> > > > > > dpctl/dump-conntrack | >> > > > > M_FORMAT_CT(20.0.0.3) | \ >> > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], >> > > [0], >> > > > > [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ >> > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | >> > > grep >> > > > > -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 >> > > > > > +200 OK >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack >> > > > > > + >> > > > > > +# Check end-to-end request using a new port for VIP >> > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O >> > > > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port >> > > 59008 >> > > > > 2>curl.out']) >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl >> > > > > > dpctl/dump-conntrack | >> > > > > M_FORMAT_CT(30.0.0.3) | \ >> > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], >> > > [0], >> > > > > [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ >> > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | >> > > grep >> > > > > -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 >> > > > > > +200 OK >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +AT_CLEANUP >> > > > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at >> > > > > > index dcc3dbbc3..9e7a2f225 100644 >> > > > > > --- a/tests/ovn-northd.at >> > > > > > +++ b/tests/ovn-northd.at >> > > > > > @@ -13864,3 +13864,323 @@ check_no_redirect >> > > > > > >> > > > > > AT_CLEANUP >> > > > > > ]) >> > > > > > + >> > > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ >> > > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + >> > > NAT >> > > > > Stateless)]) >> > > > > > +ovn_start >> > > > > > + >> > > > > > +check ovn-nbctl ls-add public >> > > > > > +check ovn-nbctl lr-add lr1 >> > > > > > + >> > > > > > +# lr1 DGP ts1 >> > > > > > +check ovn-nbctl ls-add ts1 >> > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 >> > > 172.16.10.1/24 >> > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 >> > > > > > + >> > > > > > +# lr1 DGP ts2 >> > > > > > +check ovn-nbctl ls-add ts2 >> > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 >> > > 172.16.20.1/24 >> > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 >> > > > > > + >> > > > > > +# lr1 DGP public >> > > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 >> > > 173.16.0.1/16 >> > > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 >> > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 >> > > > > > + >> > > > > > +check ovn-nbctl ls-add s1 >> > > > > > +# s1 - lr1 >> > > > > > +check ovn-nbctl lsp-add s1 s1_lr1 >> > > > > > +check ovn-nbctl lsp-set-type s1_lr1 router >> > > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 >> > > 172.16.0.1" >> > > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 >> > > > > > + >> > > > > > +# s1 - backend vm1 >> > > > > > +check ovn-nbctl lsp-add s1 vm1 >> > > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 >> > > 172.16.0.101" >> > > > > > + >> > > > > > +# s1 - backend vm2 >> > > > > > +check ovn-nbctl lsp-add s1 vm2 >> > > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 >> > > 172.16.0.102" >> > > > > > + >> > > > > > +# s1 - backend vm3 >> > > > > > +check ovn-nbctl lsp-add s1 vm3 >> > > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 >> > > 172.16.0.103" >> > > > > > + >> > > > > > +# Add the lr1 DGP ts1 to the public switch >> > > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 >> > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router >> > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router >> > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 >> > > > > nat-addresses=router >> > > > > > + >> > > > > > +# Add the lr1 DGP ts2 to the public switch >> > > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 >> > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router >> > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router >> > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 >> > > > > nat-addresses=router >> > > > > > + >> > > > > > +# Add the lr1 DGP public to the public switch >> > > > > > +check ovn-nbctl lsp-add public public_lr1 >> > > > > > +check ovn-nbctl lsp-set-type public_lr1 router >> > > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router >> > > > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public >> > > > > nat-addresses=router >> > > > > > + >> > > > > > +# Create the Load Balancer lb1 >> > > > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" >> > > > > "172.16.0.103,172.16.0.102,172.16.0.101" >> > > > > > + >> > > > > > +# Set use_stateless_nat to true >> > > > > > +check ovn-nbctl --wait=sb set load_balancer lb1 >> > > > > options:use_stateless_nat=true >> > > > > > + >> > > > > > +# Associate load balancer to s1 >> > > > > > +check ovn-nbctl ls-lb-add s1 lb1 >> > > > > > +check ovn-nbctl --wait=sb sync >> > > > > > + >> > > > > > +ovn-sbctl dump-flows s1 > s1flows >> > > > > > +AT_CAPTURE_FILE([s1flows]) >> > > > > > + >> > > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | >> > > grep >> > > > > "30.0.0.1"], [0], [dnl >> > > > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] >> > > == 1 >> > > > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) >> > > > > > +]) >> > > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep >> > > > > "30.0.0.1"], [0], [dnl >> > > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && >> > > > > ip4.dst == 30.0.0.1), >> > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) >> > > > > > +]) >> > > > > > + >> > > > > > +# Associate load balancer to lr1 with DGP >> > > > > > +check ovn-nbctl lr-lb-add lr1 lb1 >> > > > > > +check ovn-nbctl --wait=sb sync >> > > > > > + >> > > > > > +ovn-sbctl dump-flows lr1 > lr1flows >> > > > > > +AT_CAPTURE_FILE([lr1flows]) >> > > > > > + >> > > > > > +# Check stateless NAT rules for load balancer with multiple DGP >> > > > > > +# 1. Check if the backend IPs are in the ipX.dst action >> > > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep >> > > > > "30.0.0.1"], [0], [dnl >> > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && >> > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && >> > > > > is_chassis_resident("cr-lr1-ts1")), >> > > > > >> > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) >> > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && >> > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && >> > > > > is_chassis_resident("cr-lr1-ts2")), >> > > > > >> > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) >> > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && >> > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && >> > > > > is_chassis_resident("cr-lr1_public")), >> > > > > >> > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) >> > > > > > +]) >> > > > > > + >> > > > > > +# 2. Check if the DGP ports are in the match with action next >> > > > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], >> > > [dnl >> > > > > > + table=??(lr_out_undnat ), priority=0 , match=(1), >> > > > > action=(next;) >> > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && >> > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src >> > > > > == >> > > > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && >> > > > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) >> > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && >> > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src >> > > > > == >> > > > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && >> > > > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) >> > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && >> > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src >> > > > > == >> > > > > 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") >> > > && >> > > > > is_chassis_resident("cr-lr1_public") && tcp), action=(next;) >> > > > > > +]) >> > > > > > + >> > > > > > +# 3. Check if the VIP IP is in the ipX.src action >> > > > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], >> > > > > > [dnl >> > > > > > + table=??(lr_out_snat ), priority=0 , match=(1), >> > > > > action=(next;) >> > > > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), >> > > > > action=(next;) >> > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && >> > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src >> > > > > == >> > > > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && >> > > > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(ip4.src=30.0.0.1; >> > > next;) >> > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && >> > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src >> > > > > == >> > > > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && >> > > > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(ip4.src=30.0.0.1; >> > > next;) >> > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && >> > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src >> > > > > == >> > > > > 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") >> > > && >> > > > > is_chassis_resident("cr-lr1_public") && tcp), >> > > > > action=(ip4.src=30.0.0.1; >> > > > > next;) >> > > > > > +]) >> > > > > > + >> > > > > > +AT_CLEANUP >> > > > > > +]) >> > > > > > + >> > > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ >> > > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + >> > > NAT >> > > > > Stateless) - IPv6]) >> > > > > > +ovn_start >> > > > > > + >> > > > > > +check ovn-nbctl ls-add public >> > > > > > +check ovn-nbctl lr-add lr1 >> > > > > > + >> > > > > > +# lr1 DGP ts1 >> > > > > > +check ovn-nbctl ls-add ts1 >> > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 >> > > > > 2001:db8:aaaa:1::1/64 >> > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 >> > > > > > + >> > > > > > +# lr1 DGP ts2 >> > > > > > +check ovn-nbctl ls-add ts2 >> > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 >> > > > > 2001:db8:aaaa:2::1/64 >> > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 >> > > > > > + >> > > > > > +# lr1 DGP public >> > > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 >> > > > > 2001:db8:bbbb::1/64 >> > > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 >> > > > > 2001:db8:aaaa:3::1/64 >> > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 >> > > > > > + >> > > > > > +check ovn-nbctl ls-add s1 >> > > > > > +# s1 - lr1 >> > > > > > +check ovn-nbctl lsp-add s1 s1_lr1 >> > > > > > +check ovn-nbctl lsp-set-type s1_lr1 router >> > > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 >> > > > > 2001:db8:aaaa:3::1" >> > > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 >> > > > > > + >> > > > > > +# s1 - backend vm1 >> > > > > > +check ovn-nbctl lsp-add s1 vm1 >> > > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 >> > > > > 2001:db8:aaaa:3::101" >> > > > > > + >> > > > > > +# s1 - backend vm2 >> > > > > > +check ovn-nbctl lsp-add s1 vm2 >> > > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 >> > > > > 2001:db8:aaaa:3::102" >> > > > > > + >> > > > > > +# s1 - backend vm3 >> > > > > > +check ovn-nbctl lsp-add s1 vm3 >> > > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 >> > > > > 2001:db8:aaaa:3::103" >> > > > > > + >> > > > > > +# Add the lr1 DGP ts1 to the public switch >> > > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 >> > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router >> > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router >> > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 >> > > > > nat-addresses=router >> > > > > > + >> > > > > > +# Add the lr1 DGP ts2 to the public switch >> > > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 >> > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router >> > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router >> > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 >> > > > > nat-addresses=router >> > > > > > + >> > > > > > +# Add the lr1 DGP public to the public switch >> > > > > > +check ovn-nbctl lsp-add public public_lr1 >> > > > > > +check ovn-nbctl lsp-set-type public_lr1 router >> > > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router >> > > > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public >> > > > > nat-addresses=router >> > > > > > + >> > > > > > +# Create the Load Balancer lb1 >> > > > > > +check ovn-nbctl --wait=sb lb-add lb1 "2001:db8:cccc::1" >> > > > > "2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101" >> > > > > > + >> > > > > > +# Set use_stateless_nat to true >> > > > > > +check ovn-nbctl --wait=sb set load_balancer lb1 >> > > > > options:use_stateless_nat=true >> > > > > > + >> > > > > > +# Associate load balancer to s1 >> > > > > > +check ovn-nbctl ls-lb-add s1 lb1 >> > > > > > +check ovn-nbctl --wait=sb sync >> > > > > > + >> > > > > > +ovn-sbctl dump-flows s1 > s1flows >> > > > > > +AT_CAPTURE_FILE([s1flows]) >> > > > > > + >> > > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | >> > > grep >> > > > > "2001:db8:cccc::1"], [0], [dnl >> > > > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] >> > > == 1 >> > > > > && ip6.dst == 2001:db8:cccc::1), action=(xxreg1 = 2001:db8:cccc::1; >> > > > > ct_lb_mark;) >> > > > > > +]) >> > > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep >> > > > > "2001:db8:cccc::1"], [0], [dnl >> > > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && >> > > > > ip6.dst == 2001:db8:cccc::1), >> > > > > >> > > action=(ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) >> > > > > > +]) >> > > > > > + >> > > > > > +# Associate load balancer to lr1 with DGP >> > > > > > +check ovn-nbctl lr-lb-add lr1 lb1 >> > > > > > +check ovn-nbctl --wait=sb sync >> > > > > > + >> > > > > > +ovn-sbctl dump-flows lr1 > lr1flows >> > > > > > +AT_CAPTURE_FILE([lr1flows]) >> > > > > > + >> > > > > > +# Check stateless NAT rules for load balancer with multiple DGP >> > > > > > +# 1. Check if the backend IPs are in the ipX.dst action >> > > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep >> > > > > "2001:db8:cccc::1"], [0], [dnl >> > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && >> > > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && >> > > > > is_chassis_resident("cr-lr1-ts1")), >> > > > > >> > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) >> > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && >> > > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && >> > > > > is_chassis_resident("cr-lr1-ts2")), >> > > > > >> > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) >> > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && >> > > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && >> > > > > is_chassis_resident("cr-lr1_public")), >> > > > > >> > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) >> > > > > > +]) >> > > > > > + >> > > > > > +# 2. Check if the DGP ports are in the match with action next >> > > > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], >> > > [dnl >> > > > > > + table=??(lr_out_undnat ), priority=0 , match=(1), >> > > > > action=(next;) >> > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && >> > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == >> > > 2001:db8:aaaa:3::102) || >> > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport >> > > == >> > > > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), >> > > > > action=(next;) >> > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && >> > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == >> > > 2001:db8:aaaa:3::102) || >> > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport >> > > == >> > > > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), >> > > > > action=(next;) >> > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && >> > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == >> > > 2001:db8:aaaa:3::102) || >> > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || >> > > outport == >> > > > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), >> > > > > action=(next;) >> > > > > > +]) >> > > > > > + >> > > > > > +# 3. Check if the VIP IP is in the ipX.src action >> > > > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], >> > > > > > [dnl >> > > > > > + table=??(lr_out_snat ), priority=0 , match=(1), >> > > > > action=(next;) >> > > > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), >> > > > > action=(next;) >> > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && >> > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == >> > > 2001:db8:aaaa:3::102) || >> > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport >> > > == >> > > > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), >> > > > > action=(ip6.src=2001:db8:cccc::1; next;) >> > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && >> > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == >> > > 2001:db8:aaaa:3::102) || >> > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport >> > > == >> > > > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), >> > > > > action=(ip6.src=2001:db8:cccc::1; next;) >> > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && >> > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == >> > > 2001:db8:aaaa:3::102) || >> > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || >> > > outport == >> > > > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), >> > > > > action=(ip6.src=2001:db8:cccc::1; next;) >> > > > > > +]) >> > > > > > + >> > > > > > +AT_CLEANUP >> > > > > > +]) >> > > > > > + >> > > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ >> > > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (DGP)]) >> > > > > > +ovn_start >> > > > > > + >> > > > > > +check ovn-nbctl ls-add public >> > > > > > +check ovn-nbctl lr-add lr1 >> > > > > > + >> > > > > > +# lr1 DGP ts1 >> > > > > > +check ovn-nbctl ls-add ts1 >> > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 >> > > 172.16.10.1/24 >> > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-1 >> > > > > > + >> > > > > > +# lr1 DGP ts2 >> > > > > > +check ovn-nbctl ls-add ts2 >> > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 >> > > 172.16.20.1/24 >> > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-1 >> > > > > > + >> > > > > > +# lr1 DGP public >> > > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 >> > > 173.16.0.1/16 >> > > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 >> > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 >> > > > > > + >> > > > > > +check ovn-nbctl ls-add s1 >> > > > > > +# s1 - lr1 >> > > > > > +check ovn-nbctl lsp-add s1 s1_lr1 >> > > > > > +check ovn-nbctl lsp-set-type s1_lr1 router >> > > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 >> > > 172.16.0.1" >> > > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 >> > > > > > + >> > > > > > +# s1 - backend vm1 >> > > > > > +check ovn-nbctl lsp-add s1 vm1 >> > > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 >> > > 172.16.0.101" >> > > > > > + >> > > > > > +# s1 - backend vm2 >> > > > > > +check ovn-nbctl lsp-add s1 vm2 >> > > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 >> > > 172.16.0.102" >> > > > > > + >> > > > > > +# s1 - backend vm3 >> > > > > > +check ovn-nbctl lsp-add s1 vm3 >> > > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 >> > > 172.16.0.103" >> > > > > > + >> > > > > > +# Add the lr1 DGP ts1 to the public switch >> > > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 >> > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router >> > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router >> > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 >> > > > > nat-addresses=router >> > > > > > + >> > > > > > +# Add the lr1 DGP ts2 to the public switch >> > > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 >> > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router >> > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router >> > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 >> > > > > nat-addresses=router >> > > > > > + >> > > > > > +# Add the lr1 DGP public to the public switch >> > > > > > +check ovn-nbctl lsp-add public public_lr1 >> > > > > > +check ovn-nbctl lsp-set-type public_lr1 router >> > > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router >> > > > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public >> > > > > nat-addresses=router >> > > > > > + >> > > > > > +# Create the Load Balancer lb1 >> > > > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" >> > > > > "172.16.0.103,172.16.0.102,172.16.0.101" >> > > > > > + >> > > > > > +# Associate load balancer to s1 >> > > > > > +check ovn-nbctl ls-lb-add s1 lb1 >> > > > > > +check ovn-nbctl --wait=sb sync >> > > > > > + >> > > > > > +ovn-sbctl dump-flows s1 > s1flows >> > > > > > +AT_CAPTURE_FILE([s1flows]) >> > > > > > + >> > > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | >> > > grep >> > > > > "30.0.0.1"], [0], [dnl >> > > > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] >> > > == 1 >> > > > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) >> > > > > > +]) >> > > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep >> > > > > "30.0.0.1"], [0], [dnl >> > > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && >> > > > > ip4.dst == 30.0.0.1), >> > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) >> > > > > > +]) >> > > > > > + >> > > > > > +# Associate load balancer to lr1 with DGP >> > > > > > +check ovn-nbctl lr-lb-add lr1 lb1 >> > > > > > +check ovn-nbctl --wait=sb sync >> > > > > > + >> > > > > > +ovn-sbctl dump-flows lr1 > lr1flows >> > > > > > +AT_CAPTURE_FILE([lr1flows]) >> > > > > > + >> > > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep >> > > > > "30.0.0.1"], [0], [dnl >> > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && >> > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && >> > > > > is_chassis_resident("cr-lr1-ts1")), >> > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) >> > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && >> > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && >> > > > > is_chassis_resident("cr-lr1-ts2")), >> > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) >> > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && >> > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && >> > > > > is_chassis_resident("cr-lr1_public")), >> > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) >> > > > > > +]) >> > > > > > + >> > > > > > +AT_CLEANUP >> > > > > > +]) >> > > > > > -- >> > > > > > 2.34.1 >> > > > > > >> > > > > > >> > > > > > -- >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > _'Esta mensagem é direcionada apenas para os endereços constantes >> > > > > > no >> > > > > > cabeçalho inicial. Se você não está listado nos endereços >> > > > > > constantes >> > > no >> > > > > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo >> > > dessa >> > > > > > mensagem e cuja cópia, encaminhamento e/ou execução das ações >> > > > > > citadas >> > > > > estão >> > > > > > imediatamente anuladas e proibidas'._ >> > > > > > >> > > > > > >> > > > > > * **'Apesar do Magazine Luiza tomar >> > > > > > todas as precauções razoáveis para assegurar que nenhum vírus >> > > > > > esteja >> > > > > > presente nesse e-mail, a empresa não poderá aceitar a >> > > responsabilidade >> > > > > por >> > > > > > quaisquer perdas ou danos causados por esse e-mail ou por seus >> > > anexos'.* >> > > > > > >> > > > > > >> > > > > > >> > > > > > _______________________________________________ >> > > > > > dev mailing list >> > > > > > [email protected] >> > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev >> > > > > > >> > > > > >> > > > >> > > > -- >> > > > >> > > > >> > > > >> > > > >> > > > _‘Esta mensagem é direcionada apenas para os endereços constantes no >> > > > cabeçalho inicial. Se você não está listado nos endereços constantes no >> > > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa >> > > > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas >> > > estão >> > > > imediatamente anuladas e proibidas’._ >> > > > >> > > > >> > > > * **‘Apesar do Magazine Luiza tomar >> > > > todas as precauções razoáveis para assegurar que nenhum vírus esteja >> > > > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade >> > > por >> > > > quaisquer perdas ou danos causados por esse e-mail ou por seus >> > > > anexos’.* >> > > > >> > > > >> > > > >> > > > _______________________________________________ >> > > > dev mailing list >> > > > [email protected] >> > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev >> > > >> > >> > -- >> > >> > >> > >> > >> > _‘Esta mensagem é direcionada apenas para os endereços constantes no >> > cabeçalho inicial. Se você não está listado nos endereços constantes no >> > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa >> > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão >> > imediatamente anuladas e proibidas’._ >> > >> > >> > * **‘Apesar do Magazine Luiza tomar >> > todas as precauções razoáveis para assegurar que nenhum vírus esteja >> > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por >> > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.* >> > >> > >> > >> > _______________________________________________ >> > dev mailing list >> > [email protected] >> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev >> > > > ‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho > inicial. Se você não está listado nos endereços constantes no cabeçalho, > pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja > cópia, encaminhamento e/ou execução das ações citadas estão imediatamente > anuladas e proibidas’. > > ‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar > que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a > responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou > por seus anexos’. _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
