On Tue, Mar 1, 2022 at 6:31 AM Brendan Doyle <[email protected]> wrote: > > > > On 01/03/2022 01:12, Numan Siddique wrote: > > On Mon, Feb 28, 2022 at 11:40 AM Brendan Doyle <[email protected]> > > wrote: > >> > >> > >> On 20/02/2022 23:38, Han Zhou wrote: > >> > >> > >> > >> On Thu, Feb 17, 2022 at 3:23 AM Brendan Doyle <[email protected]> > >> wrote: > >>> Hi, > >>> > >>> So I have a Distributed Gateway Port (DGP) on a Gateway through which > >>> VMs in the overlay can access > >>> underlay networks. If the VM is not on the chassis where the DGP is > >>> scheduled then the traffic takes > >>> the extra tunneled hop to the chassis where the DGP is and is then sent > >>> to the underlay via the localnet > >>> switch there. It would be great if I could avoid that extra hop, whilst > >>> still having the Gateway do NAT > >>> and routing. So I'm wondering if 'reside-on-redirect-chassis' or ' > >>> redirect-type' can be used to > >>> do this? and if so which one? Also will normal traffic between to VMs in > >>> the overlay on different chassis > >>> still be tunneled through the underlay? > >>> > >>> I'll do some experimentation later, but a nod in the right direction > >>> would be appreciated. > >>> > >> Hi Brendan, > >> > >> Not sure if I understood your question well. If you want to avoid the > >> gateway hop, you can just configure NAT rules with "dnat_and_snat" to do > >> distributed NAT. > >> 'reside-on-redirect-chassis' and 'redirect-type' don't seem to help for > >> your use case because you mentioned your VMs are in overlay, while those > >> options are for VLAN based logical networks. > >> > >> > >> I must admit I found the ovn-nb/ovn-architetcure doc quite confusing on > >> this. I tracked down > >> the patches and some ovn-discuss exchange and I think I understand it > >> better now. It > >> seems that rather then avoiding the extra hop to the gateway chassis, > >> these redirect options > >> just let you take that extra hop as a VLAN tagged packet in the underlay > >> rather than a geneve > >> encapsulated packet. Which is not what I was hoping they would do. I'm not > >> sure what you mean > >> by " you can just configure NAT rules with "dnat_and_snat" to do > >> distributed NAT" I could > >> not find any reference to "distributed NAT" in the man pages. We use a > >> distributed router > >> port Gateway for HA. Consider this config: > >> > >> switch 2ffbd2d8-5421-452a-b776-e32cc55f4789 (ls_vcn3) > >> port 269089c4-9464-41ec-9f63-6b3804b34b07 > >> addresses: ["52:54:00:30:38:35 192.16.1.5"] > >> port ls_vcn3_net1-lr_vcn3_net1 > >> type: router > >> addresses: ["40:44:00:00:00:90"] > >> router-port: lr_vcn3_net1-ls_vcn3_net1 > >> port 284195d2-9280-4334-900e-571ecd00327a > >> addresses: ["52:54:00:02:55:96 192.16.1.6"] > >> > >> switch af0f96b0-d7af-41ef-b5b7-fa8ae0186c2c (ls_vcn3_backbone) > >> port lsb_vcn3-lr_vcn3 > >> type: router > >> router-port: lr_vcn3-lsb_vcn3 > >> port lsb_vcn3_net1-lr_vcn3_net1 > >> type: router > >> router-port: lr_vcn3_net1-lsb_vcn3_net1 > >> > >> switch 29bdf630-8326-4aa4-aee1-d9bca9ea298c (ls_vcn3_external_ugw) > >> port ls_vcn3_external_ugw-lr_vcn3 > >> type: router > >> router-port: lr_vcn3-ls_vcn3_external_ugw > >> port ln-ls_vcn3_external_ugw > >> type: localnet > >> addresses: ["unknown"] > >> > >> router 638a1401-4e5f-450b-9d35-fe74defc8769 (lr_vcn3) > >> port lr_vcn3-ls_vcn3_external_ugw > >> mac: "40:44:00:00:01:60" > >> networks: ["253.255.80.4/16"] > >> gateway chassis: [sca15-rain17 sca15-rain05 sca15-rain06] > >> port lr_vcn3-lsb_vcn3 > >> mac: "40:44:00:00:01:50" > >> networks: ["253.255.25.2/25"] > >> nat cb880ef9-d51c-4265-abdc-db6accaa4d4a > >> external ip: "253.255.80.4" > >> logical ip: "192.16.0.0/16" > >> type: "snat" > >> > >> router f370260a-c90c-4824-91a8-fba787272dc2 (lr_vcn3_net1) > >> port lr_vcn3_net1-lsb_vcn3_net1 > >> mac: "40:44:00:00:00:a0" > >> networks: ["253.255.25.1/25"] > >> port lr_vcn3_net1-ls_vcn3_net1 > >> mac: "40:44:00:00:00:90" > >> networks: ["192.16.1.1/24"] > >> > >> ovn-nbctl lr-nat-list lr_vcn3 > >> TYPE EXTERNAL_IP EXTERNAL_PORT LOGICAL_IP > >> EXTERNAL_MAC LOGICAL_PORT > >> snat 253.255.80.4 192.16.0.0/16 > >> > >> ovn-sbctl show > >> > >> Chassis sca15-rain05 > >> hostname: sca15-rain05 > >> Encap geneve > >> ip: "253.255.2.5" > >> options: {csum="true"} > >> Port_Binding "269089c4-9464-41ec-9f63-6b3804b34b07" > >> Port_Binding cr-lr_vcn3-ls_vcn3_external_ugw > >> > >> Chassis sca15-rain06 > >> hostname: sca15-rain06 > >> Encap geneve > >> ip: "253.255.2.6" > >> options: {csum="true"} > >> Port_Binding "284195d2-9280-4334-900e-571ecd00327a" > >> > >> > >> So if the overlay VM with IP 192.16.1.5 (hosted on chassis sca15-rain05) > >> is sending traffic to the > >> underlay it will use "physical"/localnet bridge on sca15-rain05. But if > >> the overlay VM with IP 192.16.1.6 > >> (hosted on chassis sca15-rain06) is sending traffic to the underlay, the > >> traffic is tunneled to sca15-rain05 > >> first, because that is where the DR port is, and then it is sent to the > >> underlay via the physical/localnet > >> bridge there. It was that extra hop I was trying to avoid. i.e Have the > >> NAT done > >> by the flows on chassis sca15-rain06 and be sent to the underlay by the > >> localnet/physical bridge on that > >> chassis. But seems like this is not possible? > > If you want to avoid this extra hop, you need to create a > > dnat_and_snat entry on router lr_vcn3 of type "dnat_and_snat" > > and also set external_mac and logical_port columns. > > > > Something like this: > > > > ovn-nbctl lr-nat-add lr_vcn3 dnat_and_snat 253.255.80.100 192.16.1.6 > > 284195d2-9280-4334-900e-571ecd00327a 30:54:00:00:00:04 > > > > With your topology shared above, you have created a NAT entry in the > > router lr_vcn3 of type "snat".- > > And OVN doesn't support distributed snat. And I'm not sure if it is > > even possible (I could be wrong). > > > > If you define dnat_and_snat entry as I showed above in the example, > > then ovn-controller running on chassis sca15-rain06 > > will send garps to the underlay network with the IP-MAC - > > (253.255.80.100 - 30:54:00:00:00:04). And that ovn-controller will > > also > > do NATting from 192.169.1.6 to 253.255.80.100. This way, the packet > > will not be tunneled to sca15-rain05. Hope this makes sense. > Thanks Numan, yes this helps somewhat but If I understand this correctly > then > then as I can multiple VMs on multiple chassis and the distributed > router port on > the lr_vcn3 gateway (cr-lr_vcn3-ls_vcn3_external_ugw) can be scheduled and > move between multiple chassis then to ensure no extra hop I'd have to > have a > dnat_and_snat entry on lr_vcn3 with a unique underlay IP and MAC for > every VM. > So in this case where there are just two VMs I'd have to have: > > ovn-nbctl lr-nat-add lr_vcn3 dnat_and_snat 253.255.80.100 192.16.1.6 > 284195d2-9280-4334-900e-571ecd00327a 30:54:00:00:00:04 > ovn-nbctl lr-nat-add lr_vcn3 dnat_and_snat 253.255.80.101 192.16.1.5 > 269089c4-9464-41ec-9f63-6b3804b34b07 30:54:00:00:00:05 > > > Which unfortunately does not work for all the use case of gateways we have. > In some cases we do need SNAT for a CIDR Not an IP > > Also the ovn-nbctl man page states > > "The logical_port and external_mac are only accepted when router is a > distributed router (rather than a gateway router) " > > And lr_vcn3 is a gateway router.
lr_vcn3 is not a gateway router. It is distributed router with gateway router port - lr_vcn3-ls_vcn3_external_ugw To create a gateway router, you need to pin a router to a particular chassis and all the routing for the router is handled on that chassis. Eg. ovn-nbctl set logical_router lr_vcn3 options:chassis=sca15-rain17 sca15-rain05 sca15-rain06 will make the router lr_vcn3 as a gateway router. And we don't support dnat_and_snat type with external_mac and logical_port for gateway routers. Also as I mentioned OVN doesn't support distributed SNAT. In your case, you have configured an snat entry for IP - 253.255.80.4. Which means it is not possible for every compute node to do SNAT from the overlay network ip of the VMs to 253.255.80.4 (for the South to North external traffic) as upstream router will see the IP - MAC (253.255.80.4 - 40:44:00:00:01:60) from multiple ports. Hope this makes it clear. Numan > > > > > > I think Han also suggested the same. > > > > Since your overlay logical switch ls_vcn3 doesn't have a localnet > > port, 'reside-on-redirect-chassis' option doesn't come into picture > > at all. > > This option is used when you create VMs on provider networks (like for > > example ls_vcn3_external_ugw) and you connect these logical switches > > to an OVN router. > > > > Thanks > > Numan > > > > > >> Brendan > >> > >> > >> > >> > >> > >> Thanks, > >> Han > >> > >>> Thanks > >>> > >>> > >>> Brendan > >>> > >>> _______________________________________________ > >>> discuss mailing list > >>> [email protected] > >>> https://urldefense.com/v3/__https://mail.openvswitch.org/mailman/listinfo/ovs-discuss__;!!ACWV5N9M2RV99hQ!cTLgyBQpfKdvUEuLP1yuMMoCJ2NS80YuD9ZUheR22GgpnxrtsibaJUX9seQ0dj1Suzg$ > >> > >> _______________________________________________ > >> discuss mailing list > >> [email protected] > >> https://urldefense.com/v3/__https://mail.openvswitch.org/mailman/listinfo/ovs-discuss__;!!ACWV5N9M2RV99hQ!cTLgyBQpfKdvUEuLP1yuMMoCJ2NS80YuD9ZUheR22GgpnxrtsibaJUX9seQ0dj1Suzg$ > > _______________________________________________ > discuss mailing list > [email protected] > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss _______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
