On Tue, Jun 29, 2021 at 5:06 PM Brendan Doyle <[email protected]> wrote: > > > > On 29/06/2021 21:38, Numan Siddique wrote: > > On Tue, Jun 29, 2021 at 4:13 PM Brendan Doyle <[email protected]> > > wrote: > >> Hi, > >> > >> With a very simple notwork (two VMs on different chassis), 1 subnet, > >> single LS and > >> LR/Gateway. The two VMs can ping each other using their Logical IPs. > >> Each has an > >> "External IP", and each can be accessed from an external network on that > >> external IP. > >> BUT they can't ping each other using their external IPs. I would have > >> expected that > >> either: > >> > >> a) The packets are sent on the external net then hairpinned back to the OVN > >> gateway by the external net router. > >> > >> b) They are hairpinned by OVN. > >> > >> It seems that OVN attempts the latter, but does not succeed. The > >> details, NB network, > >> and pkt trace are as follows: > >> > >> ovn-nbctl show > >> switch 2710eebe-f2b3-49e4-bcd6-dcfa48ed6470 (ls1_external) > >> port ln-ls1_external > >> type: localnet > >> addresses: ["unknown"] > >> port ls1_external-lr1 > >> type: router > >> router-port: lr1-ls1_external > >> > >> switch ff909b16-d863-4e3d-a10b-2f0010f17b23 (ls1) > >> port 47433b54-ac10-42f1-ae84-cc6fbb580297 > >> addresses: ["52:54:00:be:06:16 192.16.1.6"] > >> port 00bff7c0-2e2d-41ba-9485-3b5fa9801365 > >> addresses: ["52:54:00:e6:4f:46 192.16.1.5"] > >> port ls1-lr1 > >> type: router > >> router-port: lr1-ls1 > >> > >> router 63e1b6a2-327f-4a24-b0c9-3a0e951beb2b (lr1) > >> port lr1-ls1_external > >> mac: "40:44:00:00:01:a0" > >> networks: ["253.255.80.10/16"] > >> gateway chassis: [ca-rain06 ca-rain17 ca-rain05] > >> port lr1-ls1 > >> mac: "40:44:00:00:01:30" > >> networks: ["192.16.1.1/24"] > >> nat f4675661-f4cc-4f7c-b534-ca75e090ed74 > >> external ip: "10.68.49.184" > >> logical ip: "192.16.1.5" > >> type: "dnat_and_snat" > >> nat f5592262-5fbd-4cef-8773-903875ba34d6 > >> external ip: "10.68.49.185" > >> logical ip: "192.16.1.6" > >> type: "dnat_and_snat" > >> > > Why don't the external ips belong to the subnet - 253.255.80.10/16 ? > > i.e to the network of ls1_external ? > > The 253.255.80.10/16 network is an internal "underlay" Network. An infra > structure network > of the rack product. The "External IPs", are IPs belonging to networks > outside the rack. > > So in Normal case traffic destined for a VM from outside the rack, > would send to the VM > "External IP", that arrives at the rack physical uplink router, and is > sent across the rack > physical network (253.255.0.0/16) to the OVN Gateway, which DNATs and > send to the VM > Logical IP (reverse on traffic from VM to destination outside the rack). > > > > > > I'm pretty sure if you change the external_ips from 10.68.49.184 and > > 10.68.49.185 to > > the ones belonging to 253.255.80.10/16, it would work. > > We can't do that, these are different address spaces in different > physical networks. > I could try adding the 10.68.49.184/185 IPs to the "networks" table in > lr1-ls1_external > > > > I'd suggest trying out with these patches once ? - > > https://urldefense.com/v3/__https://patchwork.ozlabs.org/project/ovn/list/?series=247106__;!!ACWV5N9M2RV99hQ!ZKO2z-ifCaUA-TPeLm7ZP9V7hkX8tZSv4HE4-Ogo2BhBcLfSbibLIh4xDsIiqu4xmH8$ > Ok, will do, are they in master, as I'm running with a fairly recent > build (maybe two weeks old)
The patches are still under review and may not apply cleanly with the tip. You can access it from here too - https://github.com/ovsrobot/ovn/commits/series_247106 Thanks Numan > > Thanks > > > Numan > > > > > >> ovn-nbctl lr-route-list lr1 > >> IPv4 Routes > >> 0.0.0.0/0 253.255.0.1 dst-ip > >> lr1-ls1_external > >> > >> ovn-trace --detailed ls1 'inport == > >> "47433b54-ac10-42f1-ae84-cc6fbb580297" && eth.dst == 40:44:00:00:01:30 > >> && eth.src == 52:54:00:be:06:16 && ip4.src == 192.16.1.6 && ip4.dst == > >> 10.68.49.184 && ip.ttl == 64 && icmp4.type == 8' > >> # > >> icmp,reg14=0x1,vlan_tci=0x0000,dl_src=52:54:00:be:06:16,dl_dst=40:44:00:00:01:30,nw_src=192.16.1.6,nw_dst=10.68.49.184,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0 > >> > >> ingress(dp="ls1", inport="47433b") > >> ---------------------------------- > >> 0. ls_in_port_sec_l2 (ovn-northd.c:4834): inport == "47433b", priority > >> 50, uuid ae50c799 > >> next; > >> 22. ls_in_l2_lkup (ovn-northd.c:7587): eth.dst == 40:44:00:00:01:30, > >> priority 50, uuid c29dec2a > >> outport = "ls1-lr1"; > >> output; > >> > >> egress(dp="ls1", inport="47433b", outport="ls1-lr1") > >> ---------------------------------------------------- > >> 0. ls_out_pre_lb (ovn-northd.c:4980): ip && outport == "ls1-lr1", > >> priority 110, uuid d4d7c7af > >> next; > >> 9. ls_out_port_sec_l2 (ovn-northd.c:4929): outport == "ls1-lr1", > >> priority 50, uuid 36b335f9 > >> output; > >> /* output to "ls1-lr1", type "patch" */ > >> > >> ingress(dp="lr1", inport="lr1-ls1") > >> ----------------------------------- > >> 0. lr_in_admission (ovn-northd.c:9575): eth.dst == 40:44:00:00:01:30 > >> && inport == "lr1-ls1", priority 50, uuid c67387d7 > >> xreg0[0..47] = 40:44:00:00:01:30; > >> next; > >> 1. lr_in_lookup_neighbor (ovn-northd.c:9654): 1, priority 0, uuid > >> c050ede1 > >> reg9[2] = 1; > >> next; > >> 2. lr_in_learn_neighbor (ovn-northd.c:9663): reg9[2] == 1, priority > >> 100, uuid e5780577 > >> next; > >> 10. lr_in_ip_routing (ovn-northd.c:8622): ip4.dst == 0.0.0.0/0, priority > >> 1, uuid 52d001c6 > >> ip.ttl--; > >> reg8[0..15] = 0; > >> reg0 = 253.255.0.1; > >> reg1 = 253.255.80.10; > >> eth.src = 40:44:00:00:01:a0; > >> outport = "lr1-ls1_external"; > >> flags.loopback = 1; > >> next; > >> 11. lr_in_ip_routing_ecmp (ovn-northd.c:9921): reg8[0..15] == 0, > >> priority 150, uuid 920ee40c > >> next; > >> 12. lr_in_policy (ovn-northd.c:10046): 1, priority 0, uuid e2014343 > >> reg8[0..15] = 0; > >> next; > >> 13. lr_in_policy_ecmp (ovn-northd.c:10048): reg8[0..15] == 0, priority > >> 150, uuid ed8c4d4d > >> next; > >> 14. lr_in_arp_resolve (ovn-northd.c:10082): ip4, priority 0, uuid 2cfde30a > >> get_arp(outport, reg0); > >> /* MAC binding to 00:00:0c:07:ac:14. */ > >> next; > >> 17. lr_in_gw_redirect (ovn-northd.c:10598): outport == > >> "lr1-ls1_external", priority 50, uuid 521a9223 > >> outport = "cr-lr1-ls1_external"; > >> next; > >> 18. lr_in_arp_request (ovn-northd.c:10671): 1, priority 0, uuid e43fdfbd > >> output; > >> /* Replacing type "chassisredirect" outport "cr-lr1-ls1_external" > >> with distributed port "lr1-ls1_external". */ > >> > >> egress(dp="lr1", inport="lr1-ls1", outport="lr1-ls1_external") > >> -------------------------------------------------------------- > >> 0. lr_out_undnat (ovn-northd.c:11459): ip && ip4.src == 192.16.1.6 && > >> outport == "lr1-ls1_external" && > >> is_chassis_resident("cr-lr1-ls1_external"), priority 100, uuid e8b081df > >> ct_dnat; > >> > >> ct_dnat /* assuming no un-dnat entry, so no change */ > >> ----------------------------------------------------- > >> 1. lr_out_snat (ovn-northd.c:11552): ip && ip4.src == 192.16.1.6 && > >> outport == "lr1-ls1_external" && > >> is_chassis_resident("cr-lr1-ls1_external"), priority 161, uuid f50e5215 > >> ct_snat(10.68.49.185); > >> > >> ct_snat(ip4.src=10.68.49.185) > >> ----------------------------- > >> 2. lr_out_egr_loop (ovn-northd.c:11846): ip4.dst == 10.68.49.184 && > >> outport == "lr1-ls1_external" && > >> is_chassis_resident("cr-lr1-ls1_external"), priority 100, uuid a6499050 > >> clone { ct_clear; inport = outport; outport = ""; flags = 0; > >> flags.loopback = 1; reg0 = 0; reg1 = 0; reg2 = 0; reg3 = 0; reg4 = 0; > >> reg5 = 0; reg6 = 0; reg7 = 0; reg8 = 0; reg9 = 0; reg9[0] = 1; > >> next(pipeline=ingress, table=0); }; > >> > >> clone > >> ----- > >> ct_clear; > >> inport = outport; > >> outport = ""; > >> flags = 0; > >> flags.loopback = 1; > >> reg0 = 0; > >> reg1 = 0; > >> reg2 = 0; > >> reg3 = 0; > >> reg4 = 0; > >> reg5 = 0; > >> reg6 = 0; > >> reg7 = 0; > >> reg8 = 0; > >> reg9 = 0; > >> reg9[0] = 1; > >> next(pipeline=ingress, table=0); > >> > >> ingress(dp="lr1", inport="lr1-ls1_external") > >> -------------------------------------------- > >> 0. lr_in_admission: no match (implicit drop) > >> > >> If we look at the section of code pointed to by ovn-northd.c:11846 > >> > >> /* Egress Loopback table: For NAT on a distributed router. > >> * If packets in the egress pipeline on the distributed > >> * gateway port have ip.dst matching a NAT external IP, then > >> * loop a clone of the packet back to the beginning of the > >> * ingress pipeline with inport = outport. */ > >> if (od->l3dgw_port) { > >> /* Distributed router. */ > >> ds_clear(match); > >> ds_put_format(match, "ip%s.dst == %s && outport == %s", > >> is_v6 ? "6" : "4", > >> nat->external_ip, > >> od->l3dgw_port->json_key); > >> if (!distributed) { > >> ds_put_format(match, " && is_chassis_resident(%s)", > >> od->l3redirect_port->json_key); > >> } else { > >> ds_put_format(match, " && is_chassis_resident(\"%s\")", > >> nat->logical_port); > >> } > >> ds_clear(actions); > >> ds_put_format(actions, > >> "clone { ct_clear; " > >> "inport = outport; outport = \"\"; " > >> "flags = 0; flags.loopback = 1; "); > >> for (int j = 0; j < MFF_N_LOG_REGS; j++) { > >> ds_put_format(actions, "reg%d = 0; ", j); > >> } > >> ds_put_format(actions, REGBIT_EGRESS_LOOPBACK" = 1; " > >> "next(pipeline=ingress, table=%d); };", > >> ovn_stage_get_table(S_ROUTER_IN_ADMISSION)); > >> ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_EGR_LOOP, > >> 100, > >> ds_cstr(match), ds_cstr(actions), > >> &nat->header_); > >> } > >> > >> It seems clear what the intent is, but the pkt is dropped immediately > >> when returned to the ingress > >> pipeline. Am I missing some config? > >> > >> > >> Thanks Brendan > >> > >> > >> _______________________________________________ > >> discuss mailing list > >> [email protected] > >> https://urldefense.com/v3/__https://mail.openvswitch.org/mailman/listinfo/ovs-discuss__;!!ACWV5N9M2RV99hQ!ZKO2z-ifCaUA-TPeLm7ZP9V7hkX8tZSv4HE4-Ogo2BhBcLfSbibLIh4xDsIi0Md7RaE$ > > _______________________________________________ > discuss mailing list > [email protected] > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss _______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
