On Wed, Jun 30, 2021 at 10:54 AM Brendan Doyle <[email protected]> wrote: > > > OK so the simple 1 line change to northd.c in: > > [ovs-dev,v8,1/6] northd: Swap src and dst eth addresses in router > egress loop. > > fixes the problem, can access all external networks, and the haripin > between 10.68.49.185 <-> 10.68.49.185 > works. Thumbs up for me on this patch! >
Thanks for testing out. I applied that patch to the main branch and backported to branch-21.06. Thanks Numan > > On 30/06/2021 10:11, Brendan Doyle wrote: > > So If I do : > > > > ovn-nbctl add logical_router_port lr1-ls1_external networks > > "10.68.49.185/32 10.68.49.184/32" > > > > Then the hairpin works and I have connectivity between 10.68.49.185 > > <-> 10.68.49.185 > > > > But This patch also look promising: > > [ovs-dev,v8,1/6] northd: Swap src and dst eth addresses in router > > egress loop. > > > > I'll try adding this, and incrementally the other patches in the series. > > > > Brendan > > > > > > On 29/06/2021 22:40, Numan Siddique wrote: > >> On Tue, Jun 29, 2021 at 5:06 PM Brendan Doyle > >> <[email protected]> wrote: > >>> > >>> > >>> On 29/06/2021 21:38, Numan Siddique wrote: > >>>> On Tue, Jun 29, 2021 at 4:13 PM Brendan Doyle > >>>> <[email protected]> wrote: > >>>>> Hi, > >>>>> > >>>>> With a very simple notwork (two VMs on different chassis), 1 subnet, > >>>>> single LS and > >>>>> LR/Gateway. The two VMs can ping each other using their Logical IPs. > >>>>> Each has an > >>>>> "External IP", and each can be accessed from an external network > >>>>> on that > >>>>> external IP. > >>>>> BUT they can't ping each other using their external IPs. I would have > >>>>> expected that > >>>>> either: > >>>>> > >>>>> a) The packets are sent on the external net then hairpinned back > >>>>> to the OVN > >>>>> gateway by the external net router. > >>>>> > >>>>> b) They are hairpinned by OVN. > >>>>> > >>>>> It seems that OVN attempts the latter, but does not succeed. The > >>>>> details, NB network, > >>>>> and pkt trace are as follows: > >>>>> > >>>>> ovn-nbctl show > >>>>> switch 2710eebe-f2b3-49e4-bcd6-dcfa48ed6470 (ls1_external) > >>>>> port ln-ls1_external > >>>>> type: localnet > >>>>> addresses: ["unknown"] > >>>>> port ls1_external-lr1 > >>>>> type: router > >>>>> router-port: lr1-ls1_external > >>>>> > >>>>> switch ff909b16-d863-4e3d-a10b-2f0010f17b23 (ls1) > >>>>> port 47433b54-ac10-42f1-ae84-cc6fbb580297 > >>>>> addresses: ["52:54:00:be:06:16 192.16.1.6"] > >>>>> port 00bff7c0-2e2d-41ba-9485-3b5fa9801365 > >>>>> addresses: ["52:54:00:e6:4f:46 192.16.1.5"] > >>>>> port ls1-lr1 > >>>>> type: router > >>>>> router-port: lr1-ls1 > >>>>> > >>>>> router 63e1b6a2-327f-4a24-b0c9-3a0e951beb2b (lr1) > >>>>> port lr1-ls1_external > >>>>> mac: "40:44:00:00:01:a0" > >>>>> networks: ["253.255.80.10/16"] > >>>>> gateway chassis: [ca-rain06 ca-rain17 ca-rain05] > >>>>> port lr1-ls1 > >>>>> mac: "40:44:00:00:01:30" > >>>>> networks: ["192.16.1.1/24"] > >>>>> nat f4675661-f4cc-4f7c-b534-ca75e090ed74 > >>>>> external ip: "10.68.49.184" > >>>>> logical ip: "192.16.1.5" > >>>>> type: "dnat_and_snat" > >>>>> nat f5592262-5fbd-4cef-8773-903875ba34d6 > >>>>> external ip: "10.68.49.185" > >>>>> logical ip: "192.16.1.6" > >>>>> type: "dnat_and_snat" > >>>>> > >>>> Why don't the external ips belong to the subnet - 253.255.80.10/16 ? > >>>> i.e to the network of ls1_external ? > >>> The 253.255.80.10/16 network is an internal "underlay" Network. An > >>> infra > >>> structure network > >>> of the rack product. The "External IPs", are IPs belonging to networks > >>> outside the rack. > >>> > >>> So in Normal case traffic destined for a VM from outside the rack, > >>> would send to the VM > >>> "External IP", that arrives at the rack physical uplink router, and is > >>> sent across the rack > >>> physical network (253.255.0.0/16) to the OVN Gateway, which DNATs and > >>> send to the VM > >>> Logical IP (reverse on traffic from VM to destination outside the > >>> rack). > >>> > >>> > >>>> I'm pretty sure if you change the external_ips from 10.68.49.184 and > >>>> 10.68.49.185 to > >>>> the ones belonging to 253.255.80.10/16, it would work. > >>> We can't do that, these are different address spaces in different > >>> physical networks. > >>> I could try adding the 10.68.49.184/185 IPs to the "networks" table in > >>> lr1-ls1_external > >>>> I'd suggest trying out with these patches once ? - > >>>> https://urldefense.com/v3/__https://patchwork.ozlabs.org/project/ovn/list/?series=247106__;!!ACWV5N9M2RV99hQ!ZKO2z-ifCaUA-TPeLm7ZP9V7hkX8tZSv4HE4-Ogo2BhBcLfSbibLIh4xDsIiqu4xmH8$ > >>>> > >>> Ok, will do, are they in master, as I'm running with a fairly recent > >>> build (maybe two weeks old) > >> The patches are still under review and may not apply cleanly with the > >> tip. You can access it from here too - > >> https://urldefense.com/v3/__https://github.com/ovsrobot/ovn/commits/series_247106__;!!ACWV5N9M2RV99hQ!e3YISaySCgi6qg3Y-8_gdx0IN_FeVsl5onOgkxhBhhgp_69r8PTAROpeu3yG3eaPN0c$ > >> > >> > >> Thanks > >> Numan > >> > >>> Thanks > >>> > >>>> Numan > >>>> > >>>> > >>>>> ovn-nbctl lr-route-list lr1 > >>>>> IPv4 Routes > >>>>> 0.0.0.0/0 253.255.0.1 dst-ip > >>>>> lr1-ls1_external > >>>>> > >>>>> ovn-trace --detailed ls1 'inport == > >>>>> "47433b54-ac10-42f1-ae84-cc6fbb580297" && eth.dst == > >>>>> 40:44:00:00:01:30 > >>>>> && eth.src == 52:54:00:be:06:16 && ip4.src == 192.16.1.6 && > >>>>> ip4.dst == > >>>>> 10.68.49.184 && ip.ttl == 64 && icmp4.type == 8' > >>>>> # > >>>>> icmp,reg14=0x1,vlan_tci=0x0000,dl_src=52:54:00:be:06:16,dl_dst=40:44:00:00:01:30,nw_src=192.16.1.6,nw_dst=10.68.49.184,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0 > >>>>> > >>>>> > >>>>> ingress(dp="ls1", inport="47433b") > >>>>> ---------------------------------- > >>>>> 0. ls_in_port_sec_l2 (ovn-northd.c:4834): inport == "47433b", > >>>>> priority > >>>>> 50, uuid ae50c799 > >>>>> next; > >>>>> 22. ls_in_l2_lkup (ovn-northd.c:7587): eth.dst == 40:44:00:00:01:30, > >>>>> priority 50, uuid c29dec2a > >>>>> outport = "ls1-lr1"; > >>>>> output; > >>>>> > >>>>> egress(dp="ls1", inport="47433b", outport="ls1-lr1") > >>>>> ---------------------------------------------------- > >>>>> 0. ls_out_pre_lb (ovn-northd.c:4980): ip && outport == "ls1-lr1", > >>>>> priority 110, uuid d4d7c7af > >>>>> next; > >>>>> 9. ls_out_port_sec_l2 (ovn-northd.c:4929): outport == "ls1-lr1", > >>>>> priority 50, uuid 36b335f9 > >>>>> output; > >>>>> /* output to "ls1-lr1", type "patch" */ > >>>>> > >>>>> ingress(dp="lr1", inport="lr1-ls1") > >>>>> ----------------------------------- > >>>>> 0. lr_in_admission (ovn-northd.c:9575): eth.dst == > >>>>> 40:44:00:00:01:30 > >>>>> && inport == "lr1-ls1", priority 50, uuid c67387d7 > >>>>> xreg0[0..47] = 40:44:00:00:01:30; > >>>>> next; > >>>>> 1. lr_in_lookup_neighbor (ovn-northd.c:9654): 1, priority 0, > >>>>> uuid c050ede1 > >>>>> reg9[2] = 1; > >>>>> next; > >>>>> 2. lr_in_learn_neighbor (ovn-northd.c:9663): reg9[2] == 1, > >>>>> priority > >>>>> 100, uuid e5780577 > >>>>> next; > >>>>> 10. lr_in_ip_routing (ovn-northd.c:8622): ip4.dst == 0.0.0.0/0, > >>>>> priority > >>>>> 1, uuid 52d001c6 > >>>>> ip.ttl--; > >>>>> reg8[0..15] = 0; > >>>>> reg0 = 253.255.0.1; > >>>>> reg1 = 253.255.80.10; > >>>>> eth.src = 40:44:00:00:01:a0; > >>>>> outport = "lr1-ls1_external"; > >>>>> flags.loopback = 1; > >>>>> next; > >>>>> 11. lr_in_ip_routing_ecmp (ovn-northd.c:9921): reg8[0..15] == 0, > >>>>> priority 150, uuid 920ee40c > >>>>> next; > >>>>> 12. lr_in_policy (ovn-northd.c:10046): 1, priority 0, uuid e2014343 > >>>>> reg8[0..15] = 0; > >>>>> next; > >>>>> 13. lr_in_policy_ecmp (ovn-northd.c:10048): reg8[0..15] == 0, > >>>>> priority > >>>>> 150, uuid ed8c4d4d > >>>>> next; > >>>>> 14. lr_in_arp_resolve (ovn-northd.c:10082): ip4, priority 0, uuid > >>>>> 2cfde30a > >>>>> get_arp(outport, reg0); > >>>>> /* MAC binding to 00:00:0c:07:ac:14. */ > >>>>> next; > >>>>> 17. lr_in_gw_redirect (ovn-northd.c:10598): outport == > >>>>> "lr1-ls1_external", priority 50, uuid 521a9223 > >>>>> outport = "cr-lr1-ls1_external"; > >>>>> next; > >>>>> 18. lr_in_arp_request (ovn-northd.c:10671): 1, priority 0, uuid > >>>>> e43fdfbd > >>>>> output; > >>>>> /* Replacing type "chassisredirect" outport > >>>>> "cr-lr1-ls1_external" > >>>>> with distributed port "lr1-ls1_external". */ > >>>>> > >>>>> egress(dp="lr1", inport="lr1-ls1", outport="lr1-ls1_external") > >>>>> -------------------------------------------------------------- > >>>>> 0. lr_out_undnat (ovn-northd.c:11459): ip && ip4.src == > >>>>> 192.16.1.6 && > >>>>> outport == "lr1-ls1_external" && > >>>>> is_chassis_resident("cr-lr1-ls1_external"), priority 100, uuid > >>>>> e8b081df > >>>>> ct_dnat; > >>>>> > >>>>> ct_dnat /* assuming no un-dnat entry, so no change */ > >>>>> ----------------------------------------------------- > >>>>> 1. lr_out_snat (ovn-northd.c:11552): ip && ip4.src == > >>>>> 192.16.1.6 && > >>>>> outport == "lr1-ls1_external" && > >>>>> is_chassis_resident("cr-lr1-ls1_external"), priority 161, uuid > >>>>> f50e5215 > >>>>> ct_snat(10.68.49.185); > >>>>> > >>>>> ct_snat(ip4.src=10.68.49.185) > >>>>> ----------------------------- > >>>>> 2. lr_out_egr_loop (ovn-northd.c:11846): ip4.dst == > >>>>> 10.68.49.184 && > >>>>> outport == "lr1-ls1_external" && > >>>>> is_chassis_resident("cr-lr1-ls1_external"), priority 100, uuid > >>>>> a6499050 > >>>>> clone { ct_clear; inport = outport; outport = ""; flags = 0; > >>>>> flags.loopback = 1; reg0 = 0; reg1 = 0; reg2 = 0; reg3 = 0; reg4 = 0; > >>>>> reg5 = 0; reg6 = 0; reg7 = 0; reg8 = 0; reg9 = 0; reg9[0] = 1; > >>>>> next(pipeline=ingress, table=0); }; > >>>>> > >>>>> clone > >>>>> ----- > >>>>> ct_clear; > >>>>> inport = outport; > >>>>> outport = ""; > >>>>> flags = 0; > >>>>> flags.loopback = 1; > >>>>> reg0 = 0; > >>>>> reg1 = 0; > >>>>> reg2 = 0; > >>>>> reg3 = 0; > >>>>> reg4 = 0; > >>>>> reg5 = 0; > >>>>> reg6 = 0; > >>>>> reg7 = 0; > >>>>> reg8 = 0; > >>>>> reg9 = 0; > >>>>> reg9[0] = 1; > >>>>> next(pipeline=ingress, table=0); > >>>>> > >>>>> ingress(dp="lr1", inport="lr1-ls1_external") > >>>>> -------------------------------------------- > >>>>> 0. lr_in_admission: no match (implicit drop) > >>>>> > >>>>> If we look at the section of code pointed to by ovn-northd.c:11846 > >>>>> > >>>>> /* Egress Loopback table: For NAT on a distributed router. > >>>>> * If packets in the egress pipeline on the distributed > >>>>> * gateway port have ip.dst matching a NAT external IP, > >>>>> then > >>>>> * loop a clone of the packet back to the beginning of the > >>>>> * ingress pipeline with inport = outport. */ > >>>>> if (od->l3dgw_port) { > >>>>> /* Distributed router. */ > >>>>> ds_clear(match); > >>>>> ds_put_format(match, "ip%s.dst == %s && outport == > >>>>> %s", > >>>>> is_v6 ? "6" : "4", > >>>>> nat->external_ip, > >>>>> od->l3dgw_port->json_key); > >>>>> if (!distributed) { > >>>>> ds_put_format(match, " && > >>>>> is_chassis_resident(%s)", > >>>>> od->l3redirect_port->json_key); > >>>>> } else { > >>>>> ds_put_format(match, " && > >>>>> is_chassis_resident(\"%s\")", > >>>>> nat->logical_port); > >>>>> } > >>>>> ds_clear(actions); > >>>>> ds_put_format(actions, > >>>>> "clone { ct_clear; " > >>>>> "inport = outport; outport = \"\"; " > >>>>> "flags = 0; flags.loopback = 1; "); > >>>>> for (int j = 0; j < MFF_N_LOG_REGS; j++) { > >>>>> ds_put_format(actions, "reg%d = 0; ", j); > >>>>> } > >>>>> ds_put_format(actions, REGBIT_EGRESS_LOOPBACK" = 1; " > >>>>> "next(pipeline=ingress, table=%d); };", > >>>>> ovn_stage_get_table(S_ROUTER_IN_ADMISSION)); > >>>>> ovn_lflow_add_with_hint(lflows, od, > >>>>> S_ROUTER_OUT_EGR_LOOP, 100, > >>>>> ds_cstr(match), > >>>>> ds_cstr(actions), > >>>>> &nat->header_); > >>>>> } > >>>>> > >>>>> It seems clear what the intent is, but the pkt is dropped immediately > >>>>> when returned to the ingress > >>>>> pipeline. Am I missing some config? > >>>>> > >>>>> > >>>>> Thanks Brendan > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> discuss mailing list > >>>>> [email protected] > >>>>> https://urldefense.com/v3/__https://mail.openvswitch.org/mailman/listinfo/ovs-discuss__;!!ACWV5N9M2RV99hQ!ZKO2z-ifCaUA-TPeLm7ZP9V7hkX8tZSv4HE4-Ogo2BhBcLfSbibLIh4xDsIi0Md7RaE$ > >>>>> > >>> _______________________________________________ > >>> discuss mailing list > >>> [email protected] > >>> https://urldefense.com/v3/__https://mail.openvswitch.org/mailman/listinfo/ovs-discuss__;!!ACWV5N9M2RV99hQ!e3YISaySCgi6qg3Y-8_gdx0IN_FeVsl5onOgkxhBhhgp_69r8PTAROpeu3yGTHgIUMg$ > >>> > > > > _______________________________________________ > > discuss mailing list > > [email protected] > > https://urldefense.com/v3/__https://mail.openvswitch.org/mailman/listinfo/ovs-discuss__;!!ACWV5N9M2RV99hQ!Z3k6-fDQJqwNn8Agn4ngwEJJy4wHaYT_i3fUlmeofW97TlsXsnMU3UMjNkbzwhW9YBM$ > > > _______________________________________________ > discuss mailing list > [email protected] > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss _______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
