On 29/06/2021 21:38, Numan Siddique wrote:
On Tue, Jun 29, 2021 at 4:13 PM Brendan Doyle <[email protected]> wrote:
Hi,

With a very simple notwork (two VMs on different chassis), 1 subnet,
single LS and
LR/Gateway. The two VMs can ping each other using their Logical IPs.
Each has an
"External IP", and each can be accessed from an external network on that
external IP.
BUT they can't ping each other using their external IPs. I would have
expected that
either:

a) The packets are sent on the external net then hairpinned back to the OVN
       gateway by the external net router.

b) They are hairpinned by OVN.

It seems that OVN attempts the latter, but does not succeed. The
details, NB network,
and pkt trace are as follows:

ovn-nbctl show
switch 2710eebe-f2b3-49e4-bcd6-dcfa48ed6470 (ls1_external)
      port ln-ls1_external
          type: localnet
          addresses: ["unknown"]
      port ls1_external-lr1
          type: router
          router-port: lr1-ls1_external

switch ff909b16-d863-4e3d-a10b-2f0010f17b23 (ls1)
      port 47433b54-ac10-42f1-ae84-cc6fbb580297
          addresses: ["52:54:00:be:06:16 192.16.1.6"]
      port 00bff7c0-2e2d-41ba-9485-3b5fa9801365
          addresses: ["52:54:00:e6:4f:46 192.16.1.5"]
      port ls1-lr1
          type: router
          router-port: lr1-ls1

router 63e1b6a2-327f-4a24-b0c9-3a0e951beb2b (lr1)
      port lr1-ls1_external
          mac: "40:44:00:00:01:a0"
          networks: ["253.255.80.10/16"]
          gateway chassis: [ca-rain06 ca-rain17 ca-rain05]
      port lr1-ls1
          mac: "40:44:00:00:01:30"
          networks: ["192.16.1.1/24"]
      nat f4675661-f4cc-4f7c-b534-ca75e090ed74
          external ip: "10.68.49.184"
          logical ip: "192.16.1.5"
          type: "dnat_and_snat"
      nat f5592262-5fbd-4cef-8773-903875ba34d6
          external ip: "10.68.49.185"
          logical ip: "192.16.1.6"
          type: "dnat_and_snat"

Why don't the external ips belong to the subnet - 253.255.80.10/16 ?
i.e to the network of ls1_external ?

The 253.255.80.10/16 network is an internal "underlay" Network. An infra structure network of the rack product. The "External IPs", are IPs belonging to networks outside the rack.

So in Normal case traffic  destined for a VM from outside the rack, would send to the VM "External IP", that arrives at the rack physical uplink router, and is sent across the rack physical network (253.255.0.0/16) to the OVN Gateway, which DNATs and send to the VM
Logical IP (reverse on traffic from VM to destination outside the rack).



I'm pretty sure if you change the external_ips from 10.68.49.184 and
10.68.49.185 to
the ones belonging to 253.255.80.10/16, it would work.

We can't do that, these are different address spaces in different physical networks. I could try adding the 10.68.49.184/185 IPs to the "networks" table in lr1-ls1_external

I'd suggest trying out with these patches once ? -
https://urldefense.com/v3/__https://patchwork.ozlabs.org/project/ovn/list/?series=247106__;!!ACWV5N9M2RV99hQ!ZKO2z-ifCaUA-TPeLm7ZP9V7hkX8tZSv4HE4-Ogo2BhBcLfSbibLIh4xDsIiqu4xmH8$
Ok, will do, are they in master, as I'm running with a fairly recent build (maybe two weeks old)

Thanks

Numan


ovn-nbctl lr-route-list lr1
IPv4 Routes
                  0.0.0.0/0               253.255.0.1 dst-ip lr1-ls1_external

ovn-trace --detailed ls1 'inport ==
"47433b54-ac10-42f1-ae84-cc6fbb580297" && eth.dst == 40:44:00:00:01:30
&& eth.src == 52:54:00:be:06:16 && ip4.src == 192.16.1.6 && ip4.dst ==
10.68.49.184 && ip.ttl == 64 && icmp4.type == 8'
#
icmp,reg14=0x1,vlan_tci=0x0000,dl_src=52:54:00:be:06:16,dl_dst=40:44:00:00:01:30,nw_src=192.16.1.6,nw_dst=10.68.49.184,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0

ingress(dp="ls1", inport="47433b")
----------------------------------
   0. ls_in_port_sec_l2 (ovn-northd.c:4834): inport == "47433b", priority
50, uuid ae50c799
      next;
22. ls_in_l2_lkup (ovn-northd.c:7587): eth.dst == 40:44:00:00:01:30,
priority 50, uuid c29dec2a
      outport = "ls1-lr1";
      output;

egress(dp="ls1", inport="47433b", outport="ls1-lr1")
----------------------------------------------------
   0. ls_out_pre_lb (ovn-northd.c:4980): ip && outport == "ls1-lr1",
priority 110, uuid d4d7c7af
      next;
   9. ls_out_port_sec_l2 (ovn-northd.c:4929): outport == "ls1-lr1",
priority 50, uuid 36b335f9
      output;
      /* output to "ls1-lr1", type "patch" */

ingress(dp="lr1", inport="lr1-ls1")
-----------------------------------
   0. lr_in_admission (ovn-northd.c:9575): eth.dst == 40:44:00:00:01:30
&& inport == "lr1-ls1", priority 50, uuid c67387d7
      xreg0[0..47] = 40:44:00:00:01:30;
      next;
   1. lr_in_lookup_neighbor (ovn-northd.c:9654): 1, priority 0, uuid c050ede1
      reg9[2] = 1;
      next;
   2. lr_in_learn_neighbor (ovn-northd.c:9663): reg9[2] == 1, priority
100, uuid e5780577
      next;
10. lr_in_ip_routing (ovn-northd.c:8622): ip4.dst == 0.0.0.0/0, priority
1, uuid 52d001c6
      ip.ttl--;
      reg8[0..15] = 0;
      reg0 = 253.255.0.1;
      reg1 = 253.255.80.10;
      eth.src = 40:44:00:00:01:a0;
      outport = "lr1-ls1_external";
      flags.loopback = 1;
      next;
11. lr_in_ip_routing_ecmp (ovn-northd.c:9921): reg8[0..15] == 0,
priority 150, uuid 920ee40c
      next;
12. lr_in_policy (ovn-northd.c:10046): 1, priority 0, uuid e2014343
      reg8[0..15] = 0;
      next;
13. lr_in_policy_ecmp (ovn-northd.c:10048): reg8[0..15] == 0, priority
150, uuid ed8c4d4d
      next;
14. lr_in_arp_resolve (ovn-northd.c:10082): ip4, priority 0, uuid 2cfde30a
      get_arp(outport, reg0);
      /* MAC binding to 00:00:0c:07:ac:14. */
      next;
17. lr_in_gw_redirect (ovn-northd.c:10598): outport ==
"lr1-ls1_external", priority 50, uuid 521a9223
      outport = "cr-lr1-ls1_external";
      next;
18. lr_in_arp_request (ovn-northd.c:10671): 1, priority 0, uuid e43fdfbd
      output;
      /* Replacing type "chassisredirect" outport "cr-lr1-ls1_external"
with distributed port "lr1-ls1_external". */

egress(dp="lr1", inport="lr1-ls1", outport="lr1-ls1_external")
--------------------------------------------------------------
   0. lr_out_undnat (ovn-northd.c:11459): ip && ip4.src == 192.16.1.6 &&
outport == "lr1-ls1_external" &&
is_chassis_resident("cr-lr1-ls1_external"), priority 100, uuid e8b081df
      ct_dnat;

ct_dnat /* assuming no un-dnat entry, so no change */
-----------------------------------------------------
   1. lr_out_snat (ovn-northd.c:11552): ip && ip4.src == 192.16.1.6 &&
outport == "lr1-ls1_external" &&
is_chassis_resident("cr-lr1-ls1_external"), priority 161, uuid f50e5215
      ct_snat(10.68.49.185);

ct_snat(ip4.src=10.68.49.185)
-----------------------------
   2. lr_out_egr_loop (ovn-northd.c:11846): ip4.dst == 10.68.49.184 &&
outport == "lr1-ls1_external" &&
is_chassis_resident("cr-lr1-ls1_external"), priority 100, uuid a6499050
      clone { ct_clear; inport = outport; outport = ""; flags = 0;
flags.loopback = 1; reg0 = 0; reg1 = 0; reg2 = 0; reg3 = 0; reg4 = 0;
reg5 = 0; reg6 = 0; reg7 = 0; reg8 = 0; reg9 = 0; reg9[0] = 1;
next(pipeline=ingress, table=0); };

clone
-----
      ct_clear;
      inport = outport;
      outport = "";
      flags = 0;
      flags.loopback = 1;
      reg0 = 0;
      reg1 = 0;
      reg2 = 0;
      reg3 = 0;
      reg4 = 0;
      reg5 = 0;
      reg6 = 0;
      reg7 = 0;
      reg8 = 0;
      reg9 = 0;
      reg9[0] = 1;
      next(pipeline=ingress, table=0);

ingress(dp="lr1", inport="lr1-ls1_external")
--------------------------------------------
   0. lr_in_admission: no match (implicit drop)

If we look at the section of code pointed to by ovn-northd.c:11846

         /* Egress Loopback table: For NAT on a distributed router.
           * If packets in the egress pipeline on the distributed
           * gateway port have ip.dst matching a NAT external IP, then
           * loop a clone of the packet back to the beginning of the
           * ingress pipeline with inport = outport. */
          if (od->l3dgw_port) {
              /* Distributed router. */
              ds_clear(match);
              ds_put_format(match, "ip%s.dst == %s && outport == %s",
                            is_v6 ? "6" : "4",
                            nat->external_ip,
                            od->l3dgw_port->json_key);
              if (!distributed) {
                  ds_put_format(match, " && is_chassis_resident(%s)",
                                od->l3redirect_port->json_key);
              } else {
                  ds_put_format(match, " && is_chassis_resident(\"%s\")",
                                nat->logical_port);
              }
             ds_clear(actions);
              ds_put_format(actions,
                            "clone { ct_clear; "
                            "inport = outport; outport = \"\"; "
                            "flags = 0; flags.loopback = 1; ");
              for (int j = 0; j < MFF_N_LOG_REGS; j++) {
                  ds_put_format(actions, "reg%d = 0; ", j);
              }
              ds_put_format(actions, REGBIT_EGRESS_LOOPBACK" = 1; "
                            "next(pipeline=ingress, table=%d); };",
ovn_stage_get_table(S_ROUTER_IN_ADMISSION));
              ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_EGR_LOOP, 100,
                                      ds_cstr(match), ds_cstr(actions),
                                      &nat->header_);
          }

It seems clear what the intent is, but the pkt is dropped immediately
when returned to the ingress
pipeline. Am I missing some config?


Thanks Brendan


_______________________________________________
discuss mailing list
[email protected]
https://urldefense.com/v3/__https://mail.openvswitch.org/mailman/listinfo/ovs-discuss__;!!ACWV5N9M2RV99hQ!ZKO2z-ifCaUA-TPeLm7ZP9V7hkX8tZSv4HE4-Ogo2BhBcLfSbibLIh4xDsIi0Md7RaE$

_______________________________________________
discuss mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Reply via email to