From: Takao Yatagai <[email protected]> A load balancer on a distributed router without a distributed gateway port gets its lr_in_dnat flows on every chassis, but no un-DNAT: only gateway routers and routers with a DGP get one. Replies leave the router with the backend as source, so the load balancer does not work.
Add, on such a router, the same priority 120 lr_out_undnat flow that routers with a DGP get, without the DGP and chassis terms. The replies are un-DNATed on whichever chassis they re-enter the router. The CMS must make sure, through its logical topology, that this is the chassis that load balanced the request, for example because a downstream router translates the source of the requests with a distributed NAT rule (logical_port + external_mac). Replies that re-enter the router on another chassis find no conntrack entry and are left as they are, as before this change. Keeping the load balancer and the SNAT on different routers also keeps the configuration compatible with NB_Global use_common_zone. Assisted-by: Claude Opus 5.5 Signed-off-by: Takao Yatagai <[email protected]> --- Documentation/ref/ovn-logical-flows.7.rst | 13 ++++ NEWS | 4 ++ northd/northd.c | 9 +++ ovn-nb.xml | 48 ++++++++++++++ tests/ovn-northd.at | 80 +++++++++++++++++++++++ 5 files changed, 154 insertions(+) diff --git a/Documentation/ref/ovn-logical-flows.7.rst b/Documentation/ref/ovn-logical-flows.7.rst index 913ebbc83..b43e1eb16 100644 --- a/Documentation/ref/ovn-logical-flows.7.rst +++ b/Documentation/ref/ovn-logical-flows.7.rst @@ -3789,6 +3789,19 @@ Egress Table 1: UNDNAT on Distributed Routers If the router is configured to force SNAT any load-balanced packets, above action will be replaced by ``flags.force_snat_for_lb = 1; ct_dnat;``. +- For each VIP of the load balancing rules of a router without a gateway port + (other than a Gateway router), a priority-120 flow is programmed on every + chassis that matches ``IP && ((IP.src == B && P.src == PORT) || ...)``, + where *IP* is ``ip4`` or ``ip6``, for the backend addresses *B* and, if + configured, L4 ports *PORT* of protocol *P* of the ``VIP``, with an action + ``ct_dnat;``. The match does not include a gateway port or + ``is_chassis_resident``: the replies are unDNATed on the chassis where they + re-enter the router, which the CMS must make sure is the chassis that load + balanced the request. If the load balancer has ``options:skip_snat`` set, + the action is ``flags.skip_snat_for_lb = 1; ct_dnat;``. With + ``use_common_zone`` set in ``NB_Global``, ``ct_dnat_in_czone`` is used + instead of ``ct_dnat``. + - For each configuration in the OVN Northbound database that asks to change the destination IP address of a packet from an IP address of *A* to *B*, a priority-100 flow matches ``ip && ip4.src == B && outport == GW``, where *GW* diff --git a/NEWS b/NEWS index 6415fb973..37e253aac 100644 --- a/NEWS +++ b/NEWS @@ -62,6 +62,10 @@ Post v26.09.0 table at startup; it defers its kernel route updates until the initial dump of the advertised routes of the local route-exchange datapaths is complete. + - A distributed router without a distributed gateway port now un-DNATs + the replies from the backends of its load balancers. The CMS must make + sure that the replies re-enter the router on the chassis that load + balanced the request. See ovn-nb(5). OVN v26.09.0 - xxx xx xxxx -------------------------- diff --git a/northd/northd.c b/northd/northd.c index f10ca9eab..ed2cdbaf2 100644 --- a/northd/northd.c +++ b/northd/northd.c @@ -13989,6 +13989,15 @@ build_lrouter_nat_flows_for_lb( if (vector_is_empty(&od->l3dgw_ports)) { bitmap_set1(gw_dp_bitmap[type], index); + /* A distributed router without a DGP load balances on every + * chassis, so un-DNAT the replies on whichever chassis they + * re-enter the router. The CMS must make sure that it is the + * chassis that load balanced the request. Gateway routers + * un-DNAT all traffic in a separate flow. */ + if (!od->is_gw_router && !vector_is_empty(&lb_vip->backends)) { + build_lrouter_lb_undnat_flow(&ctx, type, od, NULL, NULL, + lb_dps->lflow_ref); + } } else { /* Create stateless LB NAT rules when using multiple DGPs and * use_stateless_nat is true. diff --git a/ovn-nb.xml b/ovn-nb.xml index c69905a68..0d18010aa 100644 --- a/ovn-nb.xml +++ b/ovn-nb.xml @@ -2609,6 +2609,50 @@ </p> </column> + <group title="Load Balancing on Logical Routers"> + <p> + A load balancer associated with a logical router (see the + <ref column="load_balancer" table="Logical_Router"/> and + <ref column="load_balancer_group" table="Logical_Router"/> columns of + the <ref table="Logical_Router"/> table) DNATs requests to its VIPs in + the router's ingress pipeline and un-DNATs the replies from its + backends in the router's egress pipeline. Both use connection + tracking, so a request and its replies must pass the router on the + same chassis. How that is ensured depends on the router: + </p> + + <ul> + <li> + A gateway router is bound to one chassis, so all of its traffic + passes that chassis. + </li> + + <li> + On a distributed router with a distributed gateway port (DGP), + traffic through the DGP is load balanced and un-DNATed on the + gateway chassis of the DGP, unless the load balancer sets + <ref column="options" key="distributed"/>. See the + <ref column="load_balancer" table="Logical_Router"/> column for + routers with more than one DGP. + </li> + + <li> + A distributed router without a DGP load balances requests on the + chassis where they enter the router, and un-DNATs replies on the + chassis where they re-enter it. OVN does not make sure that this + is the same chassis: the CMS must, through its logical topology. + For example, a downstream router can translate the source of the + requests with a distributed NAT rule (see the + <ref column="logical_port" table="NAT"/> and + <ref column="external_mac" table="NAT"/> columns of the + <ref table="NAT"/> table), so that the replies are only routed to + the chassis of that NAT rule's logical port. Replies that re-enter + the router on another chassis are not un-DNATed, and leave it with + the backend's address as their source. + </li> + </ul> + </group> + <group title="Health Checks"> <p> OVN supports health checks for load balancer endpoints. When health @@ -3314,6 +3358,10 @@ or balancer in scenarios with DGPs associated with different gateway chassis (e.g. ECMP routes), consider using the <code>use_stateless_nat</code> option to <code>true</code> in the load balancer options column. + Load balancers also work on distributed routers without a DGP, if the + CMS makes sure that the replies re-enter the router on the chassis that + load balanced the request. See ``Load Balancing on Logical Routers'' + in the <ref table="Load_Balancer"/> table. </column> <column name="load_balancer_group"> diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at index d42ea34ae..62f72a12c 100644 --- a/tests/ovn-northd.at +++ b/tests/ovn-northd.at @@ -25837,3 +25837,83 @@ AT_CHECK([ovn-sbctl lflow-list lr0 | grep "lr_in_policy[[^_]]" | ovn_strip_lflow OVN_CLEANUP_NORTHD AT_CLEANUP ]) + +OVN_FOR_EACH_NORTHD_NO_HV([ +AT_SETUP([Load balancer un-DNAT on a router without DGP]) +ovn_start + +check ovn-nbctl ls-add sw0 +check ovn-nbctl lsp-add sw0 sw0-port1 \ + -- lsp-set-addresses sw0-port1 "50:54:00:00:00:03 10.0.0.3" + +check ovn-nbctl lr-add lr0 +check ovn-nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24 +check ovn-nbctl lsp-add sw0 sw0-lr0 \ + -- lsp-set-type sw0-lr0 router \ + -- lsp-set-addresses sw0-lr0 router \ + -- lsp-set-options sw0-lr0 router-port=lr0-sw0 + +check ovn-nbctl lb-add lb0 172.168.0.200:80 10.0.0.4:8080 +check ovn-nbctl lr-lb-add lr0 lb0 +check ovn-nbctl --wait=sb sync + +# The router load balances in lr_in_dnat on every chassis and un-DNATs the +# backend's replies on every chassis: no DGP or chassis terms. +ovn-sbctl dump-flows lr0 > lr0flows +AT_CAPTURE_FILE([lr0flows]) +AT_CHECK([grep "lr_in_dnat" lr0flows | grep "ip4.dst == 172.168.0.200" | \ + grep -c "ct_lb_mark(backends=10.0.0.4:8080)"], [0], [1 +]) +AT_CHECK([grep "lr_in_dnat" lr0flows | grep "ip4.dst == 172.168.0.200" | \ + grep -c "is_chassis_resident"], [1], [0 +]) +AT_CHECK([grep "lr_out_undnat" lr0flows | grep "priority=120" | \ + grep "ip4.src == 10.0.0.4 && tcp.src == 8080" | \ + grep -c "action=(ct_dnat;)"], [0], [1 +]) +AT_CHECK([grep "lr_out_undnat" lr0flows | grep "10.0.0.4" | \ + grep -c "is_chassis_resident\|inport\|outport"], [1], [0 +]) + +# With use_common_zone the un-DNAT uses the common zone, like the LB's +# own flows on routers with a distributed gateway port. +check ovn-nbctl --wait=sb set NB_Global . options:use_common_zone=true +ovn-sbctl dump-flows lr0 > lr0flows +AT_CHECK([grep "lr_out_undnat" lr0flows | grep "priority=120" | \ + grep "ip4.src == 10.0.0.4 && tcp.src == 8080" | \ + grep -c "action=(ct_dnat_in_czone;)"], [0], [1 +]) +check ovn-nbctl --wait=sb remove NB_Global . options use_common_zone + +# skip_snat keeps its flag, as on routers with a distributed gateway port. +check ovn-nbctl --wait=sb set load_balancer lb0 options:skip_snat=true +ovn-sbctl dump-flows lr0 > lr0flows +AT_CHECK([grep "lr_out_undnat" lr0flows | grep "priority=120" | \ + grep "ip4.src == 10.0.0.4 && tcp.src == 8080" | \ + grep -c "action=(flags.skip_snat_for_lb = 1; ct_dnat;)"], [0], [1 +]) +check ovn-nbctl --wait=sb remove load_balancer lb0 options skip_snat + +# With a DGP, the flow is bound to the DGP and its chassis instead. +check ovn-nbctl lrp-add lr0 lr0-public 00:00:00:00:ff:02 172.168.0.1/24 +check ovn-nbctl --wait=sb lrp-set-gateway-chassis lr0-public gw1 +ovn-sbctl dump-flows lr0 > lr0flows +AT_CHECK([grep "lr_out_undnat" lr0flows | grep "priority=120" | \ + grep "ip4.src == 10.0.0.4 && tcp.src == 8080" | \ + grep "outport == \"lr0-public\"" | \ + grep -c "is_chassis_resident(\"cr-lr0-public\")"], [0], [1 +]) +AT_CHECK([grep "lr_out_undnat" lr0flows | grep "priority=120" | \ + grep -c "10.0.0.4"], [0], [1 +]) +check ovn-nbctl --wait=sb lrp-del lr0-public + +# Removing the load balancer removes the un-DNAT flow. +check ovn-nbctl --wait=sb lr-lb-del lr0 lb0 +ovn-sbctl dump-flows lr0 > lr0flows +AT_CHECK([grep "lr_out_undnat" lr0flows | grep -c "10.0.0.4"], [1], [0 +]) + +OVN_CLEANUP_NORTHD +AT_CLEANUP +]) -- 2.55.0 _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
