On 9/28/26 11:45 AM, Alexandra Rukomoinikova via dev wrote:
> Several ovn-controller instances may run with the same chassis name and
> the same encapsulation IP, an anycast address that is reachable over the
> underlay through ECMP. OVN then sees the whole group of nodes as a
> single chassis: a single Chassis record, a single Encap record and a
> single binding for every port placed on it, while the underlay decides
> which node of the group terminates a given tunnel flow.
> 
> There is no coordination and no leader election between the nodes of the
> group while writing to SBDB: every ``ovn-controller`` independently does the
> full bookkeeping for that one chassis, and it is the idempotence of those
> writes that keeps them from fighting each other.
> 
> The flip side is that the nodes must be configured identically in
> chassis-related params, and that nothing which relies on conntrack can be
> used, because the group shares no runtime state.
> 
> Document all of this, and add a multinode test that deploys two
> gateway nodes as one replicated gateway, gives a hypervisor and a node that
> does not run OVN an underlay ECMP route to the anycast encapsulation IP,
> checks the north-south connectivity through each of the two nodes and
> checks that the controllers do not fight over the shared records.
> 
> Signed-off-by: Alexandra Rukomoinikova <[email protected]>
> ---

Hi Alexandra,

I have a few minor comments on this patch, please see below.

I was thinking of taking care of them myself and squash a small fixup
patch in when pushing yours to main:

https://github.com/dceara/ovn/commit/42b64d5

I didn't test this locally but CI is running in my fork:

https://github.com/dceara/ovn/actions/runs/42295260684

Assuming that passes would you be OK with this incremental change?

Regards,
Dumitru

>  Documentation/automake.mk                   |   1 +
>  Documentation/topics/index.rst              |   1 +
>  Documentation/topics/replicated-gateway.rst | 171 +++++++++++++++++
>  NEWS                                        |   5 +
>  controller/ovn-controller.8.xml             |   9 +-
>  tests/multinode-macros.at                   |  22 ++-
>  tests/multinode.at                          | 197 ++++++++++++++++++++
>  7 files changed, 399 insertions(+), 7 deletions(-)
>  create mode 100644 Documentation/topics/replicated-gateway.rst
> 
> diff --git a/Documentation/automake.mk b/Documentation/automake.mk
> index 685ba9588..b64a0193d 100644
> --- a/Documentation/automake.mk
> +++ b/Documentation/automake.mk
> @@ -25,6 +25,7 @@ DOC_SOURCE = \
>       Documentation/topics/testing.rst \
>       Documentation/topics/test-development.rst \
>       Documentation/topics/high-availability.rst \
> +     Documentation/topics/replicated-gateway.rst \
>       Documentation/topics/dynamic-routing/architecture.rst \
>       Documentation/topics/dynamic-routing/configuration-examples.rst \
>       Documentation/topics/dynamic-routing/index.rst \
> diff --git a/Documentation/topics/index.rst b/Documentation/topics/index.rst
> index 0b676a98b..d5210e03a 100644
> --- a/Documentation/topics/index.rst
> +++ b/Documentation/topics/index.rst
> @@ -40,6 +40,7 @@ OVN
>     dynamic-routing/index
>     incremental-processing/index
>     high-availability
> +   replicated-gateway
>     role-based-access-control
>     vif-plug-providers/index
>     testing
> diff --git a/Documentation/topics/replicated-gateway.rst 
> b/Documentation/topics/replicated-gateway.rst
> new file mode 100644
> index 000000000..c3e43da03
> --- /dev/null
> +++ b/Documentation/topics/replicated-gateway.rst
> @@ -0,0 +1,171 @@
> +..
> +      Licensed under the Apache License, Version 2.0 (the "License"); you may
> +      not use this file except in compliance with the License. You may obtain
> +      a copy of the License at
> +
> +          http://www.apache.org/licenses/LICENSE-2.0
> +
> +      Unless required by applicable law or agreed to in writing, software
> +      distributed under the License is distributed on an "AS IS" BASIS, 
> WITHOUT
> +      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See 
> the
> +      License for the specific language governing permissions and limitations
> +      under the License.
> +
> +      Convention for heading levels in OVN documentation:
> +
> +      =======  Heading 0 (reserved for the title in a document)
> +      -------  Heading 1
> +      ~~~~~~~  Heading 2
> +      +++++++  Heading 3
> +      '''''''  Heading 4
> +
> +      Avoid deeper levels because they do not render well.
> +
> +===================
> +Replicated Gateway
> +===================

Nit: one = too many.

> +
> +A *replicated gateway* is a group of two or more physical nodes that present
> +themselves to OVN as a single chassis. Every node of the group runs its own
> +``ovn-controller``, but all of them are configured with the same chassis name
> +(``external_ids:system-id``) and with the same encapsulation IP
> +(``external_ids:ovn-encap-ip``). The encapsulation IP is an anycast address,
> +it is announced into the underlay (statically or by a routing protocol) by 
> all
> +the nodes of the group, and the underlay load-balances the tunnel traffic
> +between them with ECMP.
> +
> +::
> +
> +                        underlay with ECMP to 10.0.0.1
> +
> +    +-------------------+                       +----------------------+
> +    |                   |  nexthop via gw-1     |  gateway node 1      |
> +    |   hypervisor      +---------------------->|  system-id: vgw      |
> +    |                   |                       |  ovn-encap-ip:       |
> +    |   ovn-encap-ip:   |                       |          10.0.0.1    |
> +    |     192.168.0.10  |                       +----------+-----------+
> +    |                   |                                  |
> +    |                   |  nexthop via gw-2     +----------+-----------+
> +    |                   +---------------------->|  gateway node 2      |
> +    |                   |                       |  system-id: vgw      |
> +    +-------------------+                       |  ovn-encap-ip:       |
> +                                                |          10.0.0.1    |
> +                                                +----------+-----------+
> +                                                           |
> +                                       physical network    |
> +    -------------------------------------------------------+------------
> +
> +From the point of view of the OVN databases there is a single chassis: a 
> single
> +``Chassis`` record, a single ``Encap`` record, and a single binding for every
> +port that is placed on the replicated gateway (for example the 
> chassisredirect
> +port of a distributed gateway port). ``ovn-northd`` and the other
> +``ovn-controller`` instances do not know, and do not need to know, how many
> +physical nodes are behind that chassis. A hypervisor keeps a single tunnel to
> +the anycast encapsulation IP, and the underlay decides which node of the 
> group
> +terminates a given flow.
> +
> +There is no coordination and no leader election between the nodes of the
> +group while writing to SBDB: every ``ovn-controller`` independently does the
> +full bookkeeping for that one chassis, and it is the idempotence of those
> +writes that keeps them from fighting each other.
> +
> +Two consequences follow from this.  First, the columns that OVN derives from
> +the local node rather than from the configuration must be forced to the same
> +value on every node.  More detailed information on this is provided in the
> +``Identical OVS and OVN configuration and features`` section below.  Second,
> +the nodes do not share any runtime state, so nothing that relies on conntrack
> +can be used on a replicated gateway, see the ``Only stateless traffic is
> +supported`` section below.

We shouldn't break in the middle of the verbatim section, that renders
weird.

> +
> +Use case
> +--------
> +This is useful when the capacity or the availability of a single gateway node
> +is not enough: the group scales horizontally, and losing one node only means
> +that the underlay stops using one of the ECMP nexthops, without any change in
> +the OVN databases and without the gateway having to be rescheduled to another
> +chassis.
> +
> +A typical deployment looks as follows:
> +
> +* Two (or more) gateway nodes are installed with the same OVN and OVS
> +  versions and with an identical ``ovn-controller`` configuration. Each of
> +  them configures the anycast encapsulation IP locally, e.g. on the loopback
> +  interface, so that the kernel accepts the tunnel packets destined to it::
> +
> +      $ ovs-vsctl set open . external-ids:system-id=vgw
> +      $ ovs-vsctl set open . external-ids:ovn-encap-ip=10.0.0.1
> +      $ ovs-vsctl set open . external-ids:ovn-encap-type=geneve
> +      $ ip address add 10.0.0.1/32 dev lo
> +
> +* The anycast address is reachable in the underlay through the physical
> +  addresses of all the nodes of the group. On the hypervisors, and on any
> +  other node that has to reach the replicated gateway, this is an ECMP 
> route::
> +
> +      $ ip route replace 10.0.0.1/32 \
> +            nexthop via 192.168.0.1 dev eth0 \
> +            nexthop via 192.168.0.2 dev eth0
> +
> +  In a real deployment the route is normally learned from the fabric, with 
> the
> +  gateway nodes advertising the anycast prefix over BGP.
> +
> +* The logical topology is configured as usual, referring to the chassis by 
> its
> +  single name, e.g.::
> +
> +      $ ovn-nbctl lrp-set-gateway-chassis lr0-public vgw 10
> +
> +Requirements and restrictions
> +-----------------------------
> +
> +The nodes of a replicated gateway are not independent chassis, they are one
> +chassis that happens to run on several machines. Everything that OVN stores
> +per chassis is therefore shared, and everything that OVN computes per chassis
> +must produce the same result on every node of the group. This leads to the
> +following requirements.
> +
> +Identical OVS and OVN configuration and features
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +All the nodes of the group must run the same OVS and OVN versions and must
> +expose the same set of features. ``ovn-controller`` reports the capabilities
> +of the local OVS instance in the ``Chassis`` record (``other_config``, for
> +example the supported datapath interface types, and the various
> +``ovn-chassis-feature`` entries), and ``ovn-northd`` generates the logical
> +flows for the chassis based on those values. If the nodes disagree, the
> +``Chassis`` record will keep flapping between the values reported by each 
> node,
> +``ovn-northd`` will keep recomputing, and the node whose capabilities do not
> +match the ones currently published may not be able to process the flows it
> +receives.
> +
> +For the same reason, every chassis scoped configuration option has to be
> +identical on all the nodes, including, for example: 
> ``external_ids:hostname``,
> +``external_ids:ovn-bridge-mappings`` and all others.
> +
> +Only stateless traffic is supported
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The nodes of a replicated gateway do not share any runtime state, and in
> +particular they do not share conntrack. The underlay picks one of them per
> +flow, and nothing guarantees that the two directions of a connection, or the
> +packets of a connection before and after a change of the ECMP nexthops (a 
> node
> +being added, removed, rebooted or simply an ECMP rehash), are handled by the
> +same node. Any feature that needs the connection state to be on the node that
> +processes the packet is therefore not supported on a replicated gateway:
> +
> +* load balancers,
> +* stateful NAT, that is ``snat`` and ``dnat_and_snat`` (unless configured as
> +  stateless), and NAT with ``ct-commit-all`` enabled,
> +* stateful ACLs, and anything else that relies on conntrack, for example
> +  load balancer hairpinning or ECMP routes with
> +  ``options:ecmp_symmetric_reply`` set.
> +
> +Only stateless forwarding, e.g. plain routing and switching, stateless NAT 
> and
> +stateless ACLs, behaves correctly, because every packet can be handled
> +independently by any node of the group.
> +
> +Example
> +-------
> +
> +The ``ovn multinode replicated gateway`` test in ``tests/multinode.at`` sets 
> up
> +a complete example of a two node replicated gateway with an underlay ECMP 
> route
> +to the anycast encapsulation IP, a hypervisor with a VM behind the gateway
> +and an external node that does not run OVN at all.
> diff --git a/NEWS b/NEWS
> index f1c56dd14..7bc3f2113 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -129,6 +129,11 @@ OVN v26.09.0 - xxx xx xxxx
>       represent a logical router port (e.g. ovn-ic transit switch LSPs).
>       Such ports are treated like type=router, including omission from
>       _MC_flood_l2.
> +   - Documented the "replicated gateway" setup, in which several 
> ovn-controller
> +     instances share one chassis name and one anycast encapsulation IP
> +     reachable over the underlay through ECMP, so that they appear to OVN as
> +     a single chassis. See Documentation/topics/replicated-gateway.rst for 
> the
> +     requirements and the restrictions.
>  
>  OVN v26.03.0 - xxx xx xxxx
>  --------------------------
> diff --git a/controller/ovn-controller.8.xml b/controller/ovn-controller.8.xml
> index 3c33654ff..37c64ebf3 100644
> --- a/controller/ovn-controller.8.xml
> +++ b/controller/ovn-controller.8.xml
> @@ -100,7 +100,14 @@
>        local OVN "etc" directory or via the <code>-n</code> command-line 
> option.
>        The following precedence is used: first, the command-line option is 
> read;
>        if not present, the <code>system-id-override</code> file is read; if 
> not
> -      present, then the name configured in the database is used.</dd>
> +      present, then the name configured in the database is used.
> +      <p>
> +        Several <code>ovn-controller</code> instances running on different
> +        nodes may share the same <code>system-id</code>, so that they appear
> +        to OVN as a single chassis backed by an anycast encapsulation IP.
> +        See the "Replicated Gateway" document for the requirements and the
> +        restrictions of such a setup.
> +      </p></dd>
>  
>        <dt><code>external_ids:hostname</code></dt>
>        <dd>The hostname to use in the Chassis table.</dd>
> diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at
> index 1c85f79f8..81ab2a5b8 100644
> --- a/tests/multinode-macros.at
> +++ b/tests/multinode-macros.at
> @@ -296,23 +296,33 @@ multinode_setup_northd() {
>      m_as $c ovn-sbctl set-connection p${REMOTE_PROT}:6642
>  }
>  
> -# multinode_setup_controller NODE ENCAP_IP REMOTE_IP [ENCAP_TYPE]
> -#
> -# Sets up controller on specified node.
> +# multinode_setup_controller NODE SYSTEM_ID ENCAP_IP REMOTE_IP
> +#                            [ENCAP_TYPE] [START_CONTROLLER]
> +#
> +# Sets up controller on specified node. SYSTEM_ID defaults to NODE; passing a
> +# different value (or the same value on more than one node) allows a test to
> +# control the chassis name the node registers with.  START_CONTROLLER 
> defaults
> +# to "yes"; with "no" only OVS is set up, so that a test can finish the node
> +# configuration before starting ovn-controller itself.
>  multinode_setup_controller() {
>      c=$1
> +    system_id=${2:-$1}
>      encap_ip=$3
>      remote_ip=$4
>      encap_type=${5:-"geneve"}
> +    start_controller=${6:-"yes"}
>  
>      # Cleanup existing one
>      m_as $c /usr/share/openvswitch/scripts/ovs-ctl stop
>      m_as $c /usr/share/ovn/scripts/ovn-ctl stop_controller
>      m_as $c sh -c "rm -f /etc/openvswitch/*.db"
>  
> -    m_as $c /usr/share/openvswitch/scripts/ovs-ctl start --system-id=$c
> -    echo "Using ${CONTROLLER_SSL_ARGS} for ovn-controller".
> -    m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller 
> ${CONTROLLER_SSL_ARGS}
> +    m_as $c /usr/share/openvswitch/scripts/ovs-ctl start 
> --system-id=$system_id
> +    if [[ "$start_controller" = "yes" ]]; then
> +        echo "Using ${CONTROLLER_SSL_ARGS} for ovn-controller".
> +        m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller \
> +            ${CONTROLLER_SSL_ARGS}
> +    fi
I think if we move the part that starts ovn-controller to the end of
this function we don't need the kind of ugly start_controller argument.
We can just set everything we need and then start ovn-controller.

>  
>      m_as $c ovs-vsctl set open . external_ids:ovn-encap-ip=$encap_ip
>      m_as $c ovs-vsctl set open . external-ids:ovn-encap-type=$encap_type
> diff --git a/tests/multinode.at b/tests/multinode.at
> index 0c277f5e8..16cfa804f 100644
> --- a/tests/multinode.at
> +++ b/tests/multinode.at
> @@ -5554,3 +5554,200 @@ AT_CHECK([cat ch2_eth2.tcpdump], [0], [dnl
>  ])
>  
>  AT_CLEANUP
> +
> +AT_SETUP([ovn multinode replicated gateway])
> +
> +check_fake_multinode_setup
> +cleanup_multinode_resources
> +
> +# Two gateway nodes are deployed as one and the same "replicated" gateway: 
> both
> +# ovn-controllers register with the same chassis name and with the same 
> anycast
> +# encap IP. The encap IP is reachable over the underlay via ECMP through the
> +# physical addresses of both gateway nodes, so a geneve tunnel to the 
> replicated
> +# gateway may land on either of them.
> +#
> +#                             underlay (eth1, 170.168.0.0/16)
> +#  ┌───────────────────┐                                   
> ┌──────────────────┐
> +#  │  ovn-chassis-1    │   1.1.1.1/32                      │    ovn-gw-1     
>  │
> +#  │  (compute node)   ├──── nexthop via gw-1 eth1 ────────┤  
> ovn-replicated-gw  │
> +#  │                   │                                   │  encap 1.1.1.1  
>  │
> +#  │  sw0p1 10.0.0.3   ├──── nexthop via gw-2 eth1 ───┐    
> └────────┬─────────┘
> +#  └───────────────────┘                              │             │
> +#                                                     │    
> ┌────────┴─────────┐
> +#  ┌───────────────────┐                              └────┤    ovn-gw-2     
>  │
> +#  │  ovn-chassis-2    │   1.1.1.1/32                      │  
> ovn-replicated-gw  │
> +#  │ (external client) ├──── nexthop via gw-1 eth1 ────────┤  encap 1.1.1.1  
>  │
> +#  │                   ├──── nexthop via gw-2 eth1 ────────┤                 
>  │
> +#  │                   │                                   
> └────────┬─────────┘
> +#  │  eth2 172.20.0.50 ├───────── provider network (eth2, br-ex) ────┘
> +#  └───────────────────┘                       lr0-public 172.20.0.100
> +#
> +# On ovn-chassis-2 (aka client) we have client port in public switch for 
> checking
> +# internal connectivity, public switch is connected to dgp port that claimed 
> by
> +# replicated chassis. Compute and client node both have underlay route for 
> replicated gw ip (1.1.1.1)
> +
> +vgw1=ovn-gw-1
> +vgw2=ovn-gw-2
> +compute=ovn-chassis-1
> +client=ovn-chassis-2
> +vgw_name=ovn-replicated-gw
> +vgw_encap_ip=1.1.1.1
> +
> +m_underlay_ip() {
> +    m_as $1 ip -4 addr show eth1 | grep inet | awk '{print $2}' | cut -d'/' 
> -f1
> +}
> +
> +central_ip=$(m_as ovn-central-az1 ip -4 addr show eth1 | grep inet | awk 
> '{print $2}' | cut -d'/' -f1)

This should now be:

central_ip=$( (m_central_as ip -4 addr show eth1 2>/dev/null ||
m_central_as ip -4 addr show eth2) | grep inet | awk '{print $2}' | cut
-d'/' -f1)

Since we merged 44d3bf28e4b7 ("multinode: Add support for INLINE_BR_EX
configuration of multinode.")

> +vgw1_ip=$(m_underlay_ip $vgw1)
> +vgw2_ip=$(m_underlay_ip $vgw2)
> +compute_ip=$(m_underlay_ip $compute)
> +
> +m_as $compute ip link del sw0p1-p
> +m_as $client ip link del sw0p2-p
> +m_as $client ip link del sw1p1-p
> +
> +# Bring up both gateway nodes as the very same chassis: same chassis name and
> +# same encap IP, which is not part of the underlay address space.
> +for c in $vgw1 $vgw2; do
> +    # Bring the node back to its original chassis name and encap IP.
> +    on_exit "multinode_setup_controller $c $c $(m_underlay_ip $c) 
> $central_ip"
> +    multinode_setup_controller $c $vgw_name $vgw_encap_ip $central_ip geneve 
> no
> +    on_exit "m_as $c ip addr del $vgw_encap_ip/32 dev lo"
> +    check m_as $c ip addr add $vgw_encap_ip/32 dev lo
> +    check m_as $c ovs-vsctl set open . external-ids:hostname=$vgw_name
> +    check m_as $c ovs-vsctl set open . 
> external-ids:ovn-bridge-mappings=public:br-ex

I don't think your test uses the ovn-bridge mapping in any way.  So then
we might as well add hostname as an optional argument to
multinode_setup_controller().

> +done
> +
> +for c in $vgw1 $vgw2; do
> +    m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller 
> ${CONTROLLER_SSL_ARGS}
> +done

If we change multinode_setup_controller() as I mentioned above, we don't
need to manually start it here.

> +
> +# Both gateway nodes must end up sharing a single chassis record with a 
> single
> +# encap record.
> +m_wait_row_count Chassis 1 name=$vgw_name
> +m_wait_row_count Encap 1 chassis_name=$vgw_name
> +m_check_column "$vgw_encap_ip" Encap ip chassis_name=$vgw_name
> +
> +# ECMP route to the replicated gateway encap IP in the underlay, both on the
> +# compute node and on the external client.
> +for c in $compute $client; do
> +    on_exit "m_as $c ip route del $vgw_encap_ip/32"
> +    check m_as $c ip route replace $vgw_encap_ip/32 \
> +        nexthop via $vgw1_ip dev eth1 \
> +        nexthop via $vgw2_ip dev eth1
> +done
> +
> +

Nit: one empty line too many.

> +# The anycast encap IP is reachable over the underlay from both nodes.
> +AT_CHECK([m_as $compute ping -q -c 3 -i 0.3 -w 2 $vgw_encap_ip | 
> FORMAT_PING], [0], [dnl
> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
> +])
> +AT_CHECK([m_as $client ping -q -c 3 -i 0.3 -w 2 $vgw_encap_ip | 
> FORMAT_PING], [0], [dnl
> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
> +])
> +
> +cr_lport_log_count() {
> +    m_as $1 sh -c "cat /var/log/ovn/ovn-controller.log 2>/dev/null | \
> +                   grep -ac 'Claiming lport cr-lr0-public'"
> +}
> +
> +vgw1_cr_logs=$(cr_lport_log_count $vgw1)
> +vgw2_cr_logs=$(cr_lport_log_count $vgw2)
> +
> +check multinode_nbctl ls-add sw0
> +check multinode_nbctl lsp-add sw0 sw0-port1
> +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 
> 10.0.0.3"
> +
> +check multinode_nbctl lr-add lr0
> +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24
> +check multinode_nbctl lsp-add-router-port sw0 sw0-lr0 lr0-sw0
> +
> +check multinode_nbctl ls-add public
> +check multinode_nbctl lsp-add public external_client
> +check multinode_nbctl lsp-set-addresses external_client "50:55:00:00:00:03 
> 172.20.0.99"
> +check multinode_nbctl lrp-add lr0 lr0-public 00:11:22:00:ff:01 
> 172.20.0.100/24
> +check multinode_nbctl lsp-add-router-port public public-lr0 lr0-public
> +check multinode_nbctl lrp-set-gateway-chassis lr0-public $vgw_name 10
> +
> +m_as $compute /data/create_fake_vm.sh sw0-port1 sw0p1 50:54:00:00:00:03 1342 
> 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a
> +m_as $client /data/create_fake_vm.sh external_client external 
> 50:55:00:00:00:03 1342 172.20.0.99 24 172.20.0.100 1001::3/64 1001::a
> +
> +check multinode_nbctl --wait=hv sync
> +m_wait_for_ports_up sw0-port1
> +m_wait_for_ports_up external_client
> +
> +# The distributed gateway port is claimed by the replicated gateway chassis.
> +m_wait_row_count Port_Binding 1 logical_port=cr-lr0-public
> +vgw_uuid=$(m_central_as ovn-sbctl --bare --columns _uuid find Chassis 
> name=$vgw_name)
> +m_wait_column "$vgw_uuid" Port_Binding chassis logical_port=cr-lr0-public
> +
> +# The two controllers share a single Chassis record and a single binding for
> +# the chassisredirect port.  Since both nodes are configured identically and
> +# both resolve to the same chassis, each of them finds the records already
> +# holding the values it was about to write and its transaction is a no-op, so
> +# once the setup has settled no transaction touching those tables is 
> expected.
> +# If the nodes fought over a record, they would add one transaction per main
> +# loop iteration to the southbound database log.
> +sb_txn_count() {
> +    m_central_as grep -ac "\"$1\":" /etc/ovn/ovnsb_db.db
> +}
> +
> +chassis_txns=$(sb_txn_count Chassis)
> +pb_txns=$(sb_txn_count Port_Binding)
> +sleep 5
> +check test $(($(sb_txn_count Chassis) - chassis_txns)) -le 1
> +check test $(($(sb_txn_count Port_Binding) - pb_txns)) -le 1
> +
> +cr_logs=$(($(cr_lport_log_count $vgw1) + $(cr_lport_log_count $vgw2)))
> +# Claim on both
> +check test $((cr_logs - vgw1_cr_logs - vgw2_cr_logs)) -le 2
> +
> +check multinode_nbctl --wait=hv sync
> +
> +# North-south connectivity through the replicated gateway, with the underlay 
> ECMP
> +# route in place on both ends.
> +check_ns_connectivity() {
> +    M_NS_CHECK_EXEC([$compute], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 
> 172.20.0.99 | FORMAT_PING], \
> +[0], [dnl
> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
> +])
> +
> +    M_NS_CHECK_EXEC([$compute], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 
> 172.20.0.100 | FORMAT_PING], \
> +[0], [dnl
> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
> +])
> +
> +    M_NS_CHECK_EXEC([$client], [external], [ping -q -c 3 -i 0.3 -w 2 
> 10.0.0.3 | FORMAT_PING], \
> +[0], [dnl
> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
> +])
> +}
> +
> +# The traffic towards the replicated gateway must be tunneled to the anycast 
> encap IP.
> +M_START_TCPDUMPS([$compute], [-c 2 -neei eth1 host $vgw_encap_ip and udp 
> port 6081], [ch1_underlay])
> +
> +check_ns_connectivity
> +
> +OVS_WAIT_UNTIL([test -s ch1_underlay.tcpdump])
> +AT_CHECK([grep -q "$vgw_encap_ip" ch1_underlay.tcpdump])
> +
> +# Each of the two replicated gateway nodes on its own must be able to carry 
> the
> +# traffic: pin the underlay route to a single nexthop at a time.
> +for nexthop in $vgw1_ip $vgw2_ip; do
> +    for c in $compute $client; do
> +        check m_as $c ip route replace $vgw_encap_ip/32 via $nexthop dev eth1
> +    done
> +
> +    check_ns_connectivity
> +done
> +
> +# Back to ECMP over both gateway nodes.
> +for c in $compute $client; do
> +    check m_as $c ip route replace $vgw_encap_ip/32 \
> +        nexthop via $vgw1_ip dev eth1 \
> +        nexthop via $vgw2_ip dev eth1
> +done
> +
> +check_ns_connectivity
> +
> +AT_CLEANUP

_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Reply via email to