On 07.11.2026 00:33, Dumitru Ceara wrote:
> On 9/28/26 11:45 AM, Alexandra Rukomoinikova via dev wrote:
>> Several ovn-controller instances may run with the same chassis name and
>> the same encapsulation IP, an anycast address that is reachable over the
>> underlay through ECMP. OVN then sees the whole group of nodes as a
>> single chassis: a single Chassis record, a single Encap record and a
>> single binding for every port placed on it, while the underlay decides
>> which node of the group terminates a given tunnel flow.
>>
>> There is no coordination and no leader election between the nodes of the
>> group while writing to SBDB: every ``ovn-controller`` independently does the
>> full bookkeeping for that one chassis, and it is the idempotence of those
>> writes that keeps them from fighting each other.
>>
>> The flip side is that the nodes must be configured identically in
>> chassis-related params, and that nothing which relies on conntrack can be
>> used, because the group shares no runtime state.
>>
>> Document all of this, and add a multinode test that deploys two
>> gateway nodes as one replicated gateway, gives a hypervisor and a node that
>> does not run OVN an underlay ECMP route to the anycast encapsulation IP,
>> checks the north-south connectivity through each of the two nodes and
>> checks that the controllers do not fight over the shared records.
>>
>> Signed-off-by: Alexandra Rukomoinikova <[email protected]>
>> ---
> Hi Alexandra,
>
> I have a few minor comments on this patch, please see below.
>
> I was thinking of taking care of them myself and squash a small fixup
> patch in when pushing yours to main:
>
> https://github.com/dceara/ovn/commit/42b64d5
>
> I didn't test this locally but CI is running in my fork:
>
> https://github.com/dceara/ovn/actions/runs/42295260684
>
> Assuming that passes would you be OK with this incremental change?
>
> Regards,
> Dumitru


Hi Dumitru! I took a fresh look at this patch, and I’d like to remove 
all references to ECMP → anycast IP and drop the mention of using BGP 
for announcements - this seems unnecessary. Is it still okay to make 
those changes rn? I can take the fixup based on fixups in your fork, 
make the changes on top of it, and send you the updated patch. šŸ™‚

thanks, sasha


>
>>   Documentation/automake.mk                   |   1 +
>>   Documentation/topics/index.rst              |   1 +
>>   Documentation/topics/replicated-gateway.rst | 171 +++++++++++++++++
>>   NEWS                                        |   5 +
>>   controller/ovn-controller.8.xml             |   9 +-
>>   tests/multinode-macros.at                   |  22 ++-
>>   tests/multinode.at                          | 197 ++++++++++++++++++++
>>   7 files changed, 399 insertions(+), 7 deletions(-)
>>   create mode 100644 Documentation/topics/replicated-gateway.rst
>>
>> diff --git a/Documentation/automake.mk b/Documentation/automake.mk
>> index 685ba9588..b64a0193d 100644
>> --- a/Documentation/automake.mk
>> +++ b/Documentation/automake.mk
>> @@ -25,6 +25,7 @@ DOC_SOURCE = \
>>      Documentation/topics/testing.rst \
>>      Documentation/topics/test-development.rst \
>>      Documentation/topics/high-availability.rst \
>> +    Documentation/topics/replicated-gateway.rst \
>>      Documentation/topics/dynamic-routing/architecture.rst \
>>      Documentation/topics/dynamic-routing/configuration-examples.rst \
>>      Documentation/topics/dynamic-routing/index.rst \
>> diff --git a/Documentation/topics/index.rst b/Documentation/topics/index.rst
>> index 0b676a98b..d5210e03a 100644
>> --- a/Documentation/topics/index.rst
>> +++ b/Documentation/topics/index.rst
>> @@ -40,6 +40,7 @@ OVN
>>      dynamic-routing/index
>>      incremental-processing/index
>>      high-availability
>> +   replicated-gateway
>>      role-based-access-control
>>      vif-plug-providers/index
>>      testing
>> diff --git a/Documentation/topics/replicated-gateway.rst 
>> b/Documentation/topics/replicated-gateway.rst
>> new file mode 100644
>> index 000000000..c3e43da03
>> --- /dev/null
>> +++ b/Documentation/topics/replicated-gateway.rst
>> @@ -0,0 +1,171 @@
>> +..
>> +      Licensed under the Apache License, Version 2.0 (the "License"); you 
>> may
>> +      not use this file except in compliance with the License. You may 
>> obtain
>> +      a copy of the License at
>> +
>> +          http://www.apache.org/licenses/LICENSE-2.0
>> +
>> +      Unless required by applicable law or agreed to in writing, software
>> +      distributed under the License is distributed on an "AS IS" BASIS, 
>> WITHOUT
>> +      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See 
>> the
>> +      License for the specific language governing permissions and 
>> limitations
>> +      under the License.
>> +
>> +      Convention for heading levels in OVN documentation:
>> +
>> +      =======  Heading 0 (reserved for the title in a document)
>> +      -------  Heading 1
>> +      ~~~~~~~  Heading 2
>> +      +++++++  Heading 3
>> +      '''''''  Heading 4
>> +
>> +      Avoid deeper levels because they do not render well.
>> +
>> +===================
>> +Replicated Gateway
>> +===================
> Nit: one = too many.
>
>> +
>> +A *replicated gateway* is a group of two or more physical nodes that present
>> +themselves to OVN as a single chassis. Every node of the group runs its own
>> +``ovn-controller``, but all of them are configured with the same chassis 
>> name
>> +(``external_ids:system-id``) and with the same encapsulation IP
>> +(``external_ids:ovn-encap-ip``). The encapsulation IP is an anycast address,
>> +it is announced into the underlay (statically or by a routing protocol) by 
>> all
>> +the nodes of the group, and the underlay load-balances the tunnel traffic
>> +between them with ECMP.
>> +
>> +::
>> +
>> +                        underlay with ECMP to 10.0.0.1
>> +
>> +    +-------------------+                       +----------------------+
>> +    |                   |  nexthop via gw-1     |  gateway node 1      |
>> +    |   hypervisor      +---------------------->|  system-id: vgw      |
>> +    |                   |                       |  ovn-encap-ip:       |
>> +    |   ovn-encap-ip:   |                       |          10.0.0.1    |
>> +    |     192.168.0.10  |                       +----------+-----------+
>> +    |                   |                                  |
>> +    |                   |  nexthop via gw-2     +----------+-----------+
>> +    |                   +---------------------->|  gateway node 2      |
>> +    |                   |                       |  system-id: vgw      |
>> +    +-------------------+                       |  ovn-encap-ip:       |
>> +                                                |          10.0.0.1    |
>> +                                                +----------+-----------+
>> +                                                           |
>> +                                       physical network    |
>> +    -------------------------------------------------------+------------
>> +
>> +From the point of view of the OVN databases there is a single chassis: a 
>> single
>> +``Chassis`` record, a single ``Encap`` record, and a single binding for 
>> every
>> +port that is placed on the replicated gateway (for example the 
>> chassisredirect
>> +port of a distributed gateway port). ``ovn-northd`` and the other
>> +``ovn-controller`` instances do not know, and do not need to know, how many
>> +physical nodes are behind that chassis. A hypervisor keeps a single tunnel 
>> to
>> +the anycast encapsulation IP, and the underlay decides which node of the 
>> group
>> +terminates a given flow.
>> +
>> +There is no coordination and no leader election between the nodes of the
>> +group while writing to SBDB: every ``ovn-controller`` independently does the
>> +full bookkeeping for that one chassis, and it is the idempotence of those
>> +writes that keeps them from fighting each other.
>> +
>> +Two consequences follow from this.  First, the columns that OVN derives from
>> +the local node rather than from the configuration must be forced to the same
>> +value on every node.  More detailed information on this is provided in the
>> +``Identical OVS and OVN configuration and features`` section below.  Second,
>> +the nodes do not share any runtime state, so nothing that relies on 
>> conntrack
>> +can be used on a replicated gateway, see the ``Only stateless traffic is
>> +supported`` section below.
> We shouldn't break in the middle of the verbatim section, that renders
> weird.
>
>> +
>> +Use case
>> +--------
>> +This is useful when the capacity or the availability of a single gateway 
>> node
>> +is not enough: the group scales horizontally, and losing one node only means
>> +that the underlay stops using one of the ECMP nexthops, without any change 
>> in
>> +the OVN databases and without the gateway having to be rescheduled to 
>> another
>> +chassis.
>> +
>> +A typical deployment looks as follows:
>> +
>> +* Two (or more) gateway nodes are installed with the same OVN and OVS
>> +  versions and with an identical ``ovn-controller`` configuration. Each of
>> +  them configures the anycast encapsulation IP locally, e.g. on the loopback
>> +  interface, so that the kernel accepts the tunnel packets destined to it::
>> +
>> +      $ ovs-vsctl set open . external-ids:system-id=vgw
>> +      $ ovs-vsctl set open . external-ids:ovn-encap-ip=10.0.0.1
>> +      $ ovs-vsctl set open . external-ids:ovn-encap-type=geneve
>> +      $ ip address add 10.0.0.1/32 dev lo
>> +
>> +* The anycast address is reachable in the underlay through the physical
>> +  addresses of all the nodes of the group. On the hypervisors, and on any
>> +  other node that has to reach the replicated gateway, this is an ECMP 
>> route::
>> +
>> +      $ ip route replace 10.0.0.1/32 \
>> +            nexthop via 192.168.0.1 dev eth0 \
>> +            nexthop via 192.168.0.2 dev eth0
>> +
>> +  In a real deployment the route is normally learned from the fabric, with 
>> the
>> +  gateway nodes advertising the anycast prefix over BGP.
>> +
>> +* The logical topology is configured as usual, referring to the chassis by 
>> its
>> +  single name, e.g.::
>> +
>> +      $ ovn-nbctl lrp-set-gateway-chassis lr0-public vgw 10
>> +
>> +Requirements and restrictions
>> +-----------------------------
>> +
>> +The nodes of a replicated gateway are not independent chassis, they are one
>> +chassis that happens to run on several machines. Everything that OVN stores
>> +per chassis is therefore shared, and everything that OVN computes per 
>> chassis
>> +must produce the same result on every node of the group. This leads to the
>> +following requirements.
>> +
>> +Identical OVS and OVN configuration and features
>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +All the nodes of the group must run the same OVS and OVN versions and must
>> +expose the same set of features. ``ovn-controller`` reports the capabilities
>> +of the local OVS instance in the ``Chassis`` record (``other_config``, for
>> +example the supported datapath interface types, and the various
>> +``ovn-chassis-feature`` entries), and ``ovn-northd`` generates the logical
>> +flows for the chassis based on those values. If the nodes disagree, the
>> +``Chassis`` record will keep flapping between the values reported by each 
>> node,
>> +``ovn-northd`` will keep recomputing, and the node whose capabilities do not
>> +match the ones currently published may not be able to process the flows it
>> +receives.
>> +
>> +For the same reason, every chassis scoped configuration option has to be
>> +identical on all the nodes, including, for example: 
>> ``external_ids:hostname``,
>> +``external_ids:ovn-bridge-mappings`` and all others.
>> +
>> +Only stateless traffic is supported
>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +The nodes of a replicated gateway do not share any runtime state, and in
>> +particular they do not share conntrack. The underlay picks one of them per
>> +flow, and nothing guarantees that the two directions of a connection, or the
>> +packets of a connection before and after a change of the ECMP nexthops (a 
>> node
>> +being added, removed, rebooted or simply an ECMP rehash), are handled by the
>> +same node. Any feature that needs the connection state to be on the node 
>> that
>> +processes the packet is therefore not supported on a replicated gateway:
>> +
>> +* load balancers,
>> +* stateful NAT, that is ``snat`` and ``dnat_and_snat`` (unless configured as
>> +  stateless), and NAT with ``ct-commit-all`` enabled,
>> +* stateful ACLs, and anything else that relies on conntrack, for example
>> +  load balancer hairpinning or ECMP routes with
>> +  ``options:ecmp_symmetric_reply`` set.
>> +
>> +Only stateless forwarding, e.g. plain routing and switching, stateless NAT 
>> and
>> +stateless ACLs, behaves correctly, because every packet can be handled
>> +independently by any node of the group.
>> +
>> +Example
>> +-------
>> +
>> +The ``ovn multinode replicated gateway`` test in ``tests/multinode.at`` 
>> sets up
>> +a complete example of a two node replicated gateway with an underlay ECMP 
>> route
>> +to the anycast encapsulation IP, a hypervisor with a VM behind the gateway
>> +and an external node that does not run OVN at all.
>> diff --git a/NEWS b/NEWS
>> index f1c56dd14..7bc3f2113 100644
>> --- a/NEWS
>> +++ b/NEWS
>> @@ -129,6 +129,11 @@ OVN v26.09.0 - xxx xx xxxx
>>        represent a logical router port (e.g. ovn-ic transit switch LSPs).
>>        Such ports are treated like type=router, including omission from
>>        _MC_flood_l2.
>> +   - Documented the "replicated gateway" setup, in which several 
>> ovn-controller
>> +     instances share one chassis name and one anycast encapsulation IP
>> +     reachable over the underlay through ECMP, so that they appear to OVN as
>> +     a single chassis. See Documentation/topics/replicated-gateway.rst for 
>> the
>> +     requirements and the restrictions.
>>   
>>   OVN v26.03.0 - xxx xx xxxx
>>   --------------------------
>> diff --git a/controller/ovn-controller.8.xml 
>> b/controller/ovn-controller.8.xml
>> index 3c33654ff..37c64ebf3 100644
>> --- a/controller/ovn-controller.8.xml
>> +++ b/controller/ovn-controller.8.xml
>> @@ -100,7 +100,14 @@
>>         local OVN "etc" directory or via the <code>-n</code> command-line 
>> option.
>>         The following precedence is used: first, the command-line option is 
>> read;
>>         if not present, the <code>system-id-override</code> file is read; if 
>> not
>> -      present, then the name configured in the database is used.</dd>
>> +      present, then the name configured in the database is used.
>> +      <p>
>> +        Several <code>ovn-controller</code> instances running on different
>> +        nodes may share the same <code>system-id</code>, so that they appear
>> +        to OVN as a single chassis backed by an anycast encapsulation IP.
>> +        See the "Replicated Gateway" document for the requirements and the
>> +        restrictions of such a setup.
>> +      </p></dd>
>>   
>>         <dt><code>external_ids:hostname</code></dt>
>>         <dd>The hostname to use in the Chassis table.</dd>
>> diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at
>> index 1c85f79f8..81ab2a5b8 100644
>> --- a/tests/multinode-macros.at
>> +++ b/tests/multinode-macros.at
>> @@ -296,23 +296,33 @@ multinode_setup_northd() {
>>       m_as $c ovn-sbctl set-connection p${REMOTE_PROT}:6642
>>   }
>>   
>> -# multinode_setup_controller NODE ENCAP_IP REMOTE_IP [ENCAP_TYPE]
>> -#
>> -# Sets up controller on specified node.
>> +# multinode_setup_controller NODE SYSTEM_ID ENCAP_IP REMOTE_IP
>> +#                            [ENCAP_TYPE] [START_CONTROLLER]
>> +#
>> +# Sets up controller on specified node. SYSTEM_ID defaults to NODE; passing 
>> a
>> +# different value (or the same value on more than one node) allows a test to
>> +# control the chassis name the node registers with.  START_CONTROLLER 
>> defaults
>> +# to "yes"; with "no" only OVS is set up, so that a test can finish the node
>> +# configuration before starting ovn-controller itself.
>>   multinode_setup_controller() {
>>       c=$1
>> +    system_id=${2:-$1}
>>       encap_ip=$3
>>       remote_ip=$4
>>       encap_type=${5:-"geneve"}
>> +    start_controller=${6:-"yes"}
>>   
>>       # Cleanup existing one
>>       m_as $c /usr/share/openvswitch/scripts/ovs-ctl stop
>>       m_as $c /usr/share/ovn/scripts/ovn-ctl stop_controller
>>       m_as $c sh -c "rm -f /etc/openvswitch/*.db"
>>   
>> -    m_as $c /usr/share/openvswitch/scripts/ovs-ctl start --system-id=$c
>> -    echo "Using ${CONTROLLER_SSL_ARGS} for ovn-controller".
>> -    m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller 
>> ${CONTROLLER_SSL_ARGS}
>> +    m_as $c /usr/share/openvswitch/scripts/ovs-ctl start 
>> --system-id=$system_id
>> +    if [[ "$start_controller" = "yes" ]]; then
>> +        echo "Using ${CONTROLLER_SSL_ARGS} for ovn-controller".
>> +        m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller \
>> +            ${CONTROLLER_SSL_ARGS}
>> +    fi
> I think if we move the part that starts ovn-controller to the end of
> this function we don't need the kind of ugly start_controller argument.
> We can just set everything we need and then start ovn-controller.
>
>>   
>>       m_as $c ovs-vsctl set open . external_ids:ovn-encap-ip=$encap_ip
>>       m_as $c ovs-vsctl set open . external-ids:ovn-encap-type=$encap_type
>> diff --git a/tests/multinode.at b/tests/multinode.at
>> index 0c277f5e8..16cfa804f 100644
>> --- a/tests/multinode.at
>> +++ b/tests/multinode.at
>> @@ -5554,3 +5554,200 @@ AT_CHECK([cat ch2_eth2.tcpdump], [0], [dnl
>>   ])
>>   
>>   AT_CLEANUP
>> +
>> +AT_SETUP([ovn multinode replicated gateway])
>> +
>> +check_fake_multinode_setup
>> +cleanup_multinode_resources
>> +
>> +# Two gateway nodes are deployed as one and the same "replicated" gateway: 
>> both
>> +# ovn-controllers register with the same chassis name and with the same 
>> anycast
>> +# encap IP. The encap IP is reachable over the underlay via ECMP through the
>> +# physical addresses of both gateway nodes, so a geneve tunnel to the 
>> replicated
>> +# gateway may land on either of them.
>> +#
>> +#                             underlay (eth1, 170.168.0.0/16)
>> +#  ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”                                   
>> ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”
>> +#  │  ovn-chassis-1    │   1.1.1.1/32                      │    ovn-gw-1    
>>   │
>> +#  │  (compute node)   ā”œā”€ā”€ā”€ā”€ nexthop via gw-1 eth1 ────────┤  
>> ovn-replicated-gw  │
>> +#  │                   │                                   │  encap 1.1.1.1 
>>   │
>> +#  │  sw0p1 10.0.0.3   ā”œā”€ā”€ā”€ā”€ nexthop via gw-2 eth1 ───┐    
>> ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜
>> +#  ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜                              │             │
>> +#                                                     │    
>> ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”“ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”
>> +#  ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”                              └────┤    ovn-gw-2    
>>   │
>> +#  │  ovn-chassis-2    │   1.1.1.1/32                      │  
>> ovn-replicated-gw  │
>> +#  │ (external client) ā”œā”€ā”€ā”€ā”€ nexthop via gw-1 eth1 ────────┤  encap 1.1.1.1 
>>   │
>> +#  │                   ā”œā”€ā”€ā”€ā”€ nexthop via gw-2 eth1 ────────┤                
>>   │
>> +#  │                   │                                   
>> ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜
>> +#  │  eth2 172.20.0.50 ā”œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ provider network (eth2, br-ex) ā”€ā”€ā”€ā”€ā”˜
>> +#  ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜                       lr0-public 172.20.0.100
>> +#
>> +# On ovn-chassis-2 (aka client) we have client port in public switch for 
>> checking
>> +# internal connectivity, public switch is connected to dgp port that 
>> claimed by
>> +# replicated chassis. Compute and client node both have underlay route for 
>> replicated gw ip (1.1.1.1)
>> +
>> +vgw1=ovn-gw-1
>> +vgw2=ovn-gw-2
>> +compute=ovn-chassis-1
>> +client=ovn-chassis-2
>> +vgw_name=ovn-replicated-gw
>> +vgw_encap_ip=1.1.1.1
>> +
>> +m_underlay_ip() {
>> +    m_as $1 ip -4 addr show eth1 | grep inet | awk '{print $2}' | cut -d'/' 
>> -f1
>> +}
>> +
>> +central_ip=$(m_as ovn-central-az1 ip -4 addr show eth1 | grep inet | awk 
>> '{print $2}' | cut -d'/' -f1)
> This should now be:
>
> central_ip=$( (m_central_as ip -4 addr show eth1 2>/dev/null ||
> m_central_as ip -4 addr show eth2) | grep inet | awk '{print $2}' | cut
> -d'/' -f1)
>
> Since we merged 44d3bf28e4b7 ("multinode: Add support for INLINE_BR_EX
> configuration of multinode.")
>
>> +vgw1_ip=$(m_underlay_ip $vgw1)
>> +vgw2_ip=$(m_underlay_ip $vgw2)
>> +compute_ip=$(m_underlay_ip $compute)
>> +
>> +m_as $compute ip link del sw0p1-p
>> +m_as $client ip link del sw0p2-p
>> +m_as $client ip link del sw1p1-p
>> +
>> +# Bring up both gateway nodes as the very same chassis: same chassis name 
>> and
>> +# same encap IP, which is not part of the underlay address space.
>> +for c in $vgw1 $vgw2; do
>> +    # Bring the node back to its original chassis name and encap IP.
>> +    on_exit "multinode_setup_controller $c $c $(m_underlay_ip $c) 
>> $central_ip"
>> +    multinode_setup_controller $c $vgw_name $vgw_encap_ip $central_ip 
>> geneve no
>> +    on_exit "m_as $c ip addr del $vgw_encap_ip/32 dev lo"
>> +    check m_as $c ip addr add $vgw_encap_ip/32 dev lo
>> +    check m_as $c ovs-vsctl set open . external-ids:hostname=$vgw_name
>> +    check m_as $c ovs-vsctl set open . 
>> external-ids:ovn-bridge-mappings=public:br-ex
> I don't think your test uses the ovn-bridge mapping in any way.  So then
> we might as well add hostname as an optional argument to
> multinode_setup_controller().
>
>> +done
>> +
>> +for c in $vgw1 $vgw2; do
>> +    m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller 
>> ${CONTROLLER_SSL_ARGS}
>> +done
> If we change multinode_setup_controller() as I mentioned above, we don't
> need to manually start it here.
>
>> +
>> +# Both gateway nodes must end up sharing a single chassis record with a 
>> single
>> +# encap record.
>> +m_wait_row_count Chassis 1 name=$vgw_name
>> +m_wait_row_count Encap 1 chassis_name=$vgw_name
>> +m_check_column "$vgw_encap_ip" Encap ip chassis_name=$vgw_name
>> +
>> +# ECMP route to the replicated gateway encap IP in the underlay, both on the
>> +# compute node and on the external client.
>> +for c in $compute $client; do
>> +    on_exit "m_as $c ip route del $vgw_encap_ip/32"
>> +    check m_as $c ip route replace $vgw_encap_ip/32 \
>> +        nexthop via $vgw1_ip dev eth1 \
>> +        nexthop via $vgw2_ip dev eth1
>> +done
>> +
>> +
> Nit: one empty line too many.
>
>> +# The anycast encap IP is reachable over the underlay from both nodes.
>> +AT_CHECK([m_as $compute ping -q -c 3 -i 0.3 -w 2 $vgw_encap_ip | 
>> FORMAT_PING], [0], [dnl
>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
>> +])
>> +AT_CHECK([m_as $client ping -q -c 3 -i 0.3 -w 2 $vgw_encap_ip | 
>> FORMAT_PING], [0], [dnl
>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
>> +])
>> +
>> +cr_lport_log_count() {
>> +    m_as $1 sh -c "cat /var/log/ovn/ovn-controller.log 2>/dev/null | \
>> +                   grep -ac 'Claiming lport cr-lr0-public'"
>> +}
>> +
>> +vgw1_cr_logs=$(cr_lport_log_count $vgw1)
>> +vgw2_cr_logs=$(cr_lport_log_count $vgw2)
>> +
>> +check multinode_nbctl ls-add sw0
>> +check multinode_nbctl lsp-add sw0 sw0-port1
>> +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 
>> 10.0.0.3"
>> +
>> +check multinode_nbctl lr-add lr0
>> +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24
>> +check multinode_nbctl lsp-add-router-port sw0 sw0-lr0 lr0-sw0
>> +
>> +check multinode_nbctl ls-add public
>> +check multinode_nbctl lsp-add public external_client
>> +check multinode_nbctl lsp-set-addresses external_client "50:55:00:00:00:03 
>> 172.20.0.99"
>> +check multinode_nbctl lrp-add lr0 lr0-public 00:11:22:00:ff:01 
>> 172.20.0.100/24
>> +check multinode_nbctl lsp-add-router-port public public-lr0 lr0-public
>> +check multinode_nbctl lrp-set-gateway-chassis lr0-public $vgw_name 10
>> +
>> +m_as $compute /data/create_fake_vm.sh sw0-port1 sw0p1 50:54:00:00:00:03 
>> 1342 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a
>> +m_as $client /data/create_fake_vm.sh external_client external 
>> 50:55:00:00:00:03 1342 172.20.0.99 24 172.20.0.100 1001::3/64 1001::a
>> +
>> +check multinode_nbctl --wait=hv sync
>> +m_wait_for_ports_up sw0-port1
>> +m_wait_for_ports_up external_client
>> +
>> +# The distributed gateway port is claimed by the replicated gateway chassis.
>> +m_wait_row_count Port_Binding 1 logical_port=cr-lr0-public
>> +vgw_uuid=$(m_central_as ovn-sbctl --bare --columns _uuid find Chassis 
>> name=$vgw_name)
>> +m_wait_column "$vgw_uuid" Port_Binding chassis logical_port=cr-lr0-public
>> +
>> +# The two controllers share a single Chassis record and a single binding for
>> +# the chassisredirect port.  Since both nodes are configured identically and
>> +# both resolve to the same chassis, each of them finds the records already
>> +# holding the values it was about to write and its transaction is a no-op, 
>> so
>> +# once the setup has settled no transaction touching those tables is 
>> expected.
>> +# If the nodes fought over a record, they would add one transaction per main
>> +# loop iteration to the southbound database log.
>> +sb_txn_count() {
>> +    m_central_as grep -ac "\"$1\":" /etc/ovn/ovnsb_db.db
>> +}
>> +
>> +chassis_txns=$(sb_txn_count Chassis)
>> +pb_txns=$(sb_txn_count Port_Binding)
>> +sleep 5
>> +check test $(($(sb_txn_count Chassis) - chassis_txns)) -le 1
>> +check test $(($(sb_txn_count Port_Binding) - pb_txns)) -le 1
>> +
>> +cr_logs=$(($(cr_lport_log_count $vgw1) + $(cr_lport_log_count $vgw2)))
>> +# Claim on both
>> +check test $((cr_logs - vgw1_cr_logs - vgw2_cr_logs)) -le 2
>> +
>> +check multinode_nbctl --wait=hv sync
>> +
>> +# North-south connectivity through the replicated gateway, with the 
>> underlay ECMP
>> +# route in place on both ends.
>> +check_ns_connectivity() {
>> +    M_NS_CHECK_EXEC([$compute], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 
>> 172.20.0.99 | FORMAT_PING], \
>> +[0], [dnl
>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
>> +])
>> +
>> +    M_NS_CHECK_EXEC([$compute], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 
>> 172.20.0.100 | FORMAT_PING], \
>> +[0], [dnl
>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
>> +])
>> +
>> +    M_NS_CHECK_EXEC([$client], [external], [ping -q -c 3 -i 0.3 -w 2 
>> 10.0.0.3 | FORMAT_PING], \
>> +[0], [dnl
>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
>> +])
>> +}
>> +
>> +# The traffic towards the replicated gateway must be tunneled to the 
>> anycast encap IP.
>> +M_START_TCPDUMPS([$compute], [-c 2 -neei eth1 host $vgw_encap_ip and udp 
>> port 6081], [ch1_underlay])
>> +
>> +check_ns_connectivity
>> +
>> +OVS_WAIT_UNTIL([test -s ch1_underlay.tcpdump])
>> +AT_CHECK([grep -q "$vgw_encap_ip" ch1_underlay.tcpdump])
>> +
>> +# Each of the two replicated gateway nodes on its own must be able to carry 
>> the
>> +# traffic: pin the underlay route to a single nexthop at a time.
>> +for nexthop in $vgw1_ip $vgw2_ip; do
>> +    for c in $compute $client; do
>> +        check m_as $c ip route replace $vgw_encap_ip/32 via $nexthop dev 
>> eth1
>> +    done
>> +
>> +    check_ns_connectivity
>> +done
>> +
>> +# Back to ECMP over both gateway nodes.
>> +for c in $compute $client; do
>> +    check m_as $c ip route replace $vgw_encap_ip/32 \
>> +        nexthop via $vgw1_ip dev eth1 \
>> +        nexthop via $vgw2_ip dev eth1
>> +done
>> +
>> +check_ns_connectivity
>> +
>> +AT_CLEANUP


_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Reply via email to