On 11/9/26 10:17 AM, Rukomoinikova Aleksandra wrote:
> On 07.11.2026 00:33, Dumitru Ceara wrote:
>> On 9/28/26 11:45 AM, Alexandra Rukomoinikova via dev wrote:
>>> Several ovn-controller instances may run with the same chassis name and
>>> the same encapsulation IP, an anycast address that is reachable over the
>>> underlay through ECMP. OVN then sees the whole group of nodes as a
>>> single chassis: a single Chassis record, a single Encap record and a
>>> single binding for every port placed on it, while the underlay decides
>>> which node of the group terminates a given tunnel flow.
>>>
>>> There is no coordination and no leader election between the nodes of the
>>> group while writing to SBDB: every ``ovn-controller`` independently does the
>>> full bookkeeping for that one chassis, and it is the idempotence of those
>>> writes that keeps them from fighting each other.
>>>
>>> The flip side is that the nodes must be configured identically in
>>> chassis-related params, and that nothing which relies on conntrack can be
>>> used, because the group shares no runtime state.
>>>
>>> Document all of this, and add a multinode test that deploys two
>>> gateway nodes as one replicated gateway, gives a hypervisor and a node that
>>> does not run OVN an underlay ECMP route to the anycast encapsulation IP,
>>> checks the north-south connectivity through each of the two nodes and
>>> checks that the controllers do not fight over the shared records.
>>>
>>> Signed-off-by: Alexandra Rukomoinikova <[email protected]>
>>> ---
>> Hi Alexandra,
>>
>> I have a few minor comments on this patch, please see below.
>>
>> I was thinking of taking care of them myself and squash a small fixup
>> patch in when pushing yours to main:
>>
>> https://github.com/dceara/ovn/commit/42b64d5
>>
>> I didn't test this locally but CI is running in my fork:
>>
>> https://github.com/dceara/ovn/actions/runs/42295260684
>>
>> Assuming that passes would you be OK with this incremental change?
>>
>> Regards,
>> Dumitru
> 

Hi Sasha,

> 
> Hi Dumitru! I took a fresh look at this patch, and I’d like to remove 
> all references to ECMP → anycast IP and drop the mention of using BGP 
> for announcements - this seems unnecessary. Is it still okay to make 

I actually liked the ECMP reference, maybe it's an idea to still mention
that and BGP as an example?  I'm not against removing it though.

> those changes rn? I can take the fixup based on fixups in your fork, 
> make the changes on top of it, and send you the updated patch. šŸ™‚
> 

I haven't merged your patch yet because CI didn't pass, one of the lanes
still hangs and I didn't have time to look into it:

https://github.com/dceara/ovn/actions/runs/42295260684/job/129055547103

So if you could take the fixup from my fork and work on a v2 that'd be
great.

Thanks,
Dumitru

> thanks, sasha
> 
> 
>>
>>>   Documentation/automake.mk                   |   1 +
>>>   Documentation/topics/index.rst              |   1 +
>>>   Documentation/topics/replicated-gateway.rst | 171 +++++++++++++++++
>>>   NEWS                                        |   5 +
>>>   controller/ovn-controller.8.xml             |   9 +-
>>>   tests/multinode-macros.at                   |  22 ++-
>>>   tests/multinode.at                          | 197 ++++++++++++++++++++
>>>   7 files changed, 399 insertions(+), 7 deletions(-)
>>>   create mode 100644 Documentation/topics/replicated-gateway.rst
>>>
>>> diff --git a/Documentation/automake.mk b/Documentation/automake.mk
>>> index 685ba9588..b64a0193d 100644
>>> --- a/Documentation/automake.mk
>>> +++ b/Documentation/automake.mk
>>> @@ -25,6 +25,7 @@ DOC_SOURCE = \
>>>     Documentation/topics/testing.rst \
>>>     Documentation/topics/test-development.rst \
>>>     Documentation/topics/high-availability.rst \
>>> +   Documentation/topics/replicated-gateway.rst \
>>>     Documentation/topics/dynamic-routing/architecture.rst \
>>>     Documentation/topics/dynamic-routing/configuration-examples.rst \
>>>     Documentation/topics/dynamic-routing/index.rst \
>>> diff --git a/Documentation/topics/index.rst b/Documentation/topics/index.rst
>>> index 0b676a98b..d5210e03a 100644
>>> --- a/Documentation/topics/index.rst
>>> +++ b/Documentation/topics/index.rst
>>> @@ -40,6 +40,7 @@ OVN
>>>      dynamic-routing/index
>>>      incremental-processing/index
>>>      high-availability
>>> +   replicated-gateway
>>>      role-based-access-control
>>>      vif-plug-providers/index
>>>      testing
>>> diff --git a/Documentation/topics/replicated-gateway.rst 
>>> b/Documentation/topics/replicated-gateway.rst
>>> new file mode 100644
>>> index 000000000..c3e43da03
>>> --- /dev/null
>>> +++ b/Documentation/topics/replicated-gateway.rst
>>> @@ -0,0 +1,171 @@
>>> +..
>>> +      Licensed under the Apache License, Version 2.0 (the "License"); you 
>>> may
>>> +      not use this file except in compliance with the License. You may 
>>> obtain
>>> +      a copy of the License at
>>> +
>>> +          http://www.apache.org/licenses/LICENSE-2.0
>>> +
>>> +      Unless required by applicable law or agreed to in writing, software
>>> +      distributed under the License is distributed on an "AS IS" BASIS, 
>>> WITHOUT
>>> +      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See 
>>> the
>>> +      License for the specific language governing permissions and 
>>> limitations
>>> +      under the License.
>>> +
>>> +      Convention for heading levels in OVN documentation:
>>> +
>>> +      =======  Heading 0 (reserved for the title in a document)
>>> +      -------  Heading 1
>>> +      ~~~~~~~  Heading 2
>>> +      +++++++  Heading 3
>>> +      '''''''  Heading 4
>>> +
>>> +      Avoid deeper levels because they do not render well.
>>> +
>>> +===================
>>> +Replicated Gateway
>>> +===================
>> Nit: one = too many.
>>
>>> +
>>> +A *replicated gateway* is a group of two or more physical nodes that 
>>> present
>>> +themselves to OVN as a single chassis. Every node of the group runs its own
>>> +``ovn-controller``, but all of them are configured with the same chassis 
>>> name
>>> +(``external_ids:system-id``) and with the same encapsulation IP
>>> +(``external_ids:ovn-encap-ip``). The encapsulation IP is an anycast 
>>> address,
>>> +it is announced into the underlay (statically or by a routing protocol) by 
>>> all
>>> +the nodes of the group, and the underlay load-balances the tunnel traffic
>>> +between them with ECMP.
>>> +
>>> +::
>>> +
>>> +                        underlay with ECMP to 10.0.0.1
>>> +
>>> +    +-------------------+                       +----------------------+
>>> +    |                   |  nexthop via gw-1     |  gateway node 1      |
>>> +    |   hypervisor      +---------------------->|  system-id: vgw      |
>>> +    |                   |                       |  ovn-encap-ip:       |
>>> +    |   ovn-encap-ip:   |                       |          10.0.0.1    |
>>> +    |     192.168.0.10  |                       +----------+-----------+
>>> +    |                   |                                  |
>>> +    |                   |  nexthop via gw-2     +----------+-----------+
>>> +    |                   +---------------------->|  gateway node 2      |
>>> +    |                   |                       |  system-id: vgw      |
>>> +    +-------------------+                       |  ovn-encap-ip:       |
>>> +                                                |          10.0.0.1    |
>>> +                                                +----------+-----------+
>>> +                                                           |
>>> +                                       physical network    |
>>> +    -------------------------------------------------------+------------
>>> +
>>> +From the point of view of the OVN databases there is a single chassis: a 
>>> single
>>> +``Chassis`` record, a single ``Encap`` record, and a single binding for 
>>> every
>>> +port that is placed on the replicated gateway (for example the 
>>> chassisredirect
>>> +port of a distributed gateway port). ``ovn-northd`` and the other
>>> +``ovn-controller`` instances do not know, and do not need to know, how many
>>> +physical nodes are behind that chassis. A hypervisor keeps a single tunnel 
>>> to
>>> +the anycast encapsulation IP, and the underlay decides which node of the 
>>> group
>>> +terminates a given flow.
>>> +
>>> +There is no coordination and no leader election between the nodes of the
>>> +group while writing to SBDB: every ``ovn-controller`` independently does 
>>> the
>>> +full bookkeeping for that one chassis, and it is the idempotence of those
>>> +writes that keeps them from fighting each other.
>>> +
>>> +Two consequences follow from this.  First, the columns that OVN derives 
>>> from
>>> +the local node rather than from the configuration must be forced to the 
>>> same
>>> +value on every node.  More detailed information on this is provided in the
>>> +``Identical OVS and OVN configuration and features`` section below.  
>>> Second,
>>> +the nodes do not share any runtime state, so nothing that relies on 
>>> conntrack
>>> +can be used on a replicated gateway, see the ``Only stateless traffic is
>>> +supported`` section below.
>> We shouldn't break in the middle of the verbatim section, that renders
>> weird.
>>
>>> +
>>> +Use case
>>> +--------
>>> +This is useful when the capacity or the availability of a single gateway 
>>> node
>>> +is not enough: the group scales horizontally, and losing one node only 
>>> means
>>> +that the underlay stops using one of the ECMP nexthops, without any change 
>>> in
>>> +the OVN databases and without the gateway having to be rescheduled to 
>>> another
>>> +chassis.
>>> +
>>> +A typical deployment looks as follows:
>>> +
>>> +* Two (or more) gateway nodes are installed with the same OVN and OVS
>>> +  versions and with an identical ``ovn-controller`` configuration. Each of
>>> +  them configures the anycast encapsulation IP locally, e.g. on the 
>>> loopback
>>> +  interface, so that the kernel accepts the tunnel packets destined to it::
>>> +
>>> +      $ ovs-vsctl set open . external-ids:system-id=vgw
>>> +      $ ovs-vsctl set open . external-ids:ovn-encap-ip=10.0.0.1
>>> +      $ ovs-vsctl set open . external-ids:ovn-encap-type=geneve
>>> +      $ ip address add 10.0.0.1/32 dev lo
>>> +
>>> +* The anycast address is reachable in the underlay through the physical
>>> +  addresses of all the nodes of the group. On the hypervisors, and on any
>>> +  other node that has to reach the replicated gateway, this is an ECMP 
>>> route::
>>> +
>>> +      $ ip route replace 10.0.0.1/32 \
>>> +            nexthop via 192.168.0.1 dev eth0 \
>>> +            nexthop via 192.168.0.2 dev eth0
>>> +
>>> +  In a real deployment the route is normally learned from the fabric, with 
>>> the
>>> +  gateway nodes advertising the anycast prefix over BGP.
>>> +
>>> +* The logical topology is configured as usual, referring to the chassis by 
>>> its
>>> +  single name, e.g.::
>>> +
>>> +      $ ovn-nbctl lrp-set-gateway-chassis lr0-public vgw 10
>>> +
>>> +Requirements and restrictions
>>> +-----------------------------
>>> +
>>> +The nodes of a replicated gateway are not independent chassis, they are one
>>> +chassis that happens to run on several machines. Everything that OVN stores
>>> +per chassis is therefore shared, and everything that OVN computes per 
>>> chassis
>>> +must produce the same result on every node of the group. This leads to the
>>> +following requirements.
>>> +
>>> +Identical OVS and OVN configuration and features
>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> +
>>> +All the nodes of the group must run the same OVS and OVN versions and must
>>> +expose the same set of features. ``ovn-controller`` reports the 
>>> capabilities
>>> +of the local OVS instance in the ``Chassis`` record (``other_config``, for
>>> +example the supported datapath interface types, and the various
>>> +``ovn-chassis-feature`` entries), and ``ovn-northd`` generates the logical
>>> +flows for the chassis based on those values. If the nodes disagree, the
>>> +``Chassis`` record will keep flapping between the values reported by each 
>>> node,
>>> +``ovn-northd`` will keep recomputing, and the node whose capabilities do 
>>> not
>>> +match the ones currently published may not be able to process the flows it
>>> +receives.
>>> +
>>> +For the same reason, every chassis scoped configuration option has to be
>>> +identical on all the nodes, including, for example: 
>>> ``external_ids:hostname``,
>>> +``external_ids:ovn-bridge-mappings`` and all others.
>>> +
>>> +Only stateless traffic is supported
>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> +
>>> +The nodes of a replicated gateway do not share any runtime state, and in
>>> +particular they do not share conntrack. The underlay picks one of them per
>>> +flow, and nothing guarantees that the two directions of a connection, or 
>>> the
>>> +packets of a connection before and after a change of the ECMP nexthops (a 
>>> node
>>> +being added, removed, rebooted or simply an ECMP rehash), are handled by 
>>> the
>>> +same node. Any feature that needs the connection state to be on the node 
>>> that
>>> +processes the packet is therefore not supported on a replicated gateway:
>>> +
>>> +* load balancers,
>>> +* stateful NAT, that is ``snat`` and ``dnat_and_snat`` (unless configured 
>>> as
>>> +  stateless), and NAT with ``ct-commit-all`` enabled,
>>> +* stateful ACLs, and anything else that relies on conntrack, for example
>>> +  load balancer hairpinning or ECMP routes with
>>> +  ``options:ecmp_symmetric_reply`` set.
>>> +
>>> +Only stateless forwarding, e.g. plain routing and switching, stateless NAT 
>>> and
>>> +stateless ACLs, behaves correctly, because every packet can be handled
>>> +independently by any node of the group.
>>> +
>>> +Example
>>> +-------
>>> +
>>> +The ``ovn multinode replicated gateway`` test in ``tests/multinode.at`` 
>>> sets up
>>> +a complete example of a two node replicated gateway with an underlay ECMP 
>>> route
>>> +to the anycast encapsulation IP, a hypervisor with a VM behind the gateway
>>> +and an external node that does not run OVN at all.
>>> diff --git a/NEWS b/NEWS
>>> index f1c56dd14..7bc3f2113 100644
>>> --- a/NEWS
>>> +++ b/NEWS
>>> @@ -129,6 +129,11 @@ OVN v26.09.0 - xxx xx xxxx
>>>        represent a logical router port (e.g. ovn-ic transit switch LSPs).
>>>        Such ports are treated like type=router, including omission from
>>>        _MC_flood_l2.
>>> +   - Documented the "replicated gateway" setup, in which several 
>>> ovn-controller
>>> +     instances share one chassis name and one anycast encapsulation IP
>>> +     reachable over the underlay through ECMP, so that they appear to OVN 
>>> as
>>> +     a single chassis. See Documentation/topics/replicated-gateway.rst for 
>>> the
>>> +     requirements and the restrictions.
>>>   
>>>   OVN v26.03.0 - xxx xx xxxx
>>>   --------------------------
>>> diff --git a/controller/ovn-controller.8.xml 
>>> b/controller/ovn-controller.8.xml
>>> index 3c33654ff..37c64ebf3 100644
>>> --- a/controller/ovn-controller.8.xml
>>> +++ b/controller/ovn-controller.8.xml
>>> @@ -100,7 +100,14 @@
>>>         local OVN "etc" directory or via the <code>-n</code> command-line 
>>> option.
>>>         The following precedence is used: first, the command-line option is 
>>> read;
>>>         if not present, the <code>system-id-override</code> file is read; 
>>> if not
>>> -      present, then the name configured in the database is used.</dd>
>>> +      present, then the name configured in the database is used.
>>> +      <p>
>>> +        Several <code>ovn-controller</code> instances running on different
>>> +        nodes may share the same <code>system-id</code>, so that they 
>>> appear
>>> +        to OVN as a single chassis backed by an anycast encapsulation IP.
>>> +        See the "Replicated Gateway" document for the requirements and the
>>> +        restrictions of such a setup.
>>> +      </p></dd>
>>>   
>>>         <dt><code>external_ids:hostname</code></dt>
>>>         <dd>The hostname to use in the Chassis table.</dd>
>>> diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at
>>> index 1c85f79f8..81ab2a5b8 100644
>>> --- a/tests/multinode-macros.at
>>> +++ b/tests/multinode-macros.at
>>> @@ -296,23 +296,33 @@ multinode_setup_northd() {
>>>       m_as $c ovn-sbctl set-connection p${REMOTE_PROT}:6642
>>>   }
>>>   
>>> -# multinode_setup_controller NODE ENCAP_IP REMOTE_IP [ENCAP_TYPE]
>>> -#
>>> -# Sets up controller on specified node.
>>> +# multinode_setup_controller NODE SYSTEM_ID ENCAP_IP REMOTE_IP
>>> +#                            [ENCAP_TYPE] [START_CONTROLLER]
>>> +#
>>> +# Sets up controller on specified node. SYSTEM_ID defaults to NODE; 
>>> passing a
>>> +# different value (or the same value on more than one node) allows a test 
>>> to
>>> +# control the chassis name the node registers with.  START_CONTROLLER 
>>> defaults
>>> +# to "yes"; with "no" only OVS is set up, so that a test can finish the 
>>> node
>>> +# configuration before starting ovn-controller itself.
>>>   multinode_setup_controller() {
>>>       c=$1
>>> +    system_id=${2:-$1}
>>>       encap_ip=$3
>>>       remote_ip=$4
>>>       encap_type=${5:-"geneve"}
>>> +    start_controller=${6:-"yes"}
>>>   
>>>       # Cleanup existing one
>>>       m_as $c /usr/share/openvswitch/scripts/ovs-ctl stop
>>>       m_as $c /usr/share/ovn/scripts/ovn-ctl stop_controller
>>>       m_as $c sh -c "rm -f /etc/openvswitch/*.db"
>>>   
>>> -    m_as $c /usr/share/openvswitch/scripts/ovs-ctl start --system-id=$c
>>> -    echo "Using ${CONTROLLER_SSL_ARGS} for ovn-controller".
>>> -    m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller 
>>> ${CONTROLLER_SSL_ARGS}
>>> +    m_as $c /usr/share/openvswitch/scripts/ovs-ctl start 
>>> --system-id=$system_id
>>> +    if [[ "$start_controller" = "yes" ]]; then
>>> +        echo "Using ${CONTROLLER_SSL_ARGS} for ovn-controller".
>>> +        m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller \
>>> +            ${CONTROLLER_SSL_ARGS}
>>> +    fi
>> I think if we move the part that starts ovn-controller to the end of
>> this function we don't need the kind of ugly start_controller argument.
>> We can just set everything we need and then start ovn-controller.
>>
>>>   
>>>       m_as $c ovs-vsctl set open . external_ids:ovn-encap-ip=$encap_ip
>>>       m_as $c ovs-vsctl set open . external-ids:ovn-encap-type=$encap_type
>>> diff --git a/tests/multinode.at b/tests/multinode.at
>>> index 0c277f5e8..16cfa804f 100644
>>> --- a/tests/multinode.at
>>> +++ b/tests/multinode.at
>>> @@ -5554,3 +5554,200 @@ AT_CHECK([cat ch2_eth2.tcpdump], [0], [dnl
>>>   ])
>>>   
>>>   AT_CLEANUP
>>> +
>>> +AT_SETUP([ovn multinode replicated gateway])
>>> +
>>> +check_fake_multinode_setup
>>> +cleanup_multinode_resources
>>> +
>>> +# Two gateway nodes are deployed as one and the same "replicated" gateway: 
>>> both
>>> +# ovn-controllers register with the same chassis name and with the same 
>>> anycast
>>> +# encap IP. The encap IP is reachable over the underlay via ECMP through 
>>> the
>>> +# physical addresses of both gateway nodes, so a geneve tunnel to the 
>>> replicated
>>> +# gateway may land on either of them.
>>> +#
>>> +#                             underlay (eth1, 170.168.0.0/16)
>>> +#  ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”                                   
>>> ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”
>>> +#  │  ovn-chassis-1    │   1.1.1.1/32                      │    ovn-gw-1   
>>>    │
>>> +#  │  (compute node)   ā”œā”€ā”€ā”€ā”€ nexthop via gw-1 eth1 ────────┤  
>>> ovn-replicated-gw  │
>>> +#  │                   │                                   │  encap 
>>> 1.1.1.1   │
>>> +#  │  sw0p1 10.0.0.3   ā”œā”€ā”€ā”€ā”€ nexthop via gw-2 eth1 ───┐    
>>> ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜
>>> +#  ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜                              │             │
>>> +#                                                     │    
>>> ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”“ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”
>>> +#  ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”                              └────┤    ovn-gw-2   
>>>    │
>>> +#  │  ovn-chassis-2    │   1.1.1.1/32                      │  
>>> ovn-replicated-gw  │
>>> +#  │ (external client) ā”œā”€ā”€ā”€ā”€ nexthop via gw-1 eth1 ────────┤  encap 
>>> 1.1.1.1   │
>>> +#  │                   ā”œā”€ā”€ā”€ā”€ nexthop via gw-2 eth1 ────────┤               
>>>    │
>>> +#  │                   │                                   
>>> ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜
>>> +#  │  eth2 172.20.0.50 ā”œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ provider network (eth2, br-ex) ā”€ā”€ā”€ā”€ā”˜
>>> +#  ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜                       lr0-public 172.20.0.100
>>> +#
>>> +# On ovn-chassis-2 (aka client) we have client port in public switch for 
>>> checking
>>> +# internal connectivity, public switch is connected to dgp port that 
>>> claimed by
>>> +# replicated chassis. Compute and client node both have underlay route for 
>>> replicated gw ip (1.1.1.1)
>>> +
>>> +vgw1=ovn-gw-1
>>> +vgw2=ovn-gw-2
>>> +compute=ovn-chassis-1
>>> +client=ovn-chassis-2
>>> +vgw_name=ovn-replicated-gw
>>> +vgw_encap_ip=1.1.1.1
>>> +
>>> +m_underlay_ip() {
>>> +    m_as $1 ip -4 addr show eth1 | grep inet | awk '{print $2}' | cut 
>>> -d'/' -f1
>>> +}
>>> +
>>> +central_ip=$(m_as ovn-central-az1 ip -4 addr show eth1 | grep inet | awk 
>>> '{print $2}' | cut -d'/' -f1)
>> This should now be:
>>
>> central_ip=$( (m_central_as ip -4 addr show eth1 2>/dev/null ||
>> m_central_as ip -4 addr show eth2) | grep inet | awk '{print $2}' | cut
>> -d'/' -f1)
>>
>> Since we merged 44d3bf28e4b7 ("multinode: Add support for INLINE_BR_EX
>> configuration of multinode.")
>>
>>> +vgw1_ip=$(m_underlay_ip $vgw1)
>>> +vgw2_ip=$(m_underlay_ip $vgw2)
>>> +compute_ip=$(m_underlay_ip $compute)
>>> +
>>> +m_as $compute ip link del sw0p1-p
>>> +m_as $client ip link del sw0p2-p
>>> +m_as $client ip link del sw1p1-p
>>> +
>>> +# Bring up both gateway nodes as the very same chassis: same chassis name 
>>> and
>>> +# same encap IP, which is not part of the underlay address space.
>>> +for c in $vgw1 $vgw2; do
>>> +    # Bring the node back to its original chassis name and encap IP.
>>> +    on_exit "multinode_setup_controller $c $c $(m_underlay_ip $c) 
>>> $central_ip"
>>> +    multinode_setup_controller $c $vgw_name $vgw_encap_ip $central_ip 
>>> geneve no
>>> +    on_exit "m_as $c ip addr del $vgw_encap_ip/32 dev lo"
>>> +    check m_as $c ip addr add $vgw_encap_ip/32 dev lo
>>> +    check m_as $c ovs-vsctl set open . external-ids:hostname=$vgw_name
>>> +    check m_as $c ovs-vsctl set open . 
>>> external-ids:ovn-bridge-mappings=public:br-ex
>> I don't think your test uses the ovn-bridge mapping in any way.  So then
>> we might as well add hostname as an optional argument to
>> multinode_setup_controller().
>>
>>> +done
>>> +
>>> +for c in $vgw1 $vgw2; do
>>> +    m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller 
>>> ${CONTROLLER_SSL_ARGS}
>>> +done
>> If we change multinode_setup_controller() as I mentioned above, we don't
>> need to manually start it here.
>>
>>> +
>>> +# Both gateway nodes must end up sharing a single chassis record with a 
>>> single
>>> +# encap record.
>>> +m_wait_row_count Chassis 1 name=$vgw_name
>>> +m_wait_row_count Encap 1 chassis_name=$vgw_name
>>> +m_check_column "$vgw_encap_ip" Encap ip chassis_name=$vgw_name
>>> +
>>> +# ECMP route to the replicated gateway encap IP in the underlay, both on 
>>> the
>>> +# compute node and on the external client.
>>> +for c in $compute $client; do
>>> +    on_exit "m_as $c ip route del $vgw_encap_ip/32"
>>> +    check m_as $c ip route replace $vgw_encap_ip/32 \
>>> +        nexthop via $vgw1_ip dev eth1 \
>>> +        nexthop via $vgw2_ip dev eth1
>>> +done
>>> +
>>> +
>> Nit: one empty line too many.
>>
>>> +# The anycast encap IP is reachable over the underlay from both nodes.
>>> +AT_CHECK([m_as $compute ping -q -c 3 -i 0.3 -w 2 $vgw_encap_ip | 
>>> FORMAT_PING], [0], [dnl
>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
>>> +])
>>> +AT_CHECK([m_as $client ping -q -c 3 -i 0.3 -w 2 $vgw_encap_ip | 
>>> FORMAT_PING], [0], [dnl
>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
>>> +])
>>> +
>>> +cr_lport_log_count() {
>>> +    m_as $1 sh -c "cat /var/log/ovn/ovn-controller.log 2>/dev/null | \
>>> +                   grep -ac 'Claiming lport cr-lr0-public'"
>>> +}
>>> +
>>> +vgw1_cr_logs=$(cr_lport_log_count $vgw1)
>>> +vgw2_cr_logs=$(cr_lport_log_count $vgw2)
>>> +
>>> +check multinode_nbctl ls-add sw0
>>> +check multinode_nbctl lsp-add sw0 sw0-port1
>>> +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 
>>> 10.0.0.3"
>>> +
>>> +check multinode_nbctl lr-add lr0
>>> +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24
>>> +check multinode_nbctl lsp-add-router-port sw0 sw0-lr0 lr0-sw0
>>> +
>>> +check multinode_nbctl ls-add public
>>> +check multinode_nbctl lsp-add public external_client
>>> +check multinode_nbctl lsp-set-addresses external_client "50:55:00:00:00:03 
>>> 172.20.0.99"
>>> +check multinode_nbctl lrp-add lr0 lr0-public 00:11:22:00:ff:01 
>>> 172.20.0.100/24
>>> +check multinode_nbctl lsp-add-router-port public public-lr0 lr0-public
>>> +check multinode_nbctl lrp-set-gateway-chassis lr0-public $vgw_name 10
>>> +
>>> +m_as $compute /data/create_fake_vm.sh sw0-port1 sw0p1 50:54:00:00:00:03 
>>> 1342 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a
>>> +m_as $client /data/create_fake_vm.sh external_client external 
>>> 50:55:00:00:00:03 1342 172.20.0.99 24 172.20.0.100 1001::3/64 1001::a
>>> +
>>> +check multinode_nbctl --wait=hv sync
>>> +m_wait_for_ports_up sw0-port1
>>> +m_wait_for_ports_up external_client
>>> +
>>> +# The distributed gateway port is claimed by the replicated gateway 
>>> chassis.
>>> +m_wait_row_count Port_Binding 1 logical_port=cr-lr0-public
>>> +vgw_uuid=$(m_central_as ovn-sbctl --bare --columns _uuid find Chassis 
>>> name=$vgw_name)
>>> +m_wait_column "$vgw_uuid" Port_Binding chassis logical_port=cr-lr0-public
>>> +
>>> +# The two controllers share a single Chassis record and a single binding 
>>> for
>>> +# the chassisredirect port.  Since both nodes are configured identically 
>>> and
>>> +# both resolve to the same chassis, each of them finds the records already
>>> +# holding the values it was about to write and its transaction is a no-op, 
>>> so
>>> +# once the setup has settled no transaction touching those tables is 
>>> expected.
>>> +# If the nodes fought over a record, they would add one transaction per 
>>> main
>>> +# loop iteration to the southbound database log.
>>> +sb_txn_count() {
>>> +    m_central_as grep -ac "\"$1\":" /etc/ovn/ovnsb_db.db
>>> +}
>>> +
>>> +chassis_txns=$(sb_txn_count Chassis)
>>> +pb_txns=$(sb_txn_count Port_Binding)
>>> +sleep 5
>>> +check test $(($(sb_txn_count Chassis) - chassis_txns)) -le 1
>>> +check test $(($(sb_txn_count Port_Binding) - pb_txns)) -le 1
>>> +
>>> +cr_logs=$(($(cr_lport_log_count $vgw1) + $(cr_lport_log_count $vgw2)))
>>> +# Claim on both
>>> +check test $((cr_logs - vgw1_cr_logs - vgw2_cr_logs)) -le 2
>>> +
>>> +check multinode_nbctl --wait=hv sync
>>> +
>>> +# North-south connectivity through the replicated gateway, with the 
>>> underlay ECMP
>>> +# route in place on both ends.
>>> +check_ns_connectivity() {
>>> +    M_NS_CHECK_EXEC([$compute], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 
>>> 172.20.0.99 | FORMAT_PING], \
>>> +[0], [dnl
>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
>>> +])
>>> +
>>> +    M_NS_CHECK_EXEC([$compute], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 
>>> 172.20.0.100 | FORMAT_PING], \
>>> +[0], [dnl
>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
>>> +])
>>> +
>>> +    M_NS_CHECK_EXEC([$client], [external], [ping -q -c 3 -i 0.3 -w 2 
>>> 10.0.0.3 | FORMAT_PING], \
>>> +[0], [dnl
>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
>>> +])
>>> +}
>>> +
>>> +# The traffic towards the replicated gateway must be tunneled to the 
>>> anycast encap IP.
>>> +M_START_TCPDUMPS([$compute], [-c 2 -neei eth1 host $vgw_encap_ip and udp 
>>> port 6081], [ch1_underlay])
>>> +
>>> +check_ns_connectivity
>>> +
>>> +OVS_WAIT_UNTIL([test -s ch1_underlay.tcpdump])
>>> +AT_CHECK([grep -q "$vgw_encap_ip" ch1_underlay.tcpdump])
>>> +
>>> +# Each of the two replicated gateway nodes on its own must be able to 
>>> carry the
>>> +# traffic: pin the underlay route to a single nexthop at a time.
>>> +for nexthop in $vgw1_ip $vgw2_ip; do
>>> +    for c in $compute $client; do
>>> +        check m_as $c ip route replace $vgw_encap_ip/32 via $nexthop dev 
>>> eth1
>>> +    done
>>> +
>>> +    check_ns_connectivity
>>> +done
>>> +
>>> +# Back to ECMP over both gateway nodes.
>>> +for c in $compute $client; do
>>> +    check m_as $c ip route replace $vgw_encap_ip/32 \
>>> +        nexthop via $vgw1_ip dev eth1 \
>>> +        nexthop via $vgw2_ip dev eth1
>>> +done
>>> +
>>> +check_ns_connectivity
>>> +
>>> +AT_CLEANUP
> 

_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Reply via email to