On 11/9/26 12:01 PM, Dumitru Ceara wrote:
> On 11/9/26 10:17 AM, Rukomoinikova Aleksandra wrote:
>> On 07.11.2026 00:33, Dumitru Ceara wrote:
>>> On 9/28/26 11:45 AM, Alexandra Rukomoinikova via dev wrote:
>>>> Several ovn-controller instances may run with the same chassis name and
>>>> the same encapsulation IP, an anycast address that is reachable over the
>>>> underlay through ECMP. OVN then sees the whole group of nodes as a
>>>> single chassis: a single Chassis record, a single Encap record and a
>>>> single binding for every port placed on it, while the underlay decides
>>>> which node of the group terminates a given tunnel flow.
>>>>
>>>> There is no coordination and no leader election between the nodes of the
>>>> group while writing to SBDB: every ``ovn-controller`` independently does 
>>>> the
>>>> full bookkeeping for that one chassis, and it is the idempotence of those
>>>> writes that keeps them from fighting each other.
>>>>
>>>> The flip side is that the nodes must be configured identically in
>>>> chassis-related params, and that nothing which relies on conntrack can be
>>>> used, because the group shares no runtime state.
>>>>
>>>> Document all of this, and add a multinode test that deploys two
>>>> gateway nodes as one replicated gateway, gives a hypervisor and a node that
>>>> does not run OVN an underlay ECMP route to the anycast encapsulation IP,
>>>> checks the north-south connectivity through each of the two nodes and
>>>> checks that the controllers do not fight over the shared records.
>>>>
>>>> Signed-off-by: Alexandra Rukomoinikova <[email protected]>
>>>> ---
>>> Hi Alexandra,
>>>
>>> I have a few minor comments on this patch, please see below.
>>>
>>> I was thinking of taking care of them myself and squash a small fixup
>>> patch in when pushing yours to main:
>>>
>>> https://github.com/dceara/ovn/commit/42b64d5
>>>
>>> I didn't test this locally but CI is running in my fork:
>>>
>>> https://github.com/dceara/ovn/actions/runs/42295260684
>>>
>>> Assuming that passes would you be OK with this incremental change?
>>>
>>> Regards,
>>> Dumitru
>>
> 
> Hi Sasha,
> 
>>
>> Hi Dumitru! I took a fresh look at this patch, and I’d like to remove 
>> all references to ECMP → anycast IP and drop the mention of using BGP 
>> for announcements - this seems unnecessary. Is it still okay to make 
> 
> I actually liked the ECMP reference, maybe it's an idea to still mention
> that and BGP as an example?  I'm not against removing it though.
> 

While at it, maybe it's worth also mentioning that other (stateful)
features might not work as expected in such deployments.  E.g., IGMP.

>> those changes rn? I can take the fixup based on fixups in your fork, 
>> make the changes on top of it, and send you the updated patch. šŸ™‚
>>
> 
> I haven't merged your patch yet because CI didn't pass, one of the lanes
> still hangs and I didn't have time to look into it:
> 
> https://github.com/dceara/ovn/actions/runs/42295260684/job/129055547103
> 
> So if you could take the fixup from my fork and work on a v2 that'd be
> great.
> 
> Thanks,
> Dumitru
> 
>> thanks, sasha
>>
>>
>>>
>>>>   Documentation/automake.mk                   |   1 +
>>>>   Documentation/topics/index.rst              |   1 +
>>>>   Documentation/topics/replicated-gateway.rst | 171 +++++++++++++++++
>>>>   NEWS                                        |   5 +
>>>>   controller/ovn-controller.8.xml             |   9 +-
>>>>   tests/multinode-macros.at                   |  22 ++-
>>>>   tests/multinode.at                          | 197 ++++++++++++++++++++
>>>>   7 files changed, 399 insertions(+), 7 deletions(-)
>>>>   create mode 100644 Documentation/topics/replicated-gateway.rst
>>>>
>>>> diff --git a/Documentation/automake.mk b/Documentation/automake.mk
>>>> index 685ba9588..b64a0193d 100644
>>>> --- a/Documentation/automake.mk
>>>> +++ b/Documentation/automake.mk
>>>> @@ -25,6 +25,7 @@ DOC_SOURCE = \
>>>>    Documentation/topics/testing.rst \
>>>>    Documentation/topics/test-development.rst \
>>>>    Documentation/topics/high-availability.rst \
>>>> +  Documentation/topics/replicated-gateway.rst \
>>>>    Documentation/topics/dynamic-routing/architecture.rst \
>>>>    Documentation/topics/dynamic-routing/configuration-examples.rst \
>>>>    Documentation/topics/dynamic-routing/index.rst \
>>>> diff --git a/Documentation/topics/index.rst 
>>>> b/Documentation/topics/index.rst
>>>> index 0b676a98b..d5210e03a 100644
>>>> --- a/Documentation/topics/index.rst
>>>> +++ b/Documentation/topics/index.rst
>>>> @@ -40,6 +40,7 @@ OVN
>>>>      dynamic-routing/index
>>>>      incremental-processing/index
>>>>      high-availability
>>>> +   replicated-gateway
>>>>      role-based-access-control
>>>>      vif-plug-providers/index
>>>>      testing
>>>> diff --git a/Documentation/topics/replicated-gateway.rst 
>>>> b/Documentation/topics/replicated-gateway.rst
>>>> new file mode 100644
>>>> index 000000000..c3e43da03
>>>> --- /dev/null
>>>> +++ b/Documentation/topics/replicated-gateway.rst
>>>> @@ -0,0 +1,171 @@
>>>> +..
>>>> +      Licensed under the Apache License, Version 2.0 (the "License"); you 
>>>> may
>>>> +      not use this file except in compliance with the License. You may 
>>>> obtain
>>>> +      a copy of the License at
>>>> +
>>>> +          http://www.apache.org/licenses/LICENSE-2.0
>>>> +
>>>> +      Unless required by applicable law or agreed to in writing, software
>>>> +      distributed under the License is distributed on an "AS IS" BASIS, 
>>>> WITHOUT
>>>> +      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 
>>>> See the
>>>> +      License for the specific language governing permissions and 
>>>> limitations
>>>> +      under the License.
>>>> +
>>>> +      Convention for heading levels in OVN documentation:
>>>> +
>>>> +      =======  Heading 0 (reserved for the title in a document)
>>>> +      -------  Heading 1
>>>> +      ~~~~~~~  Heading 2
>>>> +      +++++++  Heading 3
>>>> +      '''''''  Heading 4
>>>> +
>>>> +      Avoid deeper levels because they do not render well.
>>>> +
>>>> +===================
>>>> +Replicated Gateway
>>>> +===================
>>> Nit: one = too many.
>>>
>>>> +
>>>> +A *replicated gateway* is a group of two or more physical nodes that 
>>>> present
>>>> +themselves to OVN as a single chassis. Every node of the group runs its 
>>>> own
>>>> +``ovn-controller``, but all of them are configured with the same chassis 
>>>> name
>>>> +(``external_ids:system-id``) and with the same encapsulation IP
>>>> +(``external_ids:ovn-encap-ip``). The encapsulation IP is an anycast 
>>>> address,
>>>> +it is announced into the underlay (statically or by a routing protocol) 
>>>> by all
>>>> +the nodes of the group, and the underlay load-balances the tunnel traffic
>>>> +between them with ECMP.
>>>> +
>>>> +::
>>>> +
>>>> +                        underlay with ECMP to 10.0.0.1
>>>> +
>>>> +    +-------------------+                       +----------------------+
>>>> +    |                   |  nexthop via gw-1     |  gateway node 1      |
>>>> +    |   hypervisor      +---------------------->|  system-id: vgw      |
>>>> +    |                   |                       |  ovn-encap-ip:       |
>>>> +    |   ovn-encap-ip:   |                       |          10.0.0.1    |
>>>> +    |     192.168.0.10  |                       +----------+-----------+
>>>> +    |                   |                                  |
>>>> +    |                   |  nexthop via gw-2     +----------+-----------+
>>>> +    |                   +---------------------->|  gateway node 2      |
>>>> +    |                   |                       |  system-id: vgw      |
>>>> +    +-------------------+                       |  ovn-encap-ip:       |
>>>> +                                                |          10.0.0.1    |
>>>> +                                                +----------+-----------+
>>>> +                                                           |
>>>> +                                       physical network    |
>>>> +    -------------------------------------------------------+------------
>>>> +
>>>> +From the point of view of the OVN databases there is a single chassis: a 
>>>> single
>>>> +``Chassis`` record, a single ``Encap`` record, and a single binding for 
>>>> every
>>>> +port that is placed on the replicated gateway (for example the 
>>>> chassisredirect
>>>> +port of a distributed gateway port). ``ovn-northd`` and the other
>>>> +``ovn-controller`` instances do not know, and do not need to know, how 
>>>> many
>>>> +physical nodes are behind that chassis. A hypervisor keeps a single 
>>>> tunnel to
>>>> +the anycast encapsulation IP, and the underlay decides which node of the 
>>>> group
>>>> +terminates a given flow.
>>>> +
>>>> +There is no coordination and no leader election between the nodes of the
>>>> +group while writing to SBDB: every ``ovn-controller`` independently does 
>>>> the
>>>> +full bookkeeping for that one chassis, and it is the idempotence of those
>>>> +writes that keeps them from fighting each other.
>>>> +
>>>> +Two consequences follow from this.  First, the columns that OVN derives 
>>>> from
>>>> +the local node rather than from the configuration must be forced to the 
>>>> same
>>>> +value on every node.  More detailed information on this is provided in the
>>>> +``Identical OVS and OVN configuration and features`` section below.  
>>>> Second,
>>>> +the nodes do not share any runtime state, so nothing that relies on 
>>>> conntrack
>>>> +can be used on a replicated gateway, see the ``Only stateless traffic is
>>>> +supported`` section below.
>>> We shouldn't break in the middle of the verbatim section, that renders
>>> weird.
>>>
>>>> +
>>>> +Use case
>>>> +--------
>>>> +This is useful when the capacity or the availability of a single gateway 
>>>> node
>>>> +is not enough: the group scales horizontally, and losing one node only 
>>>> means
>>>> +that the underlay stops using one of the ECMP nexthops, without any 
>>>> change in
>>>> +the OVN databases and without the gateway having to be rescheduled to 
>>>> another
>>>> +chassis.
>>>> +
>>>> +A typical deployment looks as follows:
>>>> +
>>>> +* Two (or more) gateway nodes are installed with the same OVN and OVS
>>>> +  versions and with an identical ``ovn-controller`` configuration. Each of
>>>> +  them configures the anycast encapsulation IP locally, e.g. on the 
>>>> loopback
>>>> +  interface, so that the kernel accepts the tunnel packets destined to 
>>>> it::
>>>> +
>>>> +      $ ovs-vsctl set open . external-ids:system-id=vgw
>>>> +      $ ovs-vsctl set open . external-ids:ovn-encap-ip=10.0.0.1
>>>> +      $ ovs-vsctl set open . external-ids:ovn-encap-type=geneve
>>>> +      $ ip address add 10.0.0.1/32 dev lo
>>>> +
>>>> +* The anycast address is reachable in the underlay through the physical
>>>> +  addresses of all the nodes of the group. On the hypervisors, and on any
>>>> +  other node that has to reach the replicated gateway, this is an ECMP 
>>>> route::
>>>> +
>>>> +      $ ip route replace 10.0.0.1/32 \
>>>> +            nexthop via 192.168.0.1 dev eth0 \
>>>> +            nexthop via 192.168.0.2 dev eth0
>>>> +
>>>> +  In a real deployment the route is normally learned from the fabric, 
>>>> with the
>>>> +  gateway nodes advertising the anycast prefix over BGP.
>>>> +
>>>> +* The logical topology is configured as usual, referring to the chassis 
>>>> by its
>>>> +  single name, e.g.::
>>>> +
>>>> +      $ ovn-nbctl lrp-set-gateway-chassis lr0-public vgw 10
>>>> +
>>>> +Requirements and restrictions
>>>> +-----------------------------
>>>> +
>>>> +The nodes of a replicated gateway are not independent chassis, they are 
>>>> one
>>>> +chassis that happens to run on several machines. Everything that OVN 
>>>> stores
>>>> +per chassis is therefore shared, and everything that OVN computes per 
>>>> chassis
>>>> +must produce the same result on every node of the group. This leads to the
>>>> +following requirements.
>>>> +
>>>> +Identical OVS and OVN configuration and features
>>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>> +
>>>> +All the nodes of the group must run the same OVS and OVN versions and must
>>>> +expose the same set of features. ``ovn-controller`` reports the 
>>>> capabilities
>>>> +of the local OVS instance in the ``Chassis`` record (``other_config``, for
>>>> +example the supported datapath interface types, and the various
>>>> +``ovn-chassis-feature`` entries), and ``ovn-northd`` generates the logical
>>>> +flows for the chassis based on those values. If the nodes disagree, the
>>>> +``Chassis`` record will keep flapping between the values reported by each 
>>>> node,
>>>> +``ovn-northd`` will keep recomputing, and the node whose capabilities do 
>>>> not
>>>> +match the ones currently published may not be able to process the flows it
>>>> +receives.
>>>> +
>>>> +For the same reason, every chassis scoped configuration option has to be
>>>> +identical on all the nodes, including, for example: 
>>>> ``external_ids:hostname``,
>>>> +``external_ids:ovn-bridge-mappings`` and all others.
>>>> +
>>>> +Only stateless traffic is supported
>>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>> +
>>>> +The nodes of a replicated gateway do not share any runtime state, and in
>>>> +particular they do not share conntrack. The underlay picks one of them per
>>>> +flow, and nothing guarantees that the two directions of a connection, or 
>>>> the
>>>> +packets of a connection before and after a change of the ECMP nexthops (a 
>>>> node
>>>> +being added, removed, rebooted or simply an ECMP rehash), are handled by 
>>>> the
>>>> +same node. Any feature that needs the connection state to be on the node 
>>>> that
>>>> +processes the packet is therefore not supported on a replicated gateway:
>>>> +
>>>> +* load balancers,
>>>> +* stateful NAT, that is ``snat`` and ``dnat_and_snat`` (unless configured 
>>>> as
>>>> +  stateless), and NAT with ``ct-commit-all`` enabled,
>>>> +* stateful ACLs, and anything else that relies on conntrack, for example
>>>> +  load balancer hairpinning or ECMP routes with
>>>> +  ``options:ecmp_symmetric_reply`` set.
>>>> +
>>>> +Only stateless forwarding, e.g. plain routing and switching, stateless 
>>>> NAT and
>>>> +stateless ACLs, behaves correctly, because every packet can be handled
>>>> +independently by any node of the group.
>>>> +
>>>> +Example
>>>> +-------
>>>> +
>>>> +The ``ovn multinode replicated gateway`` test in ``tests/multinode.at`` 
>>>> sets up
>>>> +a complete example of a two node replicated gateway with an underlay ECMP 
>>>> route
>>>> +to the anycast encapsulation IP, a hypervisor with a VM behind the gateway
>>>> +and an external node that does not run OVN at all.
>>>> diff --git a/NEWS b/NEWS
>>>> index f1c56dd14..7bc3f2113 100644
>>>> --- a/NEWS
>>>> +++ b/NEWS
>>>> @@ -129,6 +129,11 @@ OVN v26.09.0 - xxx xx xxxx
>>>>        represent a logical router port (e.g. ovn-ic transit switch LSPs).
>>>>        Such ports are treated like type=router, including omission from
>>>>        _MC_flood_l2.
>>>> +   - Documented the "replicated gateway" setup, in which several 
>>>> ovn-controller
>>>> +     instances share one chassis name and one anycast encapsulation IP
>>>> +     reachable over the underlay through ECMP, so that they appear to OVN 
>>>> as
>>>> +     a single chassis. See Documentation/topics/replicated-gateway.rst 
>>>> for the
>>>> +     requirements and the restrictions.
>>>>   
>>>>   OVN v26.03.0 - xxx xx xxxx
>>>>   --------------------------
>>>> diff --git a/controller/ovn-controller.8.xml 
>>>> b/controller/ovn-controller.8.xml
>>>> index 3c33654ff..37c64ebf3 100644
>>>> --- a/controller/ovn-controller.8.xml
>>>> +++ b/controller/ovn-controller.8.xml
>>>> @@ -100,7 +100,14 @@
>>>>         local OVN "etc" directory or via the <code>-n</code> command-line 
>>>> option.
>>>>         The following precedence is used: first, the command-line option 
>>>> is read;
>>>>         if not present, the <code>system-id-override</code> file is read; 
>>>> if not
>>>> -      present, then the name configured in the database is used.</dd>
>>>> +      present, then the name configured in the database is used.
>>>> +      <p>
>>>> +        Several <code>ovn-controller</code> instances running on different
>>>> +        nodes may share the same <code>system-id</code>, so that they 
>>>> appear
>>>> +        to OVN as a single chassis backed by an anycast encapsulation IP.
>>>> +        See the "Replicated Gateway" document for the requirements and the
>>>> +        restrictions of such a setup.
>>>> +      </p></dd>
>>>>   
>>>>         <dt><code>external_ids:hostname</code></dt>
>>>>         <dd>The hostname to use in the Chassis table.</dd>
>>>> diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at
>>>> index 1c85f79f8..81ab2a5b8 100644
>>>> --- a/tests/multinode-macros.at
>>>> +++ b/tests/multinode-macros.at
>>>> @@ -296,23 +296,33 @@ multinode_setup_northd() {
>>>>       m_as $c ovn-sbctl set-connection p${REMOTE_PROT}:6642
>>>>   }
>>>>   
>>>> -# multinode_setup_controller NODE ENCAP_IP REMOTE_IP [ENCAP_TYPE]
>>>> -#
>>>> -# Sets up controller on specified node.
>>>> +# multinode_setup_controller NODE SYSTEM_ID ENCAP_IP REMOTE_IP
>>>> +#                            [ENCAP_TYPE] [START_CONTROLLER]
>>>> +#
>>>> +# Sets up controller on specified node. SYSTEM_ID defaults to NODE; 
>>>> passing a
>>>> +# different value (or the same value on more than one node) allows a test 
>>>> to
>>>> +# control the chassis name the node registers with.  START_CONTROLLER 
>>>> defaults
>>>> +# to "yes"; with "no" only OVS is set up, so that a test can finish the 
>>>> node
>>>> +# configuration before starting ovn-controller itself.
>>>>   multinode_setup_controller() {
>>>>       c=$1
>>>> +    system_id=${2:-$1}
>>>>       encap_ip=$3
>>>>       remote_ip=$4
>>>>       encap_type=${5:-"geneve"}
>>>> +    start_controller=${6:-"yes"}
>>>>   
>>>>       # Cleanup existing one
>>>>       m_as $c /usr/share/openvswitch/scripts/ovs-ctl stop
>>>>       m_as $c /usr/share/ovn/scripts/ovn-ctl stop_controller
>>>>       m_as $c sh -c "rm -f /etc/openvswitch/*.db"
>>>>   
>>>> -    m_as $c /usr/share/openvswitch/scripts/ovs-ctl start --system-id=$c
>>>> -    echo "Using ${CONTROLLER_SSL_ARGS} for ovn-controller".
>>>> -    m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller 
>>>> ${CONTROLLER_SSL_ARGS}
>>>> +    m_as $c /usr/share/openvswitch/scripts/ovs-ctl start 
>>>> --system-id=$system_id
>>>> +    if [[ "$start_controller" = "yes" ]]; then
>>>> +        echo "Using ${CONTROLLER_SSL_ARGS} for ovn-controller".
>>>> +        m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller \
>>>> +            ${CONTROLLER_SSL_ARGS}
>>>> +    fi
>>> I think if we move the part that starts ovn-controller to the end of
>>> this function we don't need the kind of ugly start_controller argument.
>>> We can just set everything we need and then start ovn-controller.
>>>
>>>>   
>>>>       m_as $c ovs-vsctl set open . external_ids:ovn-encap-ip=$encap_ip
>>>>       m_as $c ovs-vsctl set open . external-ids:ovn-encap-type=$encap_type
>>>> diff --git a/tests/multinode.at b/tests/multinode.at
>>>> index 0c277f5e8..16cfa804f 100644
>>>> --- a/tests/multinode.at
>>>> +++ b/tests/multinode.at
>>>> @@ -5554,3 +5554,200 @@ AT_CHECK([cat ch2_eth2.tcpdump], [0], [dnl
>>>>   ])
>>>>   
>>>>   AT_CLEANUP
>>>> +
>>>> +AT_SETUP([ovn multinode replicated gateway])
>>>> +
>>>> +check_fake_multinode_setup
>>>> +cleanup_multinode_resources
>>>> +
>>>> +# Two gateway nodes are deployed as one and the same "replicated" 
>>>> gateway: both
>>>> +# ovn-controllers register with the same chassis name and with the same 
>>>> anycast
>>>> +# encap IP. The encap IP is reachable over the underlay via ECMP through 
>>>> the
>>>> +# physical addresses of both gateway nodes, so a geneve tunnel to the 
>>>> replicated
>>>> +# gateway may land on either of them.
>>>> +#
>>>> +#                             underlay (eth1, 170.168.0.0/16)
>>>> +#  ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”                                   
>>>> ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”
>>>> +#  │  ovn-chassis-1    │   1.1.1.1/32                      │    ovn-gw-1  
>>>>     │
>>>> +#  │  (compute node)   ā”œā”€ā”€ā”€ā”€ nexthop via gw-1 eth1 ────────┤  
>>>> ovn-replicated-gw  │
>>>> +#  │                   │                                   │  encap 
>>>> 1.1.1.1   │
>>>> +#  │  sw0p1 10.0.0.3   ā”œā”€ā”€ā”€ā”€ nexthop via gw-2 eth1 ───┐    
>>>> ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜
>>>> +#  ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜                              │             │
>>>> +#                                                     │    
>>>> ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”“ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”
>>>> +#  ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”                              └────┤    ovn-gw-2  
>>>>     │
>>>> +#  │  ovn-chassis-2    │   1.1.1.1/32                      │  
>>>> ovn-replicated-gw  │
>>>> +#  │ (external client) ā”œā”€ā”€ā”€ā”€ nexthop via gw-1 eth1 ────────┤  encap 
>>>> 1.1.1.1   │
>>>> +#  │                   ā”œā”€ā”€ā”€ā”€ nexthop via gw-2 eth1 ────────┤              
>>>>     │
>>>> +#  │                   │                                   
>>>> ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜
>>>> +#  │  eth2 172.20.0.50 ā”œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ provider network (eth2, br-ex) ā”€ā”€ā”€ā”€ā”˜
>>>> +#  ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜                       lr0-public 172.20.0.100
>>>> +#
>>>> +# On ovn-chassis-2 (aka client) we have client port in public switch for 
>>>> checking
>>>> +# internal connectivity, public switch is connected to dgp port that 
>>>> claimed by
>>>> +# replicated chassis. Compute and client node both have underlay route 
>>>> for replicated gw ip (1.1.1.1)
>>>> +
>>>> +vgw1=ovn-gw-1
>>>> +vgw2=ovn-gw-2
>>>> +compute=ovn-chassis-1
>>>> +client=ovn-chassis-2
>>>> +vgw_name=ovn-replicated-gw
>>>> +vgw_encap_ip=1.1.1.1
>>>> +
>>>> +m_underlay_ip() {
>>>> +    m_as $1 ip -4 addr show eth1 | grep inet | awk '{print $2}' | cut 
>>>> -d'/' -f1
>>>> +}
>>>> +
>>>> +central_ip=$(m_as ovn-central-az1 ip -4 addr show eth1 | grep inet | awk 
>>>> '{print $2}' | cut -d'/' -f1)
>>> This should now be:
>>>
>>> central_ip=$( (m_central_as ip -4 addr show eth1 2>/dev/null ||
>>> m_central_as ip -4 addr show eth2) | grep inet | awk '{print $2}' | cut
>>> -d'/' -f1)
>>>
>>> Since we merged 44d3bf28e4b7 ("multinode: Add support for INLINE_BR_EX
>>> configuration of multinode.")
>>>
>>>> +vgw1_ip=$(m_underlay_ip $vgw1)
>>>> +vgw2_ip=$(m_underlay_ip $vgw2)
>>>> +compute_ip=$(m_underlay_ip $compute)
>>>> +
>>>> +m_as $compute ip link del sw0p1-p
>>>> +m_as $client ip link del sw0p2-p
>>>> +m_as $client ip link del sw1p1-p
>>>> +
>>>> +# Bring up both gateway nodes as the very same chassis: same chassis name 
>>>> and
>>>> +# same encap IP, which is not part of the underlay address space.
>>>> +for c in $vgw1 $vgw2; do
>>>> +    # Bring the node back to its original chassis name and encap IP.
>>>> +    on_exit "multinode_setup_controller $c $c $(m_underlay_ip $c) 
>>>> $central_ip"
>>>> +    multinode_setup_controller $c $vgw_name $vgw_encap_ip $central_ip 
>>>> geneve no
>>>> +    on_exit "m_as $c ip addr del $vgw_encap_ip/32 dev lo"
>>>> +    check m_as $c ip addr add $vgw_encap_ip/32 dev lo
>>>> +    check m_as $c ovs-vsctl set open . external-ids:hostname=$vgw_name
>>>> +    check m_as $c ovs-vsctl set open . 
>>>> external-ids:ovn-bridge-mappings=public:br-ex
>>> I don't think your test uses the ovn-bridge mapping in any way.  So then
>>> we might as well add hostname as an optional argument to
>>> multinode_setup_controller().
>>>
>>>> +done
>>>> +
>>>> +for c in $vgw1 $vgw2; do
>>>> +    m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller 
>>>> ${CONTROLLER_SSL_ARGS}
>>>> +done
>>> If we change multinode_setup_controller() as I mentioned above, we don't
>>> need to manually start it here.
>>>
>>>> +
>>>> +# Both gateway nodes must end up sharing a single chassis record with a 
>>>> single
>>>> +# encap record.
>>>> +m_wait_row_count Chassis 1 name=$vgw_name
>>>> +m_wait_row_count Encap 1 chassis_name=$vgw_name
>>>> +m_check_column "$vgw_encap_ip" Encap ip chassis_name=$vgw_name
>>>> +
>>>> +# ECMP route to the replicated gateway encap IP in the underlay, both on 
>>>> the
>>>> +# compute node and on the external client.
>>>> +for c in $compute $client; do
>>>> +    on_exit "m_as $c ip route del $vgw_encap_ip/32"
>>>> +    check m_as $c ip route replace $vgw_encap_ip/32 \
>>>> +        nexthop via $vgw1_ip dev eth1 \
>>>> +        nexthop via $vgw2_ip dev eth1
>>>> +done
>>>> +
>>>> +
>>> Nit: one empty line too many.
>>>
>>>> +# The anycast encap IP is reachable over the underlay from both nodes.
>>>> +AT_CHECK([m_as $compute ping -q -c 3 -i 0.3 -w 2 $vgw_encap_ip | 
>>>> FORMAT_PING], [0], [dnl
>>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
>>>> +])
>>>> +AT_CHECK([m_as $client ping -q -c 3 -i 0.3 -w 2 $vgw_encap_ip | 
>>>> FORMAT_PING], [0], [dnl
>>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
>>>> +])
>>>> +
>>>> +cr_lport_log_count() {
>>>> +    m_as $1 sh -c "cat /var/log/ovn/ovn-controller.log 2>/dev/null | \
>>>> +                   grep -ac 'Claiming lport cr-lr0-public'"
>>>> +}
>>>> +
>>>> +vgw1_cr_logs=$(cr_lport_log_count $vgw1)
>>>> +vgw2_cr_logs=$(cr_lport_log_count $vgw2)
>>>> +
>>>> +check multinode_nbctl ls-add sw0
>>>> +check multinode_nbctl lsp-add sw0 sw0-port1
>>>> +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 
>>>> 10.0.0.3"
>>>> +
>>>> +check multinode_nbctl lr-add lr0
>>>> +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24
>>>> +check multinode_nbctl lsp-add-router-port sw0 sw0-lr0 lr0-sw0
>>>> +
>>>> +check multinode_nbctl ls-add public
>>>> +check multinode_nbctl lsp-add public external_client
>>>> +check multinode_nbctl lsp-set-addresses external_client 
>>>> "50:55:00:00:00:03 172.20.0.99"
>>>> +check multinode_nbctl lrp-add lr0 lr0-public 00:11:22:00:ff:01 
>>>> 172.20.0.100/24
>>>> +check multinode_nbctl lsp-add-router-port public public-lr0 lr0-public
>>>> +check multinode_nbctl lrp-set-gateway-chassis lr0-public $vgw_name 10
>>>> +
>>>> +m_as $compute /data/create_fake_vm.sh sw0-port1 sw0p1 50:54:00:00:00:03 
>>>> 1342 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a
>>>> +m_as $client /data/create_fake_vm.sh external_client external 
>>>> 50:55:00:00:00:03 1342 172.20.0.99 24 172.20.0.100 1001::3/64 1001::a
>>>> +
>>>> +check multinode_nbctl --wait=hv sync
>>>> +m_wait_for_ports_up sw0-port1
>>>> +m_wait_for_ports_up external_client
>>>> +
>>>> +# The distributed gateway port is claimed by the replicated gateway 
>>>> chassis.
>>>> +m_wait_row_count Port_Binding 1 logical_port=cr-lr0-public
>>>> +vgw_uuid=$(m_central_as ovn-sbctl --bare --columns _uuid find Chassis 
>>>> name=$vgw_name)
>>>> +m_wait_column "$vgw_uuid" Port_Binding chassis logical_port=cr-lr0-public
>>>> +
>>>> +# The two controllers share a single Chassis record and a single binding 
>>>> for
>>>> +# the chassisredirect port.  Since both nodes are configured identically 
>>>> and
>>>> +# both resolve to the same chassis, each of them finds the records already
>>>> +# holding the values it was about to write and its transaction is a 
>>>> no-op, so
>>>> +# once the setup has settled no transaction touching those tables is 
>>>> expected.
>>>> +# If the nodes fought over a record, they would add one transaction per 
>>>> main
>>>> +# loop iteration to the southbound database log.
>>>> +sb_txn_count() {
>>>> +    m_central_as grep -ac "\"$1\":" /etc/ovn/ovnsb_db.db
>>>> +}
>>>> +
>>>> +chassis_txns=$(sb_txn_count Chassis)
>>>> +pb_txns=$(sb_txn_count Port_Binding)
>>>> +sleep 5
>>>> +check test $(($(sb_txn_count Chassis) - chassis_txns)) -le 1
>>>> +check test $(($(sb_txn_count Port_Binding) - pb_txns)) -le 1
>>>> +
>>>> +cr_logs=$(($(cr_lport_log_count $vgw1) + $(cr_lport_log_count $vgw2)))
>>>> +# Claim on both
>>>> +check test $((cr_logs - vgw1_cr_logs - vgw2_cr_logs)) -le 2
>>>> +
>>>> +check multinode_nbctl --wait=hv sync
>>>> +
>>>> +# North-south connectivity through the replicated gateway, with the 
>>>> underlay ECMP
>>>> +# route in place on both ends.
>>>> +check_ns_connectivity() {
>>>> +    M_NS_CHECK_EXEC([$compute], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 
>>>> 172.20.0.99 | FORMAT_PING], \
>>>> +[0], [dnl
>>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
>>>> +])
>>>> +
>>>> +    M_NS_CHECK_EXEC([$compute], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 
>>>> 172.20.0.100 | FORMAT_PING], \
>>>> +[0], [dnl
>>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
>>>> +])
>>>> +
>>>> +    M_NS_CHECK_EXEC([$client], [external], [ping -q -c 3 -i 0.3 -w 2 
>>>> 10.0.0.3 | FORMAT_PING], \
>>>> +[0], [dnl
>>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
>>>> +])
>>>> +}
>>>> +
>>>> +# The traffic towards the replicated gateway must be tunneled to the 
>>>> anycast encap IP.
>>>> +M_START_TCPDUMPS([$compute], [-c 2 -neei eth1 host $vgw_encap_ip and udp 
>>>> port 6081], [ch1_underlay])
>>>> +
>>>> +check_ns_connectivity
>>>> +
>>>> +OVS_WAIT_UNTIL([test -s ch1_underlay.tcpdump])
>>>> +AT_CHECK([grep -q "$vgw_encap_ip" ch1_underlay.tcpdump])
>>>> +
>>>> +# Each of the two replicated gateway nodes on its own must be able to 
>>>> carry the
>>>> +# traffic: pin the underlay route to a single nexthop at a time.
>>>> +for nexthop in $vgw1_ip $vgw2_ip; do
>>>> +    for c in $compute $client; do
>>>> +        check m_as $c ip route replace $vgw_encap_ip/32 via $nexthop dev 
>>>> eth1
>>>> +    done
>>>> +
>>>> +    check_ns_connectivity
>>>> +done
>>>> +
>>>> +# Back to ECMP over both gateway nodes.
>>>> +for c in $compute $client; do
>>>> +    check m_as $c ip route replace $vgw_encap_ip/32 \
>>>> +        nexthop via $vgw1_ip dev eth1 \
>>>> +        nexthop via $vgw2_ip dev eth1
>>>> +done
>>>> +
>>>> +check_ns_connectivity
>>>> +
>>>> +AT_CLEANUP
>>

_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Reply via email to