On 09.11.2026 14:04, Dumitru Ceara wrote: > On 11/9/26 12:01 PM, Dumitru Ceara wrote: >> On 11/9/26 10:17 AM, Rukomoinikova Aleksandra wrote: >>> On 07.11.2026 00:33, Dumitru Ceara wrote: >>>> On 9/28/26 11:45 AM, Alexandra Rukomoinikova via dev wrote: >>>>> Several ovn-controller instances may run with the same chassis name and >>>>> the same encapsulation IP, an anycast address that is reachable over the >>>>> underlay through ECMP. OVN then sees the whole group of nodes as a >>>>> single chassis: a single Chassis record, a single Encap record and a >>>>> single binding for every port placed on it, while the underlay decides >>>>> which node of the group terminates a given tunnel flow. >>>>> >>>>> There is no coordination and no leader election between the nodes of the >>>>> group while writing to SBDB: every ``ovn-controller`` independently does >>>>> the >>>>> full bookkeeping for that one chassis, and it is the idempotence of those >>>>> writes that keeps them from fighting each other. >>>>> >>>>> The flip side is that the nodes must be configured identically in >>>>> chassis-related params, and that nothing which relies on conntrack can be >>>>> used, because the group shares no runtime state. >>>>> >>>>> Document all of this, and add a multinode test that deploys two >>>>> gateway nodes as one replicated gateway, gives a hypervisor and a node >>>>> that >>>>> does not run OVN an underlay ECMP route to the anycast encapsulation IP, >>>>> checks the north-south connectivity through each of the two nodes and >>>>> checks that the controllers do not fight over the shared records. >>>>> >>>>> Signed-off-by: Alexandra Rukomoinikova <[email protected]> >>>>> --- >>>> Hi Alexandra, >>>> >>>> I have a few minor comments on this patch, please see below. >>>> >>>> I was thinking of taking care of them myself and squash a small fixup >>>> patch in when pushing yours to main: >>>> >>>> https://github.com/dceara/ovn/commit/42b64d5 >>>> >>>> I didn't test this locally but CI is running in my fork: >>>> >>>> https://github.com/dceara/ovn/actions/runs/42295260684 >>>> >>>> Assuming that passes would you be OK with this incremental change? >>>> >>>> Regards, >>>> Dumitru >> Hi Sasha, >> >>> Hi Dumitru! I took a fresh look at this patch, and Iβd like to remove >>> all references to ECMP β anycast IP and drop the mention of using BGP >>> for announcements - this seems unnecessary. Is it still okay to make >> I actually liked the ECMP reference, maybe it's an idea to still mention >> that and BGP as an example? I'm not against removing it though. >> > While at it, maybe it's worth also mentioning that other (stateful) > features might not work as expected in such deployments. E.g., IGMP. > >>> those changes rn? I can take the fixup based on fixups in your fork, >>> make the changes on top of it, and send you the updated patch. π >>> >> I haven't merged your patch yet because CI didn't pass, one of the lanes >> still hangs and I didn't have time to look into it: >> >> https://github.com/dceara/ovn/actions/runs/42295260684/job/129055547103 >> >> So if you could take the fixup from my fork and work on a v2 that'd be >> great. >> >> Thanks, >> Dumitru
Iβll look into why the test is hanging and send a v2. Regarding BGP, I meant this part: >>/+ In a real deployment the route is normally learned from the fabric, with the />>/+ gateway nodes advertising the anycast prefix over BGP./ Iβd like to fix this part and remove the mention of BGP from it. If youβre liked with the ECMP part, Iβll leave it as it is. In that case, Iβll just go through the test and remove BGP from the example above. thanks, sasha >> >>> thanks, sasha >>> >>> >>>>> Documentation/automake.mk | 1 + >>>>> Documentation/topics/index.rst | 1 + >>>>> Documentation/topics/replicated-gateway.rst | 171 +++++++++++++++++ >>>>> NEWS | 5 + >>>>> controller/ovn-controller.8.xml | 9 +- >>>>> tests/multinode-macros.at | 22 ++- >>>>> tests/multinode.at | 197 ++++++++++++++++++++ >>>>> 7 files changed, 399 insertions(+), 7 deletions(-) >>>>> create mode 100644 Documentation/topics/replicated-gateway.rst >>>>> >>>>> diff --git a/Documentation/automake.mk b/Documentation/automake.mk >>>>> index 685ba9588..b64a0193d 100644 >>>>> --- a/Documentation/automake.mk >>>>> +++ b/Documentation/automake.mk >>>>> @@ -25,6 +25,7 @@ DOC_SOURCE = \ >>>>> Documentation/topics/testing.rst \ >>>>> Documentation/topics/test-development.rst \ >>>>> Documentation/topics/high-availability.rst \ >>>>> + Documentation/topics/replicated-gateway.rst \ >>>>> Documentation/topics/dynamic-routing/architecture.rst \ >>>>> Documentation/topics/dynamic-routing/configuration-examples.rst >>>>> \ >>>>> Documentation/topics/dynamic-routing/index.rst \ >>>>> diff --git a/Documentation/topics/index.rst >>>>> b/Documentation/topics/index.rst >>>>> index 0b676a98b..d5210e03a 100644 >>>>> --- a/Documentation/topics/index.rst >>>>> +++ b/Documentation/topics/index.rst >>>>> @@ -40,6 +40,7 @@ OVN >>>>> dynamic-routing/index >>>>> incremental-processing/index >>>>> high-availability >>>>> + replicated-gateway >>>>> role-based-access-control >>>>> vif-plug-providers/index >>>>> testing >>>>> diff --git a/Documentation/topics/replicated-gateway.rst >>>>> b/Documentation/topics/replicated-gateway.rst >>>>> new file mode 100644 >>>>> index 000000000..c3e43da03 >>>>> --- /dev/null >>>>> +++ b/Documentation/topics/replicated-gateway.rst >>>>> @@ -0,0 +1,171 @@ >>>>> +.. >>>>> + Licensed under the Apache License, Version 2.0 (the "License"); >>>>> you may >>>>> + not use this file except in compliance with the License. You may >>>>> obtain >>>>> + a copy of the License at >>>>> + >>>>> + http://www.apache.org/licenses/LICENSE-2.0 >>>>> + >>>>> + Unless required by applicable law or agreed to in writing, software >>>>> + distributed under the License is distributed on an "AS IS" BASIS, >>>>> WITHOUT >>>>> + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. >>>>> See the >>>>> + License for the specific language governing permissions and >>>>> limitations >>>>> + under the License. >>>>> + >>>>> + Convention for heading levels in OVN documentation: >>>>> + >>>>> + ======= Heading 0 (reserved for the title in a document) >>>>> + ------- Heading 1 >>>>> + ~~~~~~~ Heading 2 >>>>> + +++++++ Heading 3 >>>>> + ''''''' Heading 4 >>>>> + >>>>> + Avoid deeper levels because they do not render well. >>>>> + >>>>> +=================== >>>>> +Replicated Gateway >>>>> +=================== >>>> Nit: one = too many. >>>> >>>>> + >>>>> +A *replicated gateway* is a group of two or more physical nodes that >>>>> present >>>>> +themselves to OVN as a single chassis. Every node of the group runs its >>>>> own >>>>> +``ovn-controller``, but all of them are configured with the same chassis >>>>> name >>>>> +(``external_ids:system-id``) and with the same encapsulation IP >>>>> +(``external_ids:ovn-encap-ip``). The encapsulation IP is an anycast >>>>> address, >>>>> +it is announced into the underlay (statically or by a routing protocol) >>>>> by all >>>>> +the nodes of the group, and the underlay load-balances the tunnel traffic >>>>> +between them with ECMP. >>>>> + >>>>> +:: >>>>> + >>>>> + underlay with ECMP to 10.0.0.1 >>>>> + >>>>> + +-------------------+ +----------------------+ >>>>> + | | nexthop via gw-1 | gateway node 1 | >>>>> + | hypervisor +---------------------->| system-id: vgw | >>>>> + | | | ovn-encap-ip: | >>>>> + | ovn-encap-ip: | | 10.0.0.1 | >>>>> + | 192.168.0.10 | +----------+-----------+ >>>>> + | | | >>>>> + | | nexthop via gw-2 +----------+-----------+ >>>>> + | +---------------------->| gateway node 2 | >>>>> + | | | system-id: vgw | >>>>> + +-------------------+ | ovn-encap-ip: | >>>>> + | 10.0.0.1 | >>>>> + +----------+-----------+ >>>>> + | >>>>> + physical network | >>>>> + -------------------------------------------------------+------------ >>>>> + >>>>> +From the point of view of the OVN databases there is a single chassis: a >>>>> single >>>>> +``Chassis`` record, a single ``Encap`` record, and a single binding for >>>>> every >>>>> +port that is placed on the replicated gateway (for example the >>>>> chassisredirect >>>>> +port of a distributed gateway port). ``ovn-northd`` and the other >>>>> +``ovn-controller`` instances do not know, and do not need to know, how >>>>> many >>>>> +physical nodes are behind that chassis. A hypervisor keeps a single >>>>> tunnel to >>>>> +the anycast encapsulation IP, and the underlay decides which node of the >>>>> group >>>>> +terminates a given flow. >>>>> + >>>>> +There is no coordination and no leader election between the nodes of the >>>>> +group while writing to SBDB: every ``ovn-controller`` independently does >>>>> the >>>>> +full bookkeeping for that one chassis, and it is the idempotence of those >>>>> +writes that keeps them from fighting each other. >>>>> + >>>>> +Two consequences follow from this. First, the columns that OVN derives >>>>> from >>>>> +the local node rather than from the configuration must be forced to the >>>>> same >>>>> +value on every node. More detailed information on this is provided in >>>>> the >>>>> +``Identical OVS and OVN configuration and features`` section below. >>>>> Second, >>>>> +the nodes do not share any runtime state, so nothing that relies on >>>>> conntrack >>>>> +can be used on a replicated gateway, see the ``Only stateless traffic is >>>>> +supported`` section below. >>>> We shouldn't break in the middle of the verbatim section, that renders >>>> weird. >>>> >>>>> + >>>>> +Use case >>>>> +-------- >>>>> +This is useful when the capacity or the availability of a single gateway >>>>> node >>>>> +is not enough: the group scales horizontally, and losing one node only >>>>> means >>>>> +that the underlay stops using one of the ECMP nexthops, without any >>>>> change in >>>>> +the OVN databases and without the gateway having to be rescheduled to >>>>> another >>>>> +chassis. >>>>> + >>>>> +A typical deployment looks as follows: >>>>> + >>>>> +* Two (or more) gateway nodes are installed with the same OVN and OVS >>>>> + versions and with an identical ``ovn-controller`` configuration. Each >>>>> of >>>>> + them configures the anycast encapsulation IP locally, e.g. on the >>>>> loopback >>>>> + interface, so that the kernel accepts the tunnel packets destined to >>>>> it:: >>>>> + >>>>> + $ ovs-vsctl set open . external-ids:system-id=vgw >>>>> + $ ovs-vsctl set open . external-ids:ovn-encap-ip=10.0.0.1 >>>>> + $ ovs-vsctl set open . external-ids:ovn-encap-type=geneve >>>>> + $ ip address add 10.0.0.1/32 dev lo >>>>> + >>>>> +* The anycast address is reachable in the underlay through the physical >>>>> + addresses of all the nodes of the group. On the hypervisors, and on any >>>>> + other node that has to reach the replicated gateway, this is an ECMP >>>>> route:: >>>>> + >>>>> + $ ip route replace 10.0.0.1/32 \ >>>>> + nexthop via 192.168.0.1 dev eth0 \ >>>>> + nexthop via 192.168.0.2 dev eth0 >>>>> + >>>>> + In a real deployment the route is normally learned from the fabric, >>>>> with the >>>>> + gateway nodes advertising the anycast prefix over BGP. >>>>> + >>>>> +* The logical topology is configured as usual, referring to the chassis >>>>> by its >>>>> + single name, e.g.:: >>>>> + >>>>> + $ ovn-nbctl lrp-set-gateway-chassis lr0-public vgw 10 >>>>> + >>>>> +Requirements and restrictions >>>>> +----------------------------- >>>>> + >>>>> +The nodes of a replicated gateway are not independent chassis, they are >>>>> one >>>>> +chassis that happens to run on several machines. Everything that OVN >>>>> stores >>>>> +per chassis is therefore shared, and everything that OVN computes per >>>>> chassis >>>>> +must produce the same result on every node of the group. This leads to >>>>> the >>>>> +following requirements. >>>>> + >>>>> +Identical OVS and OVN configuration and features >>>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>> + >>>>> +All the nodes of the group must run the same OVS and OVN versions and >>>>> must >>>>> +expose the same set of features. ``ovn-controller`` reports the >>>>> capabilities >>>>> +of the local OVS instance in the ``Chassis`` record (``other_config``, >>>>> for >>>>> +example the supported datapath interface types, and the various >>>>> +``ovn-chassis-feature`` entries), and ``ovn-northd`` generates the >>>>> logical >>>>> +flows for the chassis based on those values. If the nodes disagree, the >>>>> +``Chassis`` record will keep flapping between the values reported by >>>>> each node, >>>>> +``ovn-northd`` will keep recomputing, and the node whose capabilities do >>>>> not >>>>> +match the ones currently published may not be able to process the flows >>>>> it >>>>> +receives. >>>>> + >>>>> +For the same reason, every chassis scoped configuration option has to be >>>>> +identical on all the nodes, including, for example: >>>>> ``external_ids:hostname``, >>>>> +``external_ids:ovn-bridge-mappings`` and all others. >>>>> + >>>>> +Only stateless traffic is supported >>>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>> + >>>>> +The nodes of a replicated gateway do not share any runtime state, and in >>>>> +particular they do not share conntrack. The underlay picks one of them >>>>> per >>>>> +flow, and nothing guarantees that the two directions of a connection, or >>>>> the >>>>> +packets of a connection before and after a change of the ECMP nexthops >>>>> (a node >>>>> +being added, removed, rebooted or simply an ECMP rehash), are handled by >>>>> the >>>>> +same node. Any feature that needs the connection state to be on the node >>>>> that >>>>> +processes the packet is therefore not supported on a replicated gateway: >>>>> + >>>>> +* load balancers, >>>>> +* stateful NAT, that is ``snat`` and ``dnat_and_snat`` (unless >>>>> configured as >>>>> + stateless), and NAT with ``ct-commit-all`` enabled, >>>>> +* stateful ACLs, and anything else that relies on conntrack, for example >>>>> + load balancer hairpinning or ECMP routes with >>>>> + ``options:ecmp_symmetric_reply`` set. >>>>> + >>>>> +Only stateless forwarding, e.g. plain routing and switching, stateless >>>>> NAT and >>>>> +stateless ACLs, behaves correctly, because every packet can be handled >>>>> +independently by any node of the group. >>>>> + >>>>> +Example >>>>> +------- >>>>> + >>>>> +The ``ovn multinode replicated gateway`` test in ``tests/multinode.at`` >>>>> sets up >>>>> +a complete example of a two node replicated gateway with an underlay >>>>> ECMP route >>>>> +to the anycast encapsulation IP, a hypervisor with a VM behind the >>>>> gateway >>>>> +and an external node that does not run OVN at all. >>>>> diff --git a/NEWS b/NEWS >>>>> index f1c56dd14..7bc3f2113 100644 >>>>> --- a/NEWS >>>>> +++ b/NEWS >>>>> @@ -129,6 +129,11 @@ OVN v26.09.0 - xxx xx xxxx >>>>> represent a logical router port (e.g. ovn-ic transit switch LSPs). >>>>> Such ports are treated like type=router, including omission from >>>>> _MC_flood_l2. >>>>> + - Documented the "replicated gateway" setup, in which several >>>>> ovn-controller >>>>> + instances share one chassis name and one anycast encapsulation IP >>>>> + reachable over the underlay through ECMP, so that they appear to >>>>> OVN as >>>>> + a single chassis. See Documentation/topics/replicated-gateway.rst >>>>> for the >>>>> + requirements and the restrictions. >>>>> >>>>> OVN v26.03.0 - xxx xx xxxx >>>>> -------------------------- >>>>> diff --git a/controller/ovn-controller.8.xml >>>>> b/controller/ovn-controller.8.xml >>>>> index 3c33654ff..37c64ebf3 100644 >>>>> --- a/controller/ovn-controller.8.xml >>>>> +++ b/controller/ovn-controller.8.xml >>>>> @@ -100,7 +100,14 @@ >>>>> local OVN "etc" directory or via the <code>-n</code> >>>>> command-line option. >>>>> The following precedence is used: first, the command-line option >>>>> is read; >>>>> if not present, the <code>system-id-override</code> file is >>>>> read; if not >>>>> - present, then the name configured in the database is used.</dd> >>>>> + present, then the name configured in the database is used. >>>>> + <p> >>>>> + Several <code>ovn-controller</code> instances running on >>>>> different >>>>> + nodes may share the same <code>system-id</code>, so that they >>>>> appear >>>>> + to OVN as a single chassis backed by an anycast encapsulation IP. >>>>> + See the "Replicated Gateway" document for the requirements and >>>>> the >>>>> + restrictions of such a setup. >>>>> + </p></dd> >>>>> >>>>> <dt><code>external_ids:hostname</code></dt> >>>>> <dd>The hostname to use in the Chassis table.</dd> >>>>> diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at >>>>> index 1c85f79f8..81ab2a5b8 100644 >>>>> --- a/tests/multinode-macros.at >>>>> +++ b/tests/multinode-macros.at >>>>> @@ -296,23 +296,33 @@ multinode_setup_northd() { >>>>> m_as $c ovn-sbctl set-connection p${REMOTE_PROT}:6642 >>>>> } >>>>> >>>>> -# multinode_setup_controller NODE ENCAP_IP REMOTE_IP [ENCAP_TYPE] >>>>> -# >>>>> -# Sets up controller on specified node. >>>>> +# multinode_setup_controller NODE SYSTEM_ID ENCAP_IP REMOTE_IP >>>>> +# [ENCAP_TYPE] [START_CONTROLLER] >>>>> +# >>>>> +# Sets up controller on specified node. SYSTEM_ID defaults to NODE; >>>>> passing a >>>>> +# different value (or the same value on more than one node) allows a >>>>> test to >>>>> +# control the chassis name the node registers with. START_CONTROLLER >>>>> defaults >>>>> +# to "yes"; with "no" only OVS is set up, so that a test can finish the >>>>> node >>>>> +# configuration before starting ovn-controller itself. >>>>> multinode_setup_controller() { >>>>> c=$1 >>>>> + system_id=${2:-$1} >>>>> encap_ip=$3 >>>>> remote_ip=$4 >>>>> encap_type=${5:-"geneve"} >>>>> + start_controller=${6:-"yes"} >>>>> >>>>> # Cleanup existing one >>>>> m_as $c /usr/share/openvswitch/scripts/ovs-ctl stop >>>>> m_as $c /usr/share/ovn/scripts/ovn-ctl stop_controller >>>>> m_as $c sh -c "rm -f /etc/openvswitch/*.db" >>>>> >>>>> - m_as $c /usr/share/openvswitch/scripts/ovs-ctl start --system-id=$c >>>>> - echo "Using ${CONTROLLER_SSL_ARGS} for ovn-controller". >>>>> - m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller >>>>> ${CONTROLLER_SSL_ARGS} >>>>> + m_as $c /usr/share/openvswitch/scripts/ovs-ctl start >>>>> --system-id=$system_id >>>>> + if [[ "$start_controller" = "yes" ]]; then >>>>> + echo "Using ${CONTROLLER_SSL_ARGS} for ovn-controller". >>>>> + m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller \ >>>>> + ${CONTROLLER_SSL_ARGS} >>>>> + fi >>>> I think if we move the part that starts ovn-controller to the end of >>>> this function we don't need the kind of ugly start_controller argument. >>>> We can just set everything we need and then start ovn-controller. >>>> >>>>> >>>>> m_as $c ovs-vsctl set open . external_ids:ovn-encap-ip=$encap_ip >>>>> m_as $c ovs-vsctl set open . >>>>> external-ids:ovn-encap-type=$encap_type >>>>> diff --git a/tests/multinode.at b/tests/multinode.at >>>>> index 0c277f5e8..16cfa804f 100644 >>>>> --- a/tests/multinode.at >>>>> +++ b/tests/multinode.at >>>>> @@ -5554,3 +5554,200 @@ AT_CHECK([cat ch2_eth2.tcpdump], [0], [dnl >>>>> ]) >>>>> >>>>> AT_CLEANUP >>>>> + >>>>> +AT_SETUP([ovn multinode replicated gateway]) >>>>> + >>>>> +check_fake_multinode_setup >>>>> +cleanup_multinode_resources >>>>> + >>>>> +# Two gateway nodes are deployed as one and the same "replicated" >>>>> gateway: both >>>>> +# ovn-controllers register with the same chassis name and with the same >>>>> anycast >>>>> +# encap IP. The encap IP is reachable over the underlay via ECMP through >>>>> the >>>>> +# physical addresses of both gateway nodes, so a geneve tunnel to the >>>>> replicated >>>>> +# gateway may land on either of them. >>>>> +# >>>>> +# underlay (eth1, 170.168.0.0/16) >>>>> +# βββββββββββββββββββββ >>>>> ββββββββββββββββββββ >>>>> +# β ovn-chassis-1 β 1.1.1.1/32 β ovn-gw-1 >>>>> β >>>>> +# β (compute node) βββββ nexthop via gw-1 eth1 βββββββββ€ >>>>> ovn-replicated-gw β >>>>> +# β β β encap >>>>> 1.1.1.1 β >>>>> +# β sw0p1 10.0.0.3 βββββ nexthop via gw-2 eth1 ββββ >>>>> ββββββββββ¬ββββββββββ >>>>> +# βββββββββββββββββββββ β β >>>>> +# β >>>>> ββββββββββ΄ββββββββββ >>>>> +# βββββββββββββββββββββ ββββββ€ ovn-gw-2 >>>>> β >>>>> +# β ovn-chassis-2 β 1.1.1.1/32 β >>>>> ovn-replicated-gw β >>>>> +# β (external client) βββββ nexthop via gw-1 eth1 βββββββββ€ encap >>>>> 1.1.1.1 β >>>>> +# β βββββ nexthop via gw-2 eth1 βββββββββ€ >>>>> β >>>>> +# β β >>>>> ββββββββββ¬ββββββββββ >>>>> +# β eth2 172.20.0.50 ββββββββββ provider network (eth2, br-ex) βββββ >>>>> +# βββββββββββββββββββββ lr0-public 172.20.0.100 >>>>> +# >>>>> +# On ovn-chassis-2 (aka client) we have client port in public switch for >>>>> checking >>>>> +# internal connectivity, public switch is connected to dgp port that >>>>> claimed by >>>>> +# replicated chassis. Compute and client node both have underlay route >>>>> for replicated gw ip (1.1.1.1) >>>>> + >>>>> +vgw1=ovn-gw-1 >>>>> +vgw2=ovn-gw-2 >>>>> +compute=ovn-chassis-1 >>>>> +client=ovn-chassis-2 >>>>> +vgw_name=ovn-replicated-gw >>>>> +vgw_encap_ip=1.1.1.1 >>>>> + >>>>> +m_underlay_ip() { >>>>> + m_as $1 ip -4 addr show eth1 | grep inet | awk '{print $2}' | cut >>>>> -d'/' -f1 >>>>> +} >>>>> + >>>>> +central_ip=$(m_as ovn-central-az1 ip -4 addr show eth1 | grep inet | awk >>>>> '{print $2}' | cut -d'/' -f1) >>>> This should now be: >>>> >>>> central_ip=$( (m_central_as ip -4 addr show eth1 2>/dev/null || >>>> m_central_as ip -4 addr show eth2) | grep inet | awk '{print $2}' | cut >>>> -d'/' -f1) >>>> >>>> Since we merged 44d3bf28e4b7 ("multinode: Add support for INLINE_BR_EX >>>> configuration of multinode.") >>>> >>>>> +vgw1_ip=$(m_underlay_ip $vgw1) >>>>> +vgw2_ip=$(m_underlay_ip $vgw2) >>>>> +compute_ip=$(m_underlay_ip $compute) >>>>> + >>>>> +m_as $compute ip link del sw0p1-p >>>>> +m_as $client ip link del sw0p2-p >>>>> +m_as $client ip link del sw1p1-p >>>>> + >>>>> +# Bring up both gateway nodes as the very same chassis: same chassis >>>>> name and >>>>> +# same encap IP, which is not part of the underlay address space. >>>>> +for c in $vgw1 $vgw2; do >>>>> + # Bring the node back to its original chassis name and encap IP. >>>>> + on_exit "multinode_setup_controller $c $c $(m_underlay_ip $c) >>>>> $central_ip" >>>>> + multinode_setup_controller $c $vgw_name $vgw_encap_ip $central_ip >>>>> geneve no >>>>> + on_exit "m_as $c ip addr del $vgw_encap_ip/32 dev lo" >>>>> + check m_as $c ip addr add $vgw_encap_ip/32 dev lo >>>>> + check m_as $c ovs-vsctl set open . external-ids:hostname=$vgw_name >>>>> + check m_as $c ovs-vsctl set open . >>>>> external-ids:ovn-bridge-mappings=public:br-ex >>>> I don't think your test uses the ovn-bridge mapping in any way. So then >>>> we might as well add hostname as an optional argument to >>>> multinode_setup_controller(). >>>> >>>>> +done >>>>> + >>>>> +for c in $vgw1 $vgw2; do >>>>> + m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller >>>>> ${CONTROLLER_SSL_ARGS} >>>>> +done >>>> If we change multinode_setup_controller() as I mentioned above, we don't >>>> need to manually start it here. >>>> >>>>> + >>>>> +# Both gateway nodes must end up sharing a single chassis record with a >>>>> single >>>>> +# encap record. >>>>> +m_wait_row_count Chassis 1 name=$vgw_name >>>>> +m_wait_row_count Encap 1 chassis_name=$vgw_name >>>>> +m_check_column "$vgw_encap_ip" Encap ip chassis_name=$vgw_name >>>>> + >>>>> +# ECMP route to the replicated gateway encap IP in the underlay, both on >>>>> the >>>>> +# compute node and on the external client. >>>>> +for c in $compute $client; do >>>>> + on_exit "m_as $c ip route del $vgw_encap_ip/32" >>>>> + check m_as $c ip route replace $vgw_encap_ip/32 \ >>>>> + nexthop via $vgw1_ip dev eth1 \ >>>>> + nexthop via $vgw2_ip dev eth1 >>>>> +done >>>>> + >>>>> + >>>> Nit: one empty line too many. >>>> >>>>> +# The anycast encap IP is reachable over the underlay from both nodes. >>>>> +AT_CHECK([m_as $compute ping -q -c 3 -i 0.3 -w 2 $vgw_encap_ip | >>>>> FORMAT_PING], [0], [dnl >>>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >>>>> +]) >>>>> +AT_CHECK([m_as $client ping -q -c 3 -i 0.3 -w 2 $vgw_encap_ip | >>>>> FORMAT_PING], [0], [dnl >>>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >>>>> +]) >>>>> + >>>>> +cr_lport_log_count() { >>>>> + m_as $1 sh -c "cat /var/log/ovn/ovn-controller.log 2>/dev/null | \ >>>>> + grep -ac 'Claiming lport cr-lr0-public'" >>>>> +} >>>>> + >>>>> +vgw1_cr_logs=$(cr_lport_log_count $vgw1) >>>>> +vgw2_cr_logs=$(cr_lport_log_count $vgw2) >>>>> + >>>>> +check multinode_nbctl ls-add sw0 >>>>> +check multinode_nbctl lsp-add sw0 sw0-port1 >>>>> +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 >>>>> 10.0.0.3" >>>>> + >>>>> +check multinode_nbctl lr-add lr0 >>>>> +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24 >>>>> +check multinode_nbctl lsp-add-router-port sw0 sw0-lr0 lr0-sw0 >>>>> + >>>>> +check multinode_nbctl ls-add public >>>>> +check multinode_nbctl lsp-add public external_client >>>>> +check multinode_nbctl lsp-set-addresses external_client >>>>> "50:55:00:00:00:03 172.20.0.99" >>>>> +check multinode_nbctl lrp-add lr0 lr0-public 00:11:22:00:ff:01 >>>>> 172.20.0.100/24 >>>>> +check multinode_nbctl lsp-add-router-port public public-lr0 lr0-public >>>>> +check multinode_nbctl lrp-set-gateway-chassis lr0-public $vgw_name 10 >>>>> + >>>>> +m_as $compute /data/create_fake_vm.sh sw0-port1 sw0p1 50:54:00:00:00:03 >>>>> 1342 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a >>>>> +m_as $client /data/create_fake_vm.sh external_client external >>>>> 50:55:00:00:00:03 1342 172.20.0.99 24 172.20.0.100 1001::3/64 1001::a >>>>> + >>>>> +check multinode_nbctl --wait=hv sync >>>>> +m_wait_for_ports_up sw0-port1 >>>>> +m_wait_for_ports_up external_client >>>>> + >>>>> +# The distributed gateway port is claimed by the replicated gateway >>>>> chassis. >>>>> +m_wait_row_count Port_Binding 1 logical_port=cr-lr0-public >>>>> +vgw_uuid=$(m_central_as ovn-sbctl --bare --columns _uuid find Chassis >>>>> name=$vgw_name) >>>>> +m_wait_column "$vgw_uuid" Port_Binding chassis logical_port=cr-lr0-public >>>>> + >>>>> +# The two controllers share a single Chassis record and a single binding >>>>> for >>>>> +# the chassisredirect port. Since both nodes are configured identically >>>>> and >>>>> +# both resolve to the same chassis, each of them finds the records >>>>> already >>>>> +# holding the values it was about to write and its transaction is a >>>>> no-op, so >>>>> +# once the setup has settled no transaction touching those tables is >>>>> expected. >>>>> +# If the nodes fought over a record, they would add one transaction per >>>>> main >>>>> +# loop iteration to the southbound database log. >>>>> +sb_txn_count() { >>>>> + m_central_as grep -ac "\"$1\":" /etc/ovn/ovnsb_db.db >>>>> +} >>>>> + >>>>> +chassis_txns=$(sb_txn_count Chassis) >>>>> +pb_txns=$(sb_txn_count Port_Binding) >>>>> +sleep 5 >>>>> +check test $(($(sb_txn_count Chassis) - chassis_txns)) -le 1 >>>>> +check test $(($(sb_txn_count Port_Binding) - pb_txns)) -le 1 >>>>> + >>>>> +cr_logs=$(($(cr_lport_log_count $vgw1) + $(cr_lport_log_count $vgw2))) >>>>> +# Claim on both >>>>> +check test $((cr_logs - vgw1_cr_logs - vgw2_cr_logs)) -le 2 >>>>> + >>>>> +check multinode_nbctl --wait=hv sync >>>>> + >>>>> +# North-south connectivity through the replicated gateway, with the >>>>> underlay ECMP >>>>> +# route in place on both ends. >>>>> +check_ns_connectivity() { >>>>> + M_NS_CHECK_EXEC([$compute], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 >>>>> 172.20.0.99 | FORMAT_PING], \ >>>>> +[0], [dnl >>>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >>>>> +]) >>>>> + >>>>> + M_NS_CHECK_EXEC([$compute], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 >>>>> 172.20.0.100 | FORMAT_PING], \ >>>>> +[0], [dnl >>>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >>>>> +]) >>>>> + >>>>> + M_NS_CHECK_EXEC([$client], [external], [ping -q -c 3 -i 0.3 -w 2 >>>>> 10.0.0.3 | FORMAT_PING], \ >>>>> +[0], [dnl >>>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >>>>> +]) >>>>> +} >>>>> + >>>>> +# The traffic towards the replicated gateway must be tunneled to the >>>>> anycast encap IP. >>>>> +M_START_TCPDUMPS([$compute], [-c 2 -neei eth1 host $vgw_encap_ip and udp >>>>> port 6081], [ch1_underlay]) >>>>> + >>>>> +check_ns_connectivity >>>>> + >>>>> +OVS_WAIT_UNTIL([test -s ch1_underlay.tcpdump]) >>>>> +AT_CHECK([grep -q "$vgw_encap_ip" ch1_underlay.tcpdump]) >>>>> + >>>>> +# Each of the two replicated gateway nodes on its own must be able to >>>>> carry the >>>>> +# traffic: pin the underlay route to a single nexthop at a time. >>>>> +for nexthop in $vgw1_ip $vgw2_ip; do >>>>> + for c in $compute $client; do >>>>> + check m_as $c ip route replace $vgw_encap_ip/32 via $nexthop dev >>>>> eth1 >>>>> + done >>>>> + >>>>> + check_ns_connectivity >>>>> +done >>>>> + >>>>> +# Back to ECMP over both gateway nodes. >>>>> +for c in $compute $client; do >>>>> + check m_as $c ip route replace $vgw_encap_ip/32 \ >>>>> + nexthop via $vgw1_ip dev eth1 \ >>>>> + nexthop via $vgw2_ip dev eth1 >>>>> +done >>>>> + >>>>> +check_ns_connectivity >>>>> + >>>>> +AT_CLEANUP _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
