On 11/9/26 12:01 PM, Dumitru Ceara wrote: > On 11/9/26 10:17 AM, Rukomoinikova Aleksandra wrote: >> On 07.11.2026 00:33, Dumitru Ceara wrote: >>> On 9/28/26 11:45 AM, Alexandra Rukomoinikova via dev wrote: >>>> Several ovn-controller instances may run with the same chassis name and >>>> the same encapsulation IP, an anycast address that is reachable over the >>>> underlay through ECMP. OVN then sees the whole group of nodes as a >>>> single chassis: a single Chassis record, a single Encap record and a >>>> single binding for every port placed on it, while the underlay decides >>>> which node of the group terminates a given tunnel flow. >>>> >>>> There is no coordination and no leader election between the nodes of the >>>> group while writing to SBDB: every ``ovn-controller`` independently does >>>> the >>>> full bookkeeping for that one chassis, and it is the idempotence of those >>>> writes that keeps them from fighting each other. >>>> >>>> The flip side is that the nodes must be configured identically in >>>> chassis-related params, and that nothing which relies on conntrack can be >>>> used, because the group shares no runtime state. >>>> >>>> Document all of this, and add a multinode test that deploys two >>>> gateway nodes as one replicated gateway, gives a hypervisor and a node that >>>> does not run OVN an underlay ECMP route to the anycast encapsulation IP, >>>> checks the north-south connectivity through each of the two nodes and >>>> checks that the controllers do not fight over the shared records. >>>> >>>> Signed-off-by: Alexandra Rukomoinikova <[email protected]> >>>> --- >>> Hi Alexandra, >>> >>> I have a few minor comments on this patch, please see below. >>> >>> I was thinking of taking care of them myself and squash a small fixup >>> patch in when pushing yours to main: >>> >>> https://github.com/dceara/ovn/commit/42b64d5 >>> >>> I didn't test this locally but CI is running in my fork: >>> >>> https://github.com/dceara/ovn/actions/runs/42295260684 >>> >>> Assuming that passes would you be OK with this incremental change? >>> >>> Regards, >>> Dumitru >> > > Hi Sasha, > >> >> Hi Dumitru! I took a fresh look at this patch, and Iād like to remove >> all references to ECMP ā anycast IP and drop the mention of using BGP >> for announcements - this seems unnecessary. Is it still okay to make > > I actually liked the ECMP reference, maybe it's an idea to still mention > that and BGP as an example? I'm not against removing it though. >
While at it, maybe it's worth also mentioning that other (stateful) features might not work as expected in such deployments. E.g., IGMP. >> those changes rn? I can take the fixup based on fixups in your fork, >> make the changes on top of it, and send you the updated patch. š >> > > I haven't merged your patch yet because CI didn't pass, one of the lanes > still hangs and I didn't have time to look into it: > > https://github.com/dceara/ovn/actions/runs/42295260684/job/129055547103 > > So if you could take the fixup from my fork and work on a v2 that'd be > great. > > Thanks, > Dumitru > >> thanks, sasha >> >> >>> >>>> Documentation/automake.mk | 1 + >>>> Documentation/topics/index.rst | 1 + >>>> Documentation/topics/replicated-gateway.rst | 171 +++++++++++++++++ >>>> NEWS | 5 + >>>> controller/ovn-controller.8.xml | 9 +- >>>> tests/multinode-macros.at | 22 ++- >>>> tests/multinode.at | 197 ++++++++++++++++++++ >>>> 7 files changed, 399 insertions(+), 7 deletions(-) >>>> create mode 100644 Documentation/topics/replicated-gateway.rst >>>> >>>> diff --git a/Documentation/automake.mk b/Documentation/automake.mk >>>> index 685ba9588..b64a0193d 100644 >>>> --- a/Documentation/automake.mk >>>> +++ b/Documentation/automake.mk >>>> @@ -25,6 +25,7 @@ DOC_SOURCE = \ >>>> Documentation/topics/testing.rst \ >>>> Documentation/topics/test-development.rst \ >>>> Documentation/topics/high-availability.rst \ >>>> + Documentation/topics/replicated-gateway.rst \ >>>> Documentation/topics/dynamic-routing/architecture.rst \ >>>> Documentation/topics/dynamic-routing/configuration-examples.rst \ >>>> Documentation/topics/dynamic-routing/index.rst \ >>>> diff --git a/Documentation/topics/index.rst >>>> b/Documentation/topics/index.rst >>>> index 0b676a98b..d5210e03a 100644 >>>> --- a/Documentation/topics/index.rst >>>> +++ b/Documentation/topics/index.rst >>>> @@ -40,6 +40,7 @@ OVN >>>> dynamic-routing/index >>>> incremental-processing/index >>>> high-availability >>>> + replicated-gateway >>>> role-based-access-control >>>> vif-plug-providers/index >>>> testing >>>> diff --git a/Documentation/topics/replicated-gateway.rst >>>> b/Documentation/topics/replicated-gateway.rst >>>> new file mode 100644 >>>> index 000000000..c3e43da03 >>>> --- /dev/null >>>> +++ b/Documentation/topics/replicated-gateway.rst >>>> @@ -0,0 +1,171 @@ >>>> +.. >>>> + Licensed under the Apache License, Version 2.0 (the "License"); you >>>> may >>>> + not use this file except in compliance with the License. You may >>>> obtain >>>> + a copy of the License at >>>> + >>>> + http://www.apache.org/licenses/LICENSE-2.0 >>>> + >>>> + Unless required by applicable law or agreed to in writing, software >>>> + distributed under the License is distributed on an "AS IS" BASIS, >>>> WITHOUT >>>> + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. >>>> See the >>>> + License for the specific language governing permissions and >>>> limitations >>>> + under the License. >>>> + >>>> + Convention for heading levels in OVN documentation: >>>> + >>>> + ======= Heading 0 (reserved for the title in a document) >>>> + ------- Heading 1 >>>> + ~~~~~~~ Heading 2 >>>> + +++++++ Heading 3 >>>> + ''''''' Heading 4 >>>> + >>>> + Avoid deeper levels because they do not render well. >>>> + >>>> +=================== >>>> +Replicated Gateway >>>> +=================== >>> Nit: one = too many. >>> >>>> + >>>> +A *replicated gateway* is a group of two or more physical nodes that >>>> present >>>> +themselves to OVN as a single chassis. Every node of the group runs its >>>> own >>>> +``ovn-controller``, but all of them are configured with the same chassis >>>> name >>>> +(``external_ids:system-id``) and with the same encapsulation IP >>>> +(``external_ids:ovn-encap-ip``). The encapsulation IP is an anycast >>>> address, >>>> +it is announced into the underlay (statically or by a routing protocol) >>>> by all >>>> +the nodes of the group, and the underlay load-balances the tunnel traffic >>>> +between them with ECMP. >>>> + >>>> +:: >>>> + >>>> + underlay with ECMP to 10.0.0.1 >>>> + >>>> + +-------------------+ +----------------------+ >>>> + | | nexthop via gw-1 | gateway node 1 | >>>> + | hypervisor +---------------------->| system-id: vgw | >>>> + | | | ovn-encap-ip: | >>>> + | ovn-encap-ip: | | 10.0.0.1 | >>>> + | 192.168.0.10 | +----------+-----------+ >>>> + | | | >>>> + | | nexthop via gw-2 +----------+-----------+ >>>> + | +---------------------->| gateway node 2 | >>>> + | | | system-id: vgw | >>>> + +-------------------+ | ovn-encap-ip: | >>>> + | 10.0.0.1 | >>>> + +----------+-----------+ >>>> + | >>>> + physical network | >>>> + -------------------------------------------------------+------------ >>>> + >>>> +From the point of view of the OVN databases there is a single chassis: a >>>> single >>>> +``Chassis`` record, a single ``Encap`` record, and a single binding for >>>> every >>>> +port that is placed on the replicated gateway (for example the >>>> chassisredirect >>>> +port of a distributed gateway port). ``ovn-northd`` and the other >>>> +``ovn-controller`` instances do not know, and do not need to know, how >>>> many >>>> +physical nodes are behind that chassis. A hypervisor keeps a single >>>> tunnel to >>>> +the anycast encapsulation IP, and the underlay decides which node of the >>>> group >>>> +terminates a given flow. >>>> + >>>> +There is no coordination and no leader election between the nodes of the >>>> +group while writing to SBDB: every ``ovn-controller`` independently does >>>> the >>>> +full bookkeeping for that one chassis, and it is the idempotence of those >>>> +writes that keeps them from fighting each other. >>>> + >>>> +Two consequences follow from this. First, the columns that OVN derives >>>> from >>>> +the local node rather than from the configuration must be forced to the >>>> same >>>> +value on every node. More detailed information on this is provided in the >>>> +``Identical OVS and OVN configuration and features`` section below. >>>> Second, >>>> +the nodes do not share any runtime state, so nothing that relies on >>>> conntrack >>>> +can be used on a replicated gateway, see the ``Only stateless traffic is >>>> +supported`` section below. >>> We shouldn't break in the middle of the verbatim section, that renders >>> weird. >>> >>>> + >>>> +Use case >>>> +-------- >>>> +This is useful when the capacity or the availability of a single gateway >>>> node >>>> +is not enough: the group scales horizontally, and losing one node only >>>> means >>>> +that the underlay stops using one of the ECMP nexthops, without any >>>> change in >>>> +the OVN databases and without the gateway having to be rescheduled to >>>> another >>>> +chassis. >>>> + >>>> +A typical deployment looks as follows: >>>> + >>>> +* Two (or more) gateway nodes are installed with the same OVN and OVS >>>> + versions and with an identical ``ovn-controller`` configuration. Each of >>>> + them configures the anycast encapsulation IP locally, e.g. on the >>>> loopback >>>> + interface, so that the kernel accepts the tunnel packets destined to >>>> it:: >>>> + >>>> + $ ovs-vsctl set open . external-ids:system-id=vgw >>>> + $ ovs-vsctl set open . external-ids:ovn-encap-ip=10.0.0.1 >>>> + $ ovs-vsctl set open . external-ids:ovn-encap-type=geneve >>>> + $ ip address add 10.0.0.1/32 dev lo >>>> + >>>> +* The anycast address is reachable in the underlay through the physical >>>> + addresses of all the nodes of the group. On the hypervisors, and on any >>>> + other node that has to reach the replicated gateway, this is an ECMP >>>> route:: >>>> + >>>> + $ ip route replace 10.0.0.1/32 \ >>>> + nexthop via 192.168.0.1 dev eth0 \ >>>> + nexthop via 192.168.0.2 dev eth0 >>>> + >>>> + In a real deployment the route is normally learned from the fabric, >>>> with the >>>> + gateway nodes advertising the anycast prefix over BGP. >>>> + >>>> +* The logical topology is configured as usual, referring to the chassis >>>> by its >>>> + single name, e.g.:: >>>> + >>>> + $ ovn-nbctl lrp-set-gateway-chassis lr0-public vgw 10 >>>> + >>>> +Requirements and restrictions >>>> +----------------------------- >>>> + >>>> +The nodes of a replicated gateway are not independent chassis, they are >>>> one >>>> +chassis that happens to run on several machines. Everything that OVN >>>> stores >>>> +per chassis is therefore shared, and everything that OVN computes per >>>> chassis >>>> +must produce the same result on every node of the group. This leads to the >>>> +following requirements. >>>> + >>>> +Identical OVS and OVN configuration and features >>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> + >>>> +All the nodes of the group must run the same OVS and OVN versions and must >>>> +expose the same set of features. ``ovn-controller`` reports the >>>> capabilities >>>> +of the local OVS instance in the ``Chassis`` record (``other_config``, for >>>> +example the supported datapath interface types, and the various >>>> +``ovn-chassis-feature`` entries), and ``ovn-northd`` generates the logical >>>> +flows for the chassis based on those values. If the nodes disagree, the >>>> +``Chassis`` record will keep flapping between the values reported by each >>>> node, >>>> +``ovn-northd`` will keep recomputing, and the node whose capabilities do >>>> not >>>> +match the ones currently published may not be able to process the flows it >>>> +receives. >>>> + >>>> +For the same reason, every chassis scoped configuration option has to be >>>> +identical on all the nodes, including, for example: >>>> ``external_ids:hostname``, >>>> +``external_ids:ovn-bridge-mappings`` and all others. >>>> + >>>> +Only stateless traffic is supported >>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> + >>>> +The nodes of a replicated gateway do not share any runtime state, and in >>>> +particular they do not share conntrack. The underlay picks one of them per >>>> +flow, and nothing guarantees that the two directions of a connection, or >>>> the >>>> +packets of a connection before and after a change of the ECMP nexthops (a >>>> node >>>> +being added, removed, rebooted or simply an ECMP rehash), are handled by >>>> the >>>> +same node. Any feature that needs the connection state to be on the node >>>> that >>>> +processes the packet is therefore not supported on a replicated gateway: >>>> + >>>> +* load balancers, >>>> +* stateful NAT, that is ``snat`` and ``dnat_and_snat`` (unless configured >>>> as >>>> + stateless), and NAT with ``ct-commit-all`` enabled, >>>> +* stateful ACLs, and anything else that relies on conntrack, for example >>>> + load balancer hairpinning or ECMP routes with >>>> + ``options:ecmp_symmetric_reply`` set. >>>> + >>>> +Only stateless forwarding, e.g. plain routing and switching, stateless >>>> NAT and >>>> +stateless ACLs, behaves correctly, because every packet can be handled >>>> +independently by any node of the group. >>>> + >>>> +Example >>>> +------- >>>> + >>>> +The ``ovn multinode replicated gateway`` test in ``tests/multinode.at`` >>>> sets up >>>> +a complete example of a two node replicated gateway with an underlay ECMP >>>> route >>>> +to the anycast encapsulation IP, a hypervisor with a VM behind the gateway >>>> +and an external node that does not run OVN at all. >>>> diff --git a/NEWS b/NEWS >>>> index f1c56dd14..7bc3f2113 100644 >>>> --- a/NEWS >>>> +++ b/NEWS >>>> @@ -129,6 +129,11 @@ OVN v26.09.0 - xxx xx xxxx >>>> represent a logical router port (e.g. ovn-ic transit switch LSPs). >>>> Such ports are treated like type=router, including omission from >>>> _MC_flood_l2. >>>> + - Documented the "replicated gateway" setup, in which several >>>> ovn-controller >>>> + instances share one chassis name and one anycast encapsulation IP >>>> + reachable over the underlay through ECMP, so that they appear to OVN >>>> as >>>> + a single chassis. See Documentation/topics/replicated-gateway.rst >>>> for the >>>> + requirements and the restrictions. >>>> >>>> OVN v26.03.0 - xxx xx xxxx >>>> -------------------------- >>>> diff --git a/controller/ovn-controller.8.xml >>>> b/controller/ovn-controller.8.xml >>>> index 3c33654ff..37c64ebf3 100644 >>>> --- a/controller/ovn-controller.8.xml >>>> +++ b/controller/ovn-controller.8.xml >>>> @@ -100,7 +100,14 @@ >>>> local OVN "etc" directory or via the <code>-n</code> command-line >>>> option. >>>> The following precedence is used: first, the command-line option >>>> is read; >>>> if not present, the <code>system-id-override</code> file is read; >>>> if not >>>> - present, then the name configured in the database is used.</dd> >>>> + present, then the name configured in the database is used. >>>> + <p> >>>> + Several <code>ovn-controller</code> instances running on different >>>> + nodes may share the same <code>system-id</code>, so that they >>>> appear >>>> + to OVN as a single chassis backed by an anycast encapsulation IP. >>>> + See the "Replicated Gateway" document for the requirements and the >>>> + restrictions of such a setup. >>>> + </p></dd> >>>> >>>> <dt><code>external_ids:hostname</code></dt> >>>> <dd>The hostname to use in the Chassis table.</dd> >>>> diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at >>>> index 1c85f79f8..81ab2a5b8 100644 >>>> --- a/tests/multinode-macros.at >>>> +++ b/tests/multinode-macros.at >>>> @@ -296,23 +296,33 @@ multinode_setup_northd() { >>>> m_as $c ovn-sbctl set-connection p${REMOTE_PROT}:6642 >>>> } >>>> >>>> -# multinode_setup_controller NODE ENCAP_IP REMOTE_IP [ENCAP_TYPE] >>>> -# >>>> -# Sets up controller on specified node. >>>> +# multinode_setup_controller NODE SYSTEM_ID ENCAP_IP REMOTE_IP >>>> +# [ENCAP_TYPE] [START_CONTROLLER] >>>> +# >>>> +# Sets up controller on specified node. SYSTEM_ID defaults to NODE; >>>> passing a >>>> +# different value (or the same value on more than one node) allows a test >>>> to >>>> +# control the chassis name the node registers with. START_CONTROLLER >>>> defaults >>>> +# to "yes"; with "no" only OVS is set up, so that a test can finish the >>>> node >>>> +# configuration before starting ovn-controller itself. >>>> multinode_setup_controller() { >>>> c=$1 >>>> + system_id=${2:-$1} >>>> encap_ip=$3 >>>> remote_ip=$4 >>>> encap_type=${5:-"geneve"} >>>> + start_controller=${6:-"yes"} >>>> >>>> # Cleanup existing one >>>> m_as $c /usr/share/openvswitch/scripts/ovs-ctl stop >>>> m_as $c /usr/share/ovn/scripts/ovn-ctl stop_controller >>>> m_as $c sh -c "rm -f /etc/openvswitch/*.db" >>>> >>>> - m_as $c /usr/share/openvswitch/scripts/ovs-ctl start --system-id=$c >>>> - echo "Using ${CONTROLLER_SSL_ARGS} for ovn-controller". >>>> - m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller >>>> ${CONTROLLER_SSL_ARGS} >>>> + m_as $c /usr/share/openvswitch/scripts/ovs-ctl start >>>> --system-id=$system_id >>>> + if [[ "$start_controller" = "yes" ]]; then >>>> + echo "Using ${CONTROLLER_SSL_ARGS} for ovn-controller". >>>> + m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller \ >>>> + ${CONTROLLER_SSL_ARGS} >>>> + fi >>> I think if we move the part that starts ovn-controller to the end of >>> this function we don't need the kind of ugly start_controller argument. >>> We can just set everything we need and then start ovn-controller. >>> >>>> >>>> m_as $c ovs-vsctl set open . external_ids:ovn-encap-ip=$encap_ip >>>> m_as $c ovs-vsctl set open . external-ids:ovn-encap-type=$encap_type >>>> diff --git a/tests/multinode.at b/tests/multinode.at >>>> index 0c277f5e8..16cfa804f 100644 >>>> --- a/tests/multinode.at >>>> +++ b/tests/multinode.at >>>> @@ -5554,3 +5554,200 @@ AT_CHECK([cat ch2_eth2.tcpdump], [0], [dnl >>>> ]) >>>> >>>> AT_CLEANUP >>>> + >>>> +AT_SETUP([ovn multinode replicated gateway]) >>>> + >>>> +check_fake_multinode_setup >>>> +cleanup_multinode_resources >>>> + >>>> +# Two gateway nodes are deployed as one and the same "replicated" >>>> gateway: both >>>> +# ovn-controllers register with the same chassis name and with the same >>>> anycast >>>> +# encap IP. The encap IP is reachable over the underlay via ECMP through >>>> the >>>> +# physical addresses of both gateway nodes, so a geneve tunnel to the >>>> replicated >>>> +# gateway may land on either of them. >>>> +# >>>> +# underlay (eth1, 170.168.0.0/16) >>>> +# āāāāāāāāāāāāāāāāāāāāā >>>> āāāāāāāāāāāāāāāāāāāā >>>> +# ā ovn-chassis-1 ā 1.1.1.1/32 ā ovn-gw-1 >>>> ā >>>> +# ā (compute node) āāāāā nexthop via gw-1 eth1 āāāāāāāā⤠>>>> ovn-replicated-gw ā >>>> +# ā ā ā encap >>>> 1.1.1.1 ā >>>> +# ā sw0p1 10.0.0.3 āāāāā nexthop via gw-2 eth1 āāāā >>>> āāāāāāāāāā¬āāāāāāāāāā >>>> +# āāāāāāāāāāāāāāāāāāāāā ā ā >>>> +# ā >>>> āāāāāāāāāā“āāāāāāāāāā >>>> +# āāāāāāāāāāāāāāāāāāāāā āāāāā⤠ovn-gw-2 >>>> ā >>>> +# ā ovn-chassis-2 ā 1.1.1.1/32 ā >>>> ovn-replicated-gw ā >>>> +# ā (external client) āāāāā nexthop via gw-1 eth1 āāāāāāāā⤠encap >>>> 1.1.1.1 ā >>>> +# ā āāāāā nexthop via gw-2 eth1 āāāāāāāā⤠>>>> ā >>>> +# ā ā >>>> āāāāāāāāāā¬āāāāāāāāāā >>>> +# ā eth2 172.20.0.50 āāāāāāāāāā provider network (eth2, br-ex) āāāāā >>>> +# āāāāāāāāāāāāāāāāāāāāā lr0-public 172.20.0.100 >>>> +# >>>> +# On ovn-chassis-2 (aka client) we have client port in public switch for >>>> checking >>>> +# internal connectivity, public switch is connected to dgp port that >>>> claimed by >>>> +# replicated chassis. Compute and client node both have underlay route >>>> for replicated gw ip (1.1.1.1) >>>> + >>>> +vgw1=ovn-gw-1 >>>> +vgw2=ovn-gw-2 >>>> +compute=ovn-chassis-1 >>>> +client=ovn-chassis-2 >>>> +vgw_name=ovn-replicated-gw >>>> +vgw_encap_ip=1.1.1.1 >>>> + >>>> +m_underlay_ip() { >>>> + m_as $1 ip -4 addr show eth1 | grep inet | awk '{print $2}' | cut >>>> -d'/' -f1 >>>> +} >>>> + >>>> +central_ip=$(m_as ovn-central-az1 ip -4 addr show eth1 | grep inet | awk >>>> '{print $2}' | cut -d'/' -f1) >>> This should now be: >>> >>> central_ip=$( (m_central_as ip -4 addr show eth1 2>/dev/null || >>> m_central_as ip -4 addr show eth2) | grep inet | awk '{print $2}' | cut >>> -d'/' -f1) >>> >>> Since we merged 44d3bf28e4b7 ("multinode: Add support for INLINE_BR_EX >>> configuration of multinode.") >>> >>>> +vgw1_ip=$(m_underlay_ip $vgw1) >>>> +vgw2_ip=$(m_underlay_ip $vgw2) >>>> +compute_ip=$(m_underlay_ip $compute) >>>> + >>>> +m_as $compute ip link del sw0p1-p >>>> +m_as $client ip link del sw0p2-p >>>> +m_as $client ip link del sw1p1-p >>>> + >>>> +# Bring up both gateway nodes as the very same chassis: same chassis name >>>> and >>>> +# same encap IP, which is not part of the underlay address space. >>>> +for c in $vgw1 $vgw2; do >>>> + # Bring the node back to its original chassis name and encap IP. >>>> + on_exit "multinode_setup_controller $c $c $(m_underlay_ip $c) >>>> $central_ip" >>>> + multinode_setup_controller $c $vgw_name $vgw_encap_ip $central_ip >>>> geneve no >>>> + on_exit "m_as $c ip addr del $vgw_encap_ip/32 dev lo" >>>> + check m_as $c ip addr add $vgw_encap_ip/32 dev lo >>>> + check m_as $c ovs-vsctl set open . external-ids:hostname=$vgw_name >>>> + check m_as $c ovs-vsctl set open . >>>> external-ids:ovn-bridge-mappings=public:br-ex >>> I don't think your test uses the ovn-bridge mapping in any way. So then >>> we might as well add hostname as an optional argument to >>> multinode_setup_controller(). >>> >>>> +done >>>> + >>>> +for c in $vgw1 $vgw2; do >>>> + m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller >>>> ${CONTROLLER_SSL_ARGS} >>>> +done >>> If we change multinode_setup_controller() as I mentioned above, we don't >>> need to manually start it here. >>> >>>> + >>>> +# Both gateway nodes must end up sharing a single chassis record with a >>>> single >>>> +# encap record. >>>> +m_wait_row_count Chassis 1 name=$vgw_name >>>> +m_wait_row_count Encap 1 chassis_name=$vgw_name >>>> +m_check_column "$vgw_encap_ip" Encap ip chassis_name=$vgw_name >>>> + >>>> +# ECMP route to the replicated gateway encap IP in the underlay, both on >>>> the >>>> +# compute node and on the external client. >>>> +for c in $compute $client; do >>>> + on_exit "m_as $c ip route del $vgw_encap_ip/32" >>>> + check m_as $c ip route replace $vgw_encap_ip/32 \ >>>> + nexthop via $vgw1_ip dev eth1 \ >>>> + nexthop via $vgw2_ip dev eth1 >>>> +done >>>> + >>>> + >>> Nit: one empty line too many. >>> >>>> +# The anycast encap IP is reachable over the underlay from both nodes. >>>> +AT_CHECK([m_as $compute ping -q -c 3 -i 0.3 -w 2 $vgw_encap_ip | >>>> FORMAT_PING], [0], [dnl >>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >>>> +]) >>>> +AT_CHECK([m_as $client ping -q -c 3 -i 0.3 -w 2 $vgw_encap_ip | >>>> FORMAT_PING], [0], [dnl >>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >>>> +]) >>>> + >>>> +cr_lport_log_count() { >>>> + m_as $1 sh -c "cat /var/log/ovn/ovn-controller.log 2>/dev/null | \ >>>> + grep -ac 'Claiming lport cr-lr0-public'" >>>> +} >>>> + >>>> +vgw1_cr_logs=$(cr_lport_log_count $vgw1) >>>> +vgw2_cr_logs=$(cr_lport_log_count $vgw2) >>>> + >>>> +check multinode_nbctl ls-add sw0 >>>> +check multinode_nbctl lsp-add sw0 sw0-port1 >>>> +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 >>>> 10.0.0.3" >>>> + >>>> +check multinode_nbctl lr-add lr0 >>>> +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24 >>>> +check multinode_nbctl lsp-add-router-port sw0 sw0-lr0 lr0-sw0 >>>> + >>>> +check multinode_nbctl ls-add public >>>> +check multinode_nbctl lsp-add public external_client >>>> +check multinode_nbctl lsp-set-addresses external_client >>>> "50:55:00:00:00:03 172.20.0.99" >>>> +check multinode_nbctl lrp-add lr0 lr0-public 00:11:22:00:ff:01 >>>> 172.20.0.100/24 >>>> +check multinode_nbctl lsp-add-router-port public public-lr0 lr0-public >>>> +check multinode_nbctl lrp-set-gateway-chassis lr0-public $vgw_name 10 >>>> + >>>> +m_as $compute /data/create_fake_vm.sh sw0-port1 sw0p1 50:54:00:00:00:03 >>>> 1342 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a >>>> +m_as $client /data/create_fake_vm.sh external_client external >>>> 50:55:00:00:00:03 1342 172.20.0.99 24 172.20.0.100 1001::3/64 1001::a >>>> + >>>> +check multinode_nbctl --wait=hv sync >>>> +m_wait_for_ports_up sw0-port1 >>>> +m_wait_for_ports_up external_client >>>> + >>>> +# The distributed gateway port is claimed by the replicated gateway >>>> chassis. >>>> +m_wait_row_count Port_Binding 1 logical_port=cr-lr0-public >>>> +vgw_uuid=$(m_central_as ovn-sbctl --bare --columns _uuid find Chassis >>>> name=$vgw_name) >>>> +m_wait_column "$vgw_uuid" Port_Binding chassis logical_port=cr-lr0-public >>>> + >>>> +# The two controllers share a single Chassis record and a single binding >>>> for >>>> +# the chassisredirect port. Since both nodes are configured identically >>>> and >>>> +# both resolve to the same chassis, each of them finds the records already >>>> +# holding the values it was about to write and its transaction is a >>>> no-op, so >>>> +# once the setup has settled no transaction touching those tables is >>>> expected. >>>> +# If the nodes fought over a record, they would add one transaction per >>>> main >>>> +# loop iteration to the southbound database log. >>>> +sb_txn_count() { >>>> + m_central_as grep -ac "\"$1\":" /etc/ovn/ovnsb_db.db >>>> +} >>>> + >>>> +chassis_txns=$(sb_txn_count Chassis) >>>> +pb_txns=$(sb_txn_count Port_Binding) >>>> +sleep 5 >>>> +check test $(($(sb_txn_count Chassis) - chassis_txns)) -le 1 >>>> +check test $(($(sb_txn_count Port_Binding) - pb_txns)) -le 1 >>>> + >>>> +cr_logs=$(($(cr_lport_log_count $vgw1) + $(cr_lport_log_count $vgw2))) >>>> +# Claim on both >>>> +check test $((cr_logs - vgw1_cr_logs - vgw2_cr_logs)) -le 2 >>>> + >>>> +check multinode_nbctl --wait=hv sync >>>> + >>>> +# North-south connectivity through the replicated gateway, with the >>>> underlay ECMP >>>> +# route in place on both ends. >>>> +check_ns_connectivity() { >>>> + M_NS_CHECK_EXEC([$compute], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 >>>> 172.20.0.99 | FORMAT_PING], \ >>>> +[0], [dnl >>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >>>> +]) >>>> + >>>> + M_NS_CHECK_EXEC([$compute], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 >>>> 172.20.0.100 | FORMAT_PING], \ >>>> +[0], [dnl >>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >>>> +]) >>>> + >>>> + M_NS_CHECK_EXEC([$client], [external], [ping -q -c 3 -i 0.3 -w 2 >>>> 10.0.0.3 | FORMAT_PING], \ >>>> +[0], [dnl >>>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >>>> +]) >>>> +} >>>> + >>>> +# The traffic towards the replicated gateway must be tunneled to the >>>> anycast encap IP. >>>> +M_START_TCPDUMPS([$compute], [-c 2 -neei eth1 host $vgw_encap_ip and udp >>>> port 6081], [ch1_underlay]) >>>> + >>>> +check_ns_connectivity >>>> + >>>> +OVS_WAIT_UNTIL([test -s ch1_underlay.tcpdump]) >>>> +AT_CHECK([grep -q "$vgw_encap_ip" ch1_underlay.tcpdump]) >>>> + >>>> +# Each of the two replicated gateway nodes on its own must be able to >>>> carry the >>>> +# traffic: pin the underlay route to a single nexthop at a time. >>>> +for nexthop in $vgw1_ip $vgw2_ip; do >>>> + for c in $compute $client; do >>>> + check m_as $c ip route replace $vgw_encap_ip/32 via $nexthop dev >>>> eth1 >>>> + done >>>> + >>>> + check_ns_connectivity >>>> +done >>>> + >>>> +# Back to ECMP over both gateway nodes. >>>> +for c in $compute $client; do >>>> + check m_as $c ip route replace $vgw_encap_ip/32 \ >>>> + nexthop via $vgw1_ip dev eth1 \ >>>> + nexthop via $vgw2_ip dev eth1 >>>> +done >>>> + >>>> +check_ns_connectivity >>>> + >>>> +AT_CLEANUP >> _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
