On 11/9/26 10:17 AM, Rukomoinikova Aleksandra wrote: > On 07.11.2026 00:33, Dumitru Ceara wrote: >> On 9/28/26 11:45 AM, Alexandra Rukomoinikova via dev wrote: >>> Several ovn-controller instances may run with the same chassis name and >>> the same encapsulation IP, an anycast address that is reachable over the >>> underlay through ECMP. OVN then sees the whole group of nodes as a >>> single chassis: a single Chassis record, a single Encap record and a >>> single binding for every port placed on it, while the underlay decides >>> which node of the group terminates a given tunnel flow. >>> >>> There is no coordination and no leader election between the nodes of the >>> group while writing to SBDB: every ``ovn-controller`` independently does the >>> full bookkeeping for that one chassis, and it is the idempotence of those >>> writes that keeps them from fighting each other. >>> >>> The flip side is that the nodes must be configured identically in >>> chassis-related params, and that nothing which relies on conntrack can be >>> used, because the group shares no runtime state. >>> >>> Document all of this, and add a multinode test that deploys two >>> gateway nodes as one replicated gateway, gives a hypervisor and a node that >>> does not run OVN an underlay ECMP route to the anycast encapsulation IP, >>> checks the north-south connectivity through each of the two nodes and >>> checks that the controllers do not fight over the shared records. >>> >>> Signed-off-by: Alexandra Rukomoinikova <[email protected]> >>> --- >> Hi Alexandra, >> >> I have a few minor comments on this patch, please see below. >> >> I was thinking of taking care of them myself and squash a small fixup >> patch in when pushing yours to main: >> >> https://github.com/dceara/ovn/commit/42b64d5 >> >> I didn't test this locally but CI is running in my fork: >> >> https://github.com/dceara/ovn/actions/runs/42295260684 >> >> Assuming that passes would you be OK with this incremental change? >> >> Regards, >> Dumitru >
Hi Sasha, > > Hi Dumitru! I took a fresh look at this patch, and Iād like to remove > all references to ECMP ā anycast IP and drop the mention of using BGP > for announcements - this seems unnecessary. Is it still okay to make I actually liked the ECMP reference, maybe it's an idea to still mention that and BGP as an example? I'm not against removing it though. > those changes rn? I can take the fixup based on fixups in your fork, > make the changes on top of it, and send you the updated patch. š > I haven't merged your patch yet because CI didn't pass, one of the lanes still hangs and I didn't have time to look into it: https://github.com/dceara/ovn/actions/runs/42295260684/job/129055547103 So if you could take the fixup from my fork and work on a v2 that'd be great. Thanks, Dumitru > thanks, sasha > > >> >>> Documentation/automake.mk | 1 + >>> Documentation/topics/index.rst | 1 + >>> Documentation/topics/replicated-gateway.rst | 171 +++++++++++++++++ >>> NEWS | 5 + >>> controller/ovn-controller.8.xml | 9 +- >>> tests/multinode-macros.at | 22 ++- >>> tests/multinode.at | 197 ++++++++++++++++++++ >>> 7 files changed, 399 insertions(+), 7 deletions(-) >>> create mode 100644 Documentation/topics/replicated-gateway.rst >>> >>> diff --git a/Documentation/automake.mk b/Documentation/automake.mk >>> index 685ba9588..b64a0193d 100644 >>> --- a/Documentation/automake.mk >>> +++ b/Documentation/automake.mk >>> @@ -25,6 +25,7 @@ DOC_SOURCE = \ >>> Documentation/topics/testing.rst \ >>> Documentation/topics/test-development.rst \ >>> Documentation/topics/high-availability.rst \ >>> + Documentation/topics/replicated-gateway.rst \ >>> Documentation/topics/dynamic-routing/architecture.rst \ >>> Documentation/topics/dynamic-routing/configuration-examples.rst \ >>> Documentation/topics/dynamic-routing/index.rst \ >>> diff --git a/Documentation/topics/index.rst b/Documentation/topics/index.rst >>> index 0b676a98b..d5210e03a 100644 >>> --- a/Documentation/topics/index.rst >>> +++ b/Documentation/topics/index.rst >>> @@ -40,6 +40,7 @@ OVN >>> dynamic-routing/index >>> incremental-processing/index >>> high-availability >>> + replicated-gateway >>> role-based-access-control >>> vif-plug-providers/index >>> testing >>> diff --git a/Documentation/topics/replicated-gateway.rst >>> b/Documentation/topics/replicated-gateway.rst >>> new file mode 100644 >>> index 000000000..c3e43da03 >>> --- /dev/null >>> +++ b/Documentation/topics/replicated-gateway.rst >>> @@ -0,0 +1,171 @@ >>> +.. >>> + Licensed under the Apache License, Version 2.0 (the "License"); you >>> may >>> + not use this file except in compliance with the License. You may >>> obtain >>> + a copy of the License at >>> + >>> + http://www.apache.org/licenses/LICENSE-2.0 >>> + >>> + Unless required by applicable law or agreed to in writing, software >>> + distributed under the License is distributed on an "AS IS" BASIS, >>> WITHOUT >>> + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See >>> the >>> + License for the specific language governing permissions and >>> limitations >>> + under the License. >>> + >>> + Convention for heading levels in OVN documentation: >>> + >>> + ======= Heading 0 (reserved for the title in a document) >>> + ------- Heading 1 >>> + ~~~~~~~ Heading 2 >>> + +++++++ Heading 3 >>> + ''''''' Heading 4 >>> + >>> + Avoid deeper levels because they do not render well. >>> + >>> +=================== >>> +Replicated Gateway >>> +=================== >> Nit: one = too many. >> >>> + >>> +A *replicated gateway* is a group of two or more physical nodes that >>> present >>> +themselves to OVN as a single chassis. Every node of the group runs its own >>> +``ovn-controller``, but all of them are configured with the same chassis >>> name >>> +(``external_ids:system-id``) and with the same encapsulation IP >>> +(``external_ids:ovn-encap-ip``). The encapsulation IP is an anycast >>> address, >>> +it is announced into the underlay (statically or by a routing protocol) by >>> all >>> +the nodes of the group, and the underlay load-balances the tunnel traffic >>> +between them with ECMP. >>> + >>> +:: >>> + >>> + underlay with ECMP to 10.0.0.1 >>> + >>> + +-------------------+ +----------------------+ >>> + | | nexthop via gw-1 | gateway node 1 | >>> + | hypervisor +---------------------->| system-id: vgw | >>> + | | | ovn-encap-ip: | >>> + | ovn-encap-ip: | | 10.0.0.1 | >>> + | 192.168.0.10 | +----------+-----------+ >>> + | | | >>> + | | nexthop via gw-2 +----------+-----------+ >>> + | +---------------------->| gateway node 2 | >>> + | | | system-id: vgw | >>> + +-------------------+ | ovn-encap-ip: | >>> + | 10.0.0.1 | >>> + +----------+-----------+ >>> + | >>> + physical network | >>> + -------------------------------------------------------+------------ >>> + >>> +From the point of view of the OVN databases there is a single chassis: a >>> single >>> +``Chassis`` record, a single ``Encap`` record, and a single binding for >>> every >>> +port that is placed on the replicated gateway (for example the >>> chassisredirect >>> +port of a distributed gateway port). ``ovn-northd`` and the other >>> +``ovn-controller`` instances do not know, and do not need to know, how many >>> +physical nodes are behind that chassis. A hypervisor keeps a single tunnel >>> to >>> +the anycast encapsulation IP, and the underlay decides which node of the >>> group >>> +terminates a given flow. >>> + >>> +There is no coordination and no leader election between the nodes of the >>> +group while writing to SBDB: every ``ovn-controller`` independently does >>> the >>> +full bookkeeping for that one chassis, and it is the idempotence of those >>> +writes that keeps them from fighting each other. >>> + >>> +Two consequences follow from this. First, the columns that OVN derives >>> from >>> +the local node rather than from the configuration must be forced to the >>> same >>> +value on every node. More detailed information on this is provided in the >>> +``Identical OVS and OVN configuration and features`` section below. >>> Second, >>> +the nodes do not share any runtime state, so nothing that relies on >>> conntrack >>> +can be used on a replicated gateway, see the ``Only stateless traffic is >>> +supported`` section below. >> We shouldn't break in the middle of the verbatim section, that renders >> weird. >> >>> + >>> +Use case >>> +-------- >>> +This is useful when the capacity or the availability of a single gateway >>> node >>> +is not enough: the group scales horizontally, and losing one node only >>> means >>> +that the underlay stops using one of the ECMP nexthops, without any change >>> in >>> +the OVN databases and without the gateway having to be rescheduled to >>> another >>> +chassis. >>> + >>> +A typical deployment looks as follows: >>> + >>> +* Two (or more) gateway nodes are installed with the same OVN and OVS >>> + versions and with an identical ``ovn-controller`` configuration. Each of >>> + them configures the anycast encapsulation IP locally, e.g. on the >>> loopback >>> + interface, so that the kernel accepts the tunnel packets destined to it:: >>> + >>> + $ ovs-vsctl set open . external-ids:system-id=vgw >>> + $ ovs-vsctl set open . external-ids:ovn-encap-ip=10.0.0.1 >>> + $ ovs-vsctl set open . external-ids:ovn-encap-type=geneve >>> + $ ip address add 10.0.0.1/32 dev lo >>> + >>> +* The anycast address is reachable in the underlay through the physical >>> + addresses of all the nodes of the group. On the hypervisors, and on any >>> + other node that has to reach the replicated gateway, this is an ECMP >>> route:: >>> + >>> + $ ip route replace 10.0.0.1/32 \ >>> + nexthop via 192.168.0.1 dev eth0 \ >>> + nexthop via 192.168.0.2 dev eth0 >>> + >>> + In a real deployment the route is normally learned from the fabric, with >>> the >>> + gateway nodes advertising the anycast prefix over BGP. >>> + >>> +* The logical topology is configured as usual, referring to the chassis by >>> its >>> + single name, e.g.:: >>> + >>> + $ ovn-nbctl lrp-set-gateway-chassis lr0-public vgw 10 >>> + >>> +Requirements and restrictions >>> +----------------------------- >>> + >>> +The nodes of a replicated gateway are not independent chassis, they are one >>> +chassis that happens to run on several machines. Everything that OVN stores >>> +per chassis is therefore shared, and everything that OVN computes per >>> chassis >>> +must produce the same result on every node of the group. This leads to the >>> +following requirements. >>> + >>> +Identical OVS and OVN configuration and features >>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> + >>> +All the nodes of the group must run the same OVS and OVN versions and must >>> +expose the same set of features. ``ovn-controller`` reports the >>> capabilities >>> +of the local OVS instance in the ``Chassis`` record (``other_config``, for >>> +example the supported datapath interface types, and the various >>> +``ovn-chassis-feature`` entries), and ``ovn-northd`` generates the logical >>> +flows for the chassis based on those values. If the nodes disagree, the >>> +``Chassis`` record will keep flapping between the values reported by each >>> node, >>> +``ovn-northd`` will keep recomputing, and the node whose capabilities do >>> not >>> +match the ones currently published may not be able to process the flows it >>> +receives. >>> + >>> +For the same reason, every chassis scoped configuration option has to be >>> +identical on all the nodes, including, for example: >>> ``external_ids:hostname``, >>> +``external_ids:ovn-bridge-mappings`` and all others. >>> + >>> +Only stateless traffic is supported >>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> + >>> +The nodes of a replicated gateway do not share any runtime state, and in >>> +particular they do not share conntrack. The underlay picks one of them per >>> +flow, and nothing guarantees that the two directions of a connection, or >>> the >>> +packets of a connection before and after a change of the ECMP nexthops (a >>> node >>> +being added, removed, rebooted or simply an ECMP rehash), are handled by >>> the >>> +same node. Any feature that needs the connection state to be on the node >>> that >>> +processes the packet is therefore not supported on a replicated gateway: >>> + >>> +* load balancers, >>> +* stateful NAT, that is ``snat`` and ``dnat_and_snat`` (unless configured >>> as >>> + stateless), and NAT with ``ct-commit-all`` enabled, >>> +* stateful ACLs, and anything else that relies on conntrack, for example >>> + load balancer hairpinning or ECMP routes with >>> + ``options:ecmp_symmetric_reply`` set. >>> + >>> +Only stateless forwarding, e.g. plain routing and switching, stateless NAT >>> and >>> +stateless ACLs, behaves correctly, because every packet can be handled >>> +independently by any node of the group. >>> + >>> +Example >>> +------- >>> + >>> +The ``ovn multinode replicated gateway`` test in ``tests/multinode.at`` >>> sets up >>> +a complete example of a two node replicated gateway with an underlay ECMP >>> route >>> +to the anycast encapsulation IP, a hypervisor with a VM behind the gateway >>> +and an external node that does not run OVN at all. >>> diff --git a/NEWS b/NEWS >>> index f1c56dd14..7bc3f2113 100644 >>> --- a/NEWS >>> +++ b/NEWS >>> @@ -129,6 +129,11 @@ OVN v26.09.0 - xxx xx xxxx >>> represent a logical router port (e.g. ovn-ic transit switch LSPs). >>> Such ports are treated like type=router, including omission from >>> _MC_flood_l2. >>> + - Documented the "replicated gateway" setup, in which several >>> ovn-controller >>> + instances share one chassis name and one anycast encapsulation IP >>> + reachable over the underlay through ECMP, so that they appear to OVN >>> as >>> + a single chassis. See Documentation/topics/replicated-gateway.rst for >>> the >>> + requirements and the restrictions. >>> >>> OVN v26.03.0 - xxx xx xxxx >>> -------------------------- >>> diff --git a/controller/ovn-controller.8.xml >>> b/controller/ovn-controller.8.xml >>> index 3c33654ff..37c64ebf3 100644 >>> --- a/controller/ovn-controller.8.xml >>> +++ b/controller/ovn-controller.8.xml >>> @@ -100,7 +100,14 @@ >>> local OVN "etc" directory or via the <code>-n</code> command-line >>> option. >>> The following precedence is used: first, the command-line option is >>> read; >>> if not present, the <code>system-id-override</code> file is read; >>> if not >>> - present, then the name configured in the database is used.</dd> >>> + present, then the name configured in the database is used. >>> + <p> >>> + Several <code>ovn-controller</code> instances running on different >>> + nodes may share the same <code>system-id</code>, so that they >>> appear >>> + to OVN as a single chassis backed by an anycast encapsulation IP. >>> + See the "Replicated Gateway" document for the requirements and the >>> + restrictions of such a setup. >>> + </p></dd> >>> >>> <dt><code>external_ids:hostname</code></dt> >>> <dd>The hostname to use in the Chassis table.</dd> >>> diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at >>> index 1c85f79f8..81ab2a5b8 100644 >>> --- a/tests/multinode-macros.at >>> +++ b/tests/multinode-macros.at >>> @@ -296,23 +296,33 @@ multinode_setup_northd() { >>> m_as $c ovn-sbctl set-connection p${REMOTE_PROT}:6642 >>> } >>> >>> -# multinode_setup_controller NODE ENCAP_IP REMOTE_IP [ENCAP_TYPE] >>> -# >>> -# Sets up controller on specified node. >>> +# multinode_setup_controller NODE SYSTEM_ID ENCAP_IP REMOTE_IP >>> +# [ENCAP_TYPE] [START_CONTROLLER] >>> +# >>> +# Sets up controller on specified node. SYSTEM_ID defaults to NODE; >>> passing a >>> +# different value (or the same value on more than one node) allows a test >>> to >>> +# control the chassis name the node registers with. START_CONTROLLER >>> defaults >>> +# to "yes"; with "no" only OVS is set up, so that a test can finish the >>> node >>> +# configuration before starting ovn-controller itself. >>> multinode_setup_controller() { >>> c=$1 >>> + system_id=${2:-$1} >>> encap_ip=$3 >>> remote_ip=$4 >>> encap_type=${5:-"geneve"} >>> + start_controller=${6:-"yes"} >>> >>> # Cleanup existing one >>> m_as $c /usr/share/openvswitch/scripts/ovs-ctl stop >>> m_as $c /usr/share/ovn/scripts/ovn-ctl stop_controller >>> m_as $c sh -c "rm -f /etc/openvswitch/*.db" >>> >>> - m_as $c /usr/share/openvswitch/scripts/ovs-ctl start --system-id=$c >>> - echo "Using ${CONTROLLER_SSL_ARGS} for ovn-controller". >>> - m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller >>> ${CONTROLLER_SSL_ARGS} >>> + m_as $c /usr/share/openvswitch/scripts/ovs-ctl start >>> --system-id=$system_id >>> + if [[ "$start_controller" = "yes" ]]; then >>> + echo "Using ${CONTROLLER_SSL_ARGS} for ovn-controller". >>> + m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller \ >>> + ${CONTROLLER_SSL_ARGS} >>> + fi >> I think if we move the part that starts ovn-controller to the end of >> this function we don't need the kind of ugly start_controller argument. >> We can just set everything we need and then start ovn-controller. >> >>> >>> m_as $c ovs-vsctl set open . external_ids:ovn-encap-ip=$encap_ip >>> m_as $c ovs-vsctl set open . external-ids:ovn-encap-type=$encap_type >>> diff --git a/tests/multinode.at b/tests/multinode.at >>> index 0c277f5e8..16cfa804f 100644 >>> --- a/tests/multinode.at >>> +++ b/tests/multinode.at >>> @@ -5554,3 +5554,200 @@ AT_CHECK([cat ch2_eth2.tcpdump], [0], [dnl >>> ]) >>> >>> AT_CLEANUP >>> + >>> +AT_SETUP([ovn multinode replicated gateway]) >>> + >>> +check_fake_multinode_setup >>> +cleanup_multinode_resources >>> + >>> +# Two gateway nodes are deployed as one and the same "replicated" gateway: >>> both >>> +# ovn-controllers register with the same chassis name and with the same >>> anycast >>> +# encap IP. The encap IP is reachable over the underlay via ECMP through >>> the >>> +# physical addresses of both gateway nodes, so a geneve tunnel to the >>> replicated >>> +# gateway may land on either of them. >>> +# >>> +# underlay (eth1, 170.168.0.0/16) >>> +# āāāāāāāāāāāāāāāāāāāāā >>> āāāāāāāāāāāāāāāāāāāā >>> +# ā ovn-chassis-1 ā 1.1.1.1/32 ā ovn-gw-1 >>> ā >>> +# ā (compute node) āāāāā nexthop via gw-1 eth1 āāāāāāāā⤠>>> ovn-replicated-gw ā >>> +# ā ā ā encap >>> 1.1.1.1 ā >>> +# ā sw0p1 10.0.0.3 āāāāā nexthop via gw-2 eth1 āāāā >>> āāāāāāāāāā¬āāāāāāāāāā >>> +# āāāāāāāāāāāāāāāāāāāāā ā ā >>> +# ā >>> āāāāāāāāāā“āāāāāāāāāā >>> +# āāāāāāāāāāāāāāāāāāāāā āāāāā⤠ovn-gw-2 >>> ā >>> +# ā ovn-chassis-2 ā 1.1.1.1/32 ā >>> ovn-replicated-gw ā >>> +# ā (external client) āāāāā nexthop via gw-1 eth1 āāāāāāāā⤠encap >>> 1.1.1.1 ā >>> +# ā āāāāā nexthop via gw-2 eth1 āāāāāāāā⤠>>> ā >>> +# ā ā >>> āāāāāāāāāā¬āāāāāāāāāā >>> +# ā eth2 172.20.0.50 āāāāāāāāāā provider network (eth2, br-ex) āāāāā >>> +# āāāāāāāāāāāāāāāāāāāāā lr0-public 172.20.0.100 >>> +# >>> +# On ovn-chassis-2 (aka client) we have client port in public switch for >>> checking >>> +# internal connectivity, public switch is connected to dgp port that >>> claimed by >>> +# replicated chassis. Compute and client node both have underlay route for >>> replicated gw ip (1.1.1.1) >>> + >>> +vgw1=ovn-gw-1 >>> +vgw2=ovn-gw-2 >>> +compute=ovn-chassis-1 >>> +client=ovn-chassis-2 >>> +vgw_name=ovn-replicated-gw >>> +vgw_encap_ip=1.1.1.1 >>> + >>> +m_underlay_ip() { >>> + m_as $1 ip -4 addr show eth1 | grep inet | awk '{print $2}' | cut >>> -d'/' -f1 >>> +} >>> + >>> +central_ip=$(m_as ovn-central-az1 ip -4 addr show eth1 | grep inet | awk >>> '{print $2}' | cut -d'/' -f1) >> This should now be: >> >> central_ip=$( (m_central_as ip -4 addr show eth1 2>/dev/null || >> m_central_as ip -4 addr show eth2) | grep inet | awk '{print $2}' | cut >> -d'/' -f1) >> >> Since we merged 44d3bf28e4b7 ("multinode: Add support for INLINE_BR_EX >> configuration of multinode.") >> >>> +vgw1_ip=$(m_underlay_ip $vgw1) >>> +vgw2_ip=$(m_underlay_ip $vgw2) >>> +compute_ip=$(m_underlay_ip $compute) >>> + >>> +m_as $compute ip link del sw0p1-p >>> +m_as $client ip link del sw0p2-p >>> +m_as $client ip link del sw1p1-p >>> + >>> +# Bring up both gateway nodes as the very same chassis: same chassis name >>> and >>> +# same encap IP, which is not part of the underlay address space. >>> +for c in $vgw1 $vgw2; do >>> + # Bring the node back to its original chassis name and encap IP. >>> + on_exit "multinode_setup_controller $c $c $(m_underlay_ip $c) >>> $central_ip" >>> + multinode_setup_controller $c $vgw_name $vgw_encap_ip $central_ip >>> geneve no >>> + on_exit "m_as $c ip addr del $vgw_encap_ip/32 dev lo" >>> + check m_as $c ip addr add $vgw_encap_ip/32 dev lo >>> + check m_as $c ovs-vsctl set open . external-ids:hostname=$vgw_name >>> + check m_as $c ovs-vsctl set open . >>> external-ids:ovn-bridge-mappings=public:br-ex >> I don't think your test uses the ovn-bridge mapping in any way. So then >> we might as well add hostname as an optional argument to >> multinode_setup_controller(). >> >>> +done >>> + >>> +for c in $vgw1 $vgw2; do >>> + m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller >>> ${CONTROLLER_SSL_ARGS} >>> +done >> If we change multinode_setup_controller() as I mentioned above, we don't >> need to manually start it here. >> >>> + >>> +# Both gateway nodes must end up sharing a single chassis record with a >>> single >>> +# encap record. >>> +m_wait_row_count Chassis 1 name=$vgw_name >>> +m_wait_row_count Encap 1 chassis_name=$vgw_name >>> +m_check_column "$vgw_encap_ip" Encap ip chassis_name=$vgw_name >>> + >>> +# ECMP route to the replicated gateway encap IP in the underlay, both on >>> the >>> +# compute node and on the external client. >>> +for c in $compute $client; do >>> + on_exit "m_as $c ip route del $vgw_encap_ip/32" >>> + check m_as $c ip route replace $vgw_encap_ip/32 \ >>> + nexthop via $vgw1_ip dev eth1 \ >>> + nexthop via $vgw2_ip dev eth1 >>> +done >>> + >>> + >> Nit: one empty line too many. >> >>> +# The anycast encap IP is reachable over the underlay from both nodes. >>> +AT_CHECK([m_as $compute ping -q -c 3 -i 0.3 -w 2 $vgw_encap_ip | >>> FORMAT_PING], [0], [dnl >>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >>> +]) >>> +AT_CHECK([m_as $client ping -q -c 3 -i 0.3 -w 2 $vgw_encap_ip | >>> FORMAT_PING], [0], [dnl >>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >>> +]) >>> + >>> +cr_lport_log_count() { >>> + m_as $1 sh -c "cat /var/log/ovn/ovn-controller.log 2>/dev/null | \ >>> + grep -ac 'Claiming lport cr-lr0-public'" >>> +} >>> + >>> +vgw1_cr_logs=$(cr_lport_log_count $vgw1) >>> +vgw2_cr_logs=$(cr_lport_log_count $vgw2) >>> + >>> +check multinode_nbctl ls-add sw0 >>> +check multinode_nbctl lsp-add sw0 sw0-port1 >>> +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 >>> 10.0.0.3" >>> + >>> +check multinode_nbctl lr-add lr0 >>> +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24 >>> +check multinode_nbctl lsp-add-router-port sw0 sw0-lr0 lr0-sw0 >>> + >>> +check multinode_nbctl ls-add public >>> +check multinode_nbctl lsp-add public external_client >>> +check multinode_nbctl lsp-set-addresses external_client "50:55:00:00:00:03 >>> 172.20.0.99" >>> +check multinode_nbctl lrp-add lr0 lr0-public 00:11:22:00:ff:01 >>> 172.20.0.100/24 >>> +check multinode_nbctl lsp-add-router-port public public-lr0 lr0-public >>> +check multinode_nbctl lrp-set-gateway-chassis lr0-public $vgw_name 10 >>> + >>> +m_as $compute /data/create_fake_vm.sh sw0-port1 sw0p1 50:54:00:00:00:03 >>> 1342 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a >>> +m_as $client /data/create_fake_vm.sh external_client external >>> 50:55:00:00:00:03 1342 172.20.0.99 24 172.20.0.100 1001::3/64 1001::a >>> + >>> +check multinode_nbctl --wait=hv sync >>> +m_wait_for_ports_up sw0-port1 >>> +m_wait_for_ports_up external_client >>> + >>> +# The distributed gateway port is claimed by the replicated gateway >>> chassis. >>> +m_wait_row_count Port_Binding 1 logical_port=cr-lr0-public >>> +vgw_uuid=$(m_central_as ovn-sbctl --bare --columns _uuid find Chassis >>> name=$vgw_name) >>> +m_wait_column "$vgw_uuid" Port_Binding chassis logical_port=cr-lr0-public >>> + >>> +# The two controllers share a single Chassis record and a single binding >>> for >>> +# the chassisredirect port. Since both nodes are configured identically >>> and >>> +# both resolve to the same chassis, each of them finds the records already >>> +# holding the values it was about to write and its transaction is a no-op, >>> so >>> +# once the setup has settled no transaction touching those tables is >>> expected. >>> +# If the nodes fought over a record, they would add one transaction per >>> main >>> +# loop iteration to the southbound database log. >>> +sb_txn_count() { >>> + m_central_as grep -ac "\"$1\":" /etc/ovn/ovnsb_db.db >>> +} >>> + >>> +chassis_txns=$(sb_txn_count Chassis) >>> +pb_txns=$(sb_txn_count Port_Binding) >>> +sleep 5 >>> +check test $(($(sb_txn_count Chassis) - chassis_txns)) -le 1 >>> +check test $(($(sb_txn_count Port_Binding) - pb_txns)) -le 1 >>> + >>> +cr_logs=$(($(cr_lport_log_count $vgw1) + $(cr_lport_log_count $vgw2))) >>> +# Claim on both >>> +check test $((cr_logs - vgw1_cr_logs - vgw2_cr_logs)) -le 2 >>> + >>> +check multinode_nbctl --wait=hv sync >>> + >>> +# North-south connectivity through the replicated gateway, with the >>> underlay ECMP >>> +# route in place on both ends. >>> +check_ns_connectivity() { >>> + M_NS_CHECK_EXEC([$compute], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 >>> 172.20.0.99 | FORMAT_PING], \ >>> +[0], [dnl >>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >>> +]) >>> + >>> + M_NS_CHECK_EXEC([$compute], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 >>> 172.20.0.100 | FORMAT_PING], \ >>> +[0], [dnl >>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >>> +]) >>> + >>> + M_NS_CHECK_EXEC([$client], [external], [ping -q -c 3 -i 0.3 -w 2 >>> 10.0.0.3 | FORMAT_PING], \ >>> +[0], [dnl >>> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >>> +]) >>> +} >>> + >>> +# The traffic towards the replicated gateway must be tunneled to the >>> anycast encap IP. >>> +M_START_TCPDUMPS([$compute], [-c 2 -neei eth1 host $vgw_encap_ip and udp >>> port 6081], [ch1_underlay]) >>> + >>> +check_ns_connectivity >>> + >>> +OVS_WAIT_UNTIL([test -s ch1_underlay.tcpdump]) >>> +AT_CHECK([grep -q "$vgw_encap_ip" ch1_underlay.tcpdump]) >>> + >>> +# Each of the two replicated gateway nodes on its own must be able to >>> carry the >>> +# traffic: pin the underlay route to a single nexthop at a time. >>> +for nexthop in $vgw1_ip $vgw2_ip; do >>> + for c in $compute $client; do >>> + check m_as $c ip route replace $vgw_encap_ip/32 via $nexthop dev >>> eth1 >>> + done >>> + >>> + check_ns_connectivity >>> +done >>> + >>> +# Back to ECMP over both gateway nodes. >>> +for c in $compute $client; do >>> + check m_as $c ip route replace $vgw_encap_ip/32 \ >>> + nexthop via $vgw1_ip dev eth1 \ >>> + nexthop via $vgw2_ip dev eth1 >>> +done >>> + >>> +check_ns_connectivity >>> + >>> +AT_CLEANUP > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
