On 07.11.2026 00:33, Dumitru Ceara wrote: > On 9/28/26 11:45 AM, Alexandra Rukomoinikova via dev wrote: >> Several ovn-controller instances may run with the same chassis name and >> the same encapsulation IP, an anycast address that is reachable over the >> underlay through ECMP. OVN then sees the whole group of nodes as a >> single chassis: a single Chassis record, a single Encap record and a >> single binding for every port placed on it, while the underlay decides >> which node of the group terminates a given tunnel flow. >> >> There is no coordination and no leader election between the nodes of the >> group while writing to SBDB: every ``ovn-controller`` independently does the >> full bookkeeping for that one chassis, and it is the idempotence of those >> writes that keeps them from fighting each other. >> >> The flip side is that the nodes must be configured identically in >> chassis-related params, and that nothing which relies on conntrack can be >> used, because the group shares no runtime state. >> >> Document all of this, and add a multinode test that deploys two >> gateway nodes as one replicated gateway, gives a hypervisor and a node that >> does not run OVN an underlay ECMP route to the anycast encapsulation IP, >> checks the north-south connectivity through each of the two nodes and >> checks that the controllers do not fight over the shared records. >> >> Signed-off-by: Alexandra Rukomoinikova <[email protected]> >> --- > Hi Alexandra, > > I have a few minor comments on this patch, please see below. > > I was thinking of taking care of them myself and squash a small fixup > patch in when pushing yours to main: > > https://github.com/dceara/ovn/commit/42b64d5 > > I didn't test this locally but CI is running in my fork: > > https://github.com/dceara/ovn/actions/runs/42295260684 > > Assuming that passes would you be OK with this incremental change?
Hi Dumitru, Yes, I like these changes, thanks a lot for review and fixing that. Thanks, Sasha. > > Regards, > Dumitru > >> Documentation/automake.mk | 1 + >> Documentation/topics/index.rst | 1 + >> Documentation/topics/replicated-gateway.rst | 171 +++++++++++++++++ >> NEWS | 5 + >> controller/ovn-controller.8.xml | 9 +- >> tests/multinode-macros.at | 22 ++- >> tests/multinode.at | 197 ++++++++++++++++++++ >> 7 files changed, 399 insertions(+), 7 deletions(-) >> create mode 100644 Documentation/topics/replicated-gateway.rst >> >> diff --git a/Documentation/automake.mk b/Documentation/automake.mk >> index 685ba9588..b64a0193d 100644 >> --- a/Documentation/automake.mk >> +++ b/Documentation/automake.mk >> @@ -25,6 +25,7 @@ DOC_SOURCE = \ >> Documentation/topics/testing.rst \ >> Documentation/topics/test-development.rst \ >> Documentation/topics/high-availability.rst \ >> + Documentation/topics/replicated-gateway.rst \ >> Documentation/topics/dynamic-routing/architecture.rst \ >> Documentation/topics/dynamic-routing/configuration-examples.rst \ >> Documentation/topics/dynamic-routing/index.rst \ >> diff --git a/Documentation/topics/index.rst b/Documentation/topics/index.rst >> index 0b676a98b..d5210e03a 100644 >> --- a/Documentation/topics/index.rst >> +++ b/Documentation/topics/index.rst >> @@ -40,6 +40,7 @@ OVN >> dynamic-routing/index >> incremental-processing/index >> high-availability >> + replicated-gateway >> role-based-access-control >> vif-plug-providers/index >> testing >> diff --git a/Documentation/topics/replicated-gateway.rst >> b/Documentation/topics/replicated-gateway.rst >> new file mode 100644 >> index 000000000..c3e43da03 >> --- /dev/null >> +++ b/Documentation/topics/replicated-gateway.rst >> @@ -0,0 +1,171 @@ >> +.. >> + Licensed under the Apache License, Version 2.0 (the "License"); you >> may >> + not use this file except in compliance with the License. You may >> obtain >> + a copy of the License at >> + >> + http://www.apache.org/licenses/LICENSE-2.0 >> + >> + Unless required by applicable law or agreed to in writing, software >> + distributed under the License is distributed on an "AS IS" BASIS, >> WITHOUT >> + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See >> the >> + License for the specific language governing permissions and >> limitations >> + under the License. >> + >> + Convention for heading levels in OVN documentation: >> + >> + ======= Heading 0 (reserved for the title in a document) >> + ------- Heading 1 >> + ~~~~~~~ Heading 2 >> + +++++++ Heading 3 >> + ''''''' Heading 4 >> + >> + Avoid deeper levels because they do not render well. >> + >> +=================== >> +Replicated Gateway >> +=================== > Nit: one = too many. > >> + >> +A *replicated gateway* is a group of two or more physical nodes that present >> +themselves to OVN as a single chassis. Every node of the group runs its own >> +``ovn-controller``, but all of them are configured with the same chassis >> name >> +(``external_ids:system-id``) and with the same encapsulation IP >> +(``external_ids:ovn-encap-ip``). The encapsulation IP is an anycast address, >> +it is announced into the underlay (statically or by a routing protocol) by >> all >> +the nodes of the group, and the underlay load-balances the tunnel traffic >> +between them with ECMP. >> + >> +:: >> + >> + underlay with ECMP to 10.0.0.1 >> + >> + +-------------------+ +----------------------+ >> + | | nexthop via gw-1 | gateway node 1 | >> + | hypervisor +---------------------->| system-id: vgw | >> + | | | ovn-encap-ip: | >> + | ovn-encap-ip: | | 10.0.0.1 | >> + | 192.168.0.10 | +----------+-----------+ >> + | | | >> + | | nexthop via gw-2 +----------+-----------+ >> + | +---------------------->| gateway node 2 | >> + | | | system-id: vgw | >> + +-------------------+ | ovn-encap-ip: | >> + | 10.0.0.1 | >> + +----------+-----------+ >> + | >> + physical network | >> + -------------------------------------------------------+------------ >> + >> +From the point of view of the OVN databases there is a single chassis: a >> single >> +``Chassis`` record, a single ``Encap`` record, and a single binding for >> every >> +port that is placed on the replicated gateway (for example the >> chassisredirect >> +port of a distributed gateway port). ``ovn-northd`` and the other >> +``ovn-controller`` instances do not know, and do not need to know, how many >> +physical nodes are behind that chassis. A hypervisor keeps a single tunnel >> to >> +the anycast encapsulation IP, and the underlay decides which node of the >> group >> +terminates a given flow. >> + >> +There is no coordination and no leader election between the nodes of the >> +group while writing to SBDB: every ``ovn-controller`` independently does the >> +full bookkeeping for that one chassis, and it is the idempotence of those >> +writes that keeps them from fighting each other. >> + >> +Two consequences follow from this. First, the columns that OVN derives from >> +the local node rather than from the configuration must be forced to the same >> +value on every node. More detailed information on this is provided in the >> +``Identical OVS and OVN configuration and features`` section below. Second, >> +the nodes do not share any runtime state, so nothing that relies on >> conntrack >> +can be used on a replicated gateway, see the ``Only stateless traffic is >> +supported`` section below. > We shouldn't break in the middle of the verbatim section, that renders > weird. > >> + >> +Use case >> +-------- >> +This is useful when the capacity or the availability of a single gateway >> node >> +is not enough: the group scales horizontally, and losing one node only means >> +that the underlay stops using one of the ECMP nexthops, without any change >> in >> +the OVN databases and without the gateway having to be rescheduled to >> another >> +chassis. >> + >> +A typical deployment looks as follows: >> + >> +* Two (or more) gateway nodes are installed with the same OVN and OVS >> + versions and with an identical ``ovn-controller`` configuration. Each of >> + them configures the anycast encapsulation IP locally, e.g. on the loopback >> + interface, so that the kernel accepts the tunnel packets destined to it:: >> + >> + $ ovs-vsctl set open . external-ids:system-id=vgw >> + $ ovs-vsctl set open . external-ids:ovn-encap-ip=10.0.0.1 >> + $ ovs-vsctl set open . external-ids:ovn-encap-type=geneve >> + $ ip address add 10.0.0.1/32 dev lo >> + >> +* The anycast address is reachable in the underlay through the physical >> + addresses of all the nodes of the group. On the hypervisors, and on any >> + other node that has to reach the replicated gateway, this is an ECMP >> route:: >> + >> + $ ip route replace 10.0.0.1/32 \ >> + nexthop via 192.168.0.1 dev eth0 \ >> + nexthop via 192.168.0.2 dev eth0 >> + >> + In a real deployment the route is normally learned from the fabric, with >> the >> + gateway nodes advertising the anycast prefix over BGP. >> + >> +* The logical topology is configured as usual, referring to the chassis by >> its >> + single name, e.g.:: >> + >> + $ ovn-nbctl lrp-set-gateway-chassis lr0-public vgw 10 >> + >> +Requirements and restrictions >> +----------------------------- >> + >> +The nodes of a replicated gateway are not independent chassis, they are one >> +chassis that happens to run on several machines. Everything that OVN stores >> +per chassis is therefore shared, and everything that OVN computes per >> chassis >> +must produce the same result on every node of the group. This leads to the >> +following requirements. >> + >> +Identical OVS and OVN configuration and features >> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> + >> +All the nodes of the group must run the same OVS and OVN versions and must >> +expose the same set of features. ``ovn-controller`` reports the capabilities >> +of the local OVS instance in the ``Chassis`` record (``other_config``, for >> +example the supported datapath interface types, and the various >> +``ovn-chassis-feature`` entries), and ``ovn-northd`` generates the logical >> +flows for the chassis based on those values. If the nodes disagree, the >> +``Chassis`` record will keep flapping between the values reported by each >> node, >> +``ovn-northd`` will keep recomputing, and the node whose capabilities do not >> +match the ones currently published may not be able to process the flows it >> +receives. >> + >> +For the same reason, every chassis scoped configuration option has to be >> +identical on all the nodes, including, for example: >> ``external_ids:hostname``, >> +``external_ids:ovn-bridge-mappings`` and all others. >> + >> +Only stateless traffic is supported >> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> + >> +The nodes of a replicated gateway do not share any runtime state, and in >> +particular they do not share conntrack. The underlay picks one of them per >> +flow, and nothing guarantees that the two directions of a connection, or the >> +packets of a connection before and after a change of the ECMP nexthops (a >> node >> +being added, removed, rebooted or simply an ECMP rehash), are handled by the >> +same node. Any feature that needs the connection state to be on the node >> that >> +processes the packet is therefore not supported on a replicated gateway: >> + >> +* load balancers, >> +* stateful NAT, that is ``snat`` and ``dnat_and_snat`` (unless configured as >> + stateless), and NAT with ``ct-commit-all`` enabled, >> +* stateful ACLs, and anything else that relies on conntrack, for example >> + load balancer hairpinning or ECMP routes with >> + ``options:ecmp_symmetric_reply`` set. >> + >> +Only stateless forwarding, e.g. plain routing and switching, stateless NAT >> and >> +stateless ACLs, behaves correctly, because every packet can be handled >> +independently by any node of the group. >> + >> +Example >> +------- >> + >> +The ``ovn multinode replicated gateway`` test in ``tests/multinode.at`` >> sets up >> +a complete example of a two node replicated gateway with an underlay ECMP >> route >> +to the anycast encapsulation IP, a hypervisor with a VM behind the gateway >> +and an external node that does not run OVN at all. >> diff --git a/NEWS b/NEWS >> index f1c56dd14..7bc3f2113 100644 >> --- a/NEWS >> +++ b/NEWS >> @@ -129,6 +129,11 @@ OVN v26.09.0 - xxx xx xxxx >> represent a logical router port (e.g. ovn-ic transit switch LSPs). >> Such ports are treated like type=router, including omission from >> _MC_flood_l2. >> + - Documented the "replicated gateway" setup, in which several >> ovn-controller >> + instances share one chassis name and one anycast encapsulation IP >> + reachable over the underlay through ECMP, so that they appear to OVN as >> + a single chassis. See Documentation/topics/replicated-gateway.rst for >> the >> + requirements and the restrictions. >> >> OVN v26.03.0 - xxx xx xxxx >> -------------------------- >> diff --git a/controller/ovn-controller.8.xml >> b/controller/ovn-controller.8.xml >> index 3c33654ff..37c64ebf3 100644 >> --- a/controller/ovn-controller.8.xml >> +++ b/controller/ovn-controller.8.xml >> @@ -100,7 +100,14 @@ >> local OVN "etc" directory or via the <code>-n</code> command-line >> option. >> The following precedence is used: first, the command-line option is >> read; >> if not present, the <code>system-id-override</code> file is read; if >> not >> - present, then the name configured in the database is used.</dd> >> + present, then the name configured in the database is used. >> + <p> >> + Several <code>ovn-controller</code> instances running on different >> + nodes may share the same <code>system-id</code>, so that they appear >> + to OVN as a single chassis backed by an anycast encapsulation IP. >> + See the "Replicated Gateway" document for the requirements and the >> + restrictions of such a setup. >> + </p></dd> >> >> <dt><code>external_ids:hostname</code></dt> >> <dd>The hostname to use in the Chassis table.</dd> >> diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at >> index 1c85f79f8..81ab2a5b8 100644 >> --- a/tests/multinode-macros.at >> +++ b/tests/multinode-macros.at >> @@ -296,23 +296,33 @@ multinode_setup_northd() { >> m_as $c ovn-sbctl set-connection p${REMOTE_PROT}:6642 >> } >> >> -# multinode_setup_controller NODE ENCAP_IP REMOTE_IP [ENCAP_TYPE] >> -# >> -# Sets up controller on specified node. >> +# multinode_setup_controller NODE SYSTEM_ID ENCAP_IP REMOTE_IP >> +# [ENCAP_TYPE] [START_CONTROLLER] >> +# >> +# Sets up controller on specified node. SYSTEM_ID defaults to NODE; passing >> a >> +# different value (or the same value on more than one node) allows a test to >> +# control the chassis name the node registers with. START_CONTROLLER >> defaults >> +# to "yes"; with "no" only OVS is set up, so that a test can finish the node >> +# configuration before starting ovn-controller itself. >> multinode_setup_controller() { >> c=$1 >> + system_id=${2:-$1} >> encap_ip=$3 >> remote_ip=$4 >> encap_type=${5:-"geneve"} >> + start_controller=${6:-"yes"} >> >> # Cleanup existing one >> m_as $c /usr/share/openvswitch/scripts/ovs-ctl stop >> m_as $c /usr/share/ovn/scripts/ovn-ctl stop_controller >> m_as $c sh -c "rm -f /etc/openvswitch/*.db" >> >> - m_as $c /usr/share/openvswitch/scripts/ovs-ctl start --system-id=$c >> - echo "Using ${CONTROLLER_SSL_ARGS} for ovn-controller". >> - m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller >> ${CONTROLLER_SSL_ARGS} >> + m_as $c /usr/share/openvswitch/scripts/ovs-ctl start >> --system-id=$system_id >> + if [[ "$start_controller" = "yes" ]]; then >> + echo "Using ${CONTROLLER_SSL_ARGS} for ovn-controller". >> + m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller \ >> + ${CONTROLLER_SSL_ARGS} >> + fi > I think if we move the part that starts ovn-controller to the end of > this function we don't need the kind of ugly start_controller argument. > We can just set everything we need and then start ovn-controller. > >> >> m_as $c ovs-vsctl set open . external_ids:ovn-encap-ip=$encap_ip >> m_as $c ovs-vsctl set open . external-ids:ovn-encap-type=$encap_type >> diff --git a/tests/multinode.at b/tests/multinode.at >> index 0c277f5e8..16cfa804f 100644 >> --- a/tests/multinode.at >> +++ b/tests/multinode.at >> @@ -5554,3 +5554,200 @@ AT_CHECK([cat ch2_eth2.tcpdump], [0], [dnl >> ]) >> >> AT_CLEANUP >> + >> +AT_SETUP([ovn multinode replicated gateway]) >> + >> +check_fake_multinode_setup >> +cleanup_multinode_resources >> + >> +# Two gateway nodes are deployed as one and the same "replicated" gateway: >> both >> +# ovn-controllers register with the same chassis name and with the same >> anycast >> +# encap IP. The encap IP is reachable over the underlay via ECMP through the >> +# physical addresses of both gateway nodes, so a geneve tunnel to the >> replicated >> +# gateway may land on either of them. >> +# >> +# underlay (eth1, 170.168.0.0/16) >> +# ┌───────────────────┐ >> ┌──────────────────┐ >> +# │ ovn-chassis-1 │ 1.1.1.1/32 │ ovn-gw-1 >> │ >> +# │ (compute node) ├──── nexthop via gw-1 eth1 ────────┤ >> ovn-replicated-gw │ >> +# │ │ │ encap 1.1.1.1 >> │ >> +# │ sw0p1 10.0.0.3 ├──── nexthop via gw-2 eth1 ───┐ >> └────────┬─────────┘ >> +# └───────────────────┘ │ │ >> +# │ >> ┌────────┴─────────┐ >> +# ┌───────────────────┐ └────┤ ovn-gw-2 >> │ >> +# │ ovn-chassis-2 │ 1.1.1.1/32 │ >> ovn-replicated-gw │ >> +# │ (external client) ├──── nexthop via gw-1 eth1 ────────┤ encap 1.1.1.1 >> │ >> +# │ ├──── nexthop via gw-2 eth1 ────────┤ >> │ >> +# │ │ >> └────────┬─────────┘ >> +# │ eth2 172.20.0.50 ├───────── provider network (eth2, br-ex) ────┘ >> +# └───────────────────┘ lr0-public 172.20.0.100 >> +# >> +# On ovn-chassis-2 (aka client) we have client port in public switch for >> checking >> +# internal connectivity, public switch is connected to dgp port that >> claimed by >> +# replicated chassis. Compute and client node both have underlay route for >> replicated gw ip (1.1.1.1) >> + >> +vgw1=ovn-gw-1 >> +vgw2=ovn-gw-2 >> +compute=ovn-chassis-1 >> +client=ovn-chassis-2 >> +vgw_name=ovn-replicated-gw >> +vgw_encap_ip=1.1.1.1 >> + >> +m_underlay_ip() { >> + m_as $1 ip -4 addr show eth1 | grep inet | awk '{print $2}' | cut -d'/' >> -f1 >> +} >> + >> +central_ip=$(m_as ovn-central-az1 ip -4 addr show eth1 | grep inet | awk >> '{print $2}' | cut -d'/' -f1) > This should now be: > > central_ip=$( (m_central_as ip -4 addr show eth1 2>/dev/null || > m_central_as ip -4 addr show eth2) | grep inet | awk '{print $2}' | cut > -d'/' -f1) > > Since we merged 44d3bf28e4b7 ("multinode: Add support for INLINE_BR_EX > configuration of multinode.") > >> +vgw1_ip=$(m_underlay_ip $vgw1) >> +vgw2_ip=$(m_underlay_ip $vgw2) >> +compute_ip=$(m_underlay_ip $compute) >> + >> +m_as $compute ip link del sw0p1-p >> +m_as $client ip link del sw0p2-p >> +m_as $client ip link del sw1p1-p >> + >> +# Bring up both gateway nodes as the very same chassis: same chassis name >> and >> +# same encap IP, which is not part of the underlay address space. >> +for c in $vgw1 $vgw2; do >> + # Bring the node back to its original chassis name and encap IP. >> + on_exit "multinode_setup_controller $c $c $(m_underlay_ip $c) >> $central_ip" >> + multinode_setup_controller $c $vgw_name $vgw_encap_ip $central_ip >> geneve no >> + on_exit "m_as $c ip addr del $vgw_encap_ip/32 dev lo" >> + check m_as $c ip addr add $vgw_encap_ip/32 dev lo >> + check m_as $c ovs-vsctl set open . external-ids:hostname=$vgw_name >> + check m_as $c ovs-vsctl set open . >> external-ids:ovn-bridge-mappings=public:br-ex > I don't think your test uses the ovn-bridge mapping in any way. So then > we might as well add hostname as an optional argument to > multinode_setup_controller(). > >> +done >> + >> +for c in $vgw1 $vgw2; do >> + m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller >> ${CONTROLLER_SSL_ARGS} >> +done > If we change multinode_setup_controller() as I mentioned above, we don't > need to manually start it here. > >> + >> +# Both gateway nodes must end up sharing a single chassis record with a >> single >> +# encap record. >> +m_wait_row_count Chassis 1 name=$vgw_name >> +m_wait_row_count Encap 1 chassis_name=$vgw_name >> +m_check_column "$vgw_encap_ip" Encap ip chassis_name=$vgw_name >> + >> +# ECMP route to the replicated gateway encap IP in the underlay, both on the >> +# compute node and on the external client. >> +for c in $compute $client; do >> + on_exit "m_as $c ip route del $vgw_encap_ip/32" >> + check m_as $c ip route replace $vgw_encap_ip/32 \ >> + nexthop via $vgw1_ip dev eth1 \ >> + nexthop via $vgw2_ip dev eth1 >> +done >> + >> + > Nit: one empty line too many. > >> +# The anycast encap IP is reachable over the underlay from both nodes. >> +AT_CHECK([m_as $compute ping -q -c 3 -i 0.3 -w 2 $vgw_encap_ip | >> FORMAT_PING], [0], [dnl >> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >> +]) >> +AT_CHECK([m_as $client ping -q -c 3 -i 0.3 -w 2 $vgw_encap_ip | >> FORMAT_PING], [0], [dnl >> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >> +]) >> + >> +cr_lport_log_count() { >> + m_as $1 sh -c "cat /var/log/ovn/ovn-controller.log 2>/dev/null | \ >> + grep -ac 'Claiming lport cr-lr0-public'" >> +} >> + >> +vgw1_cr_logs=$(cr_lport_log_count $vgw1) >> +vgw2_cr_logs=$(cr_lport_log_count $vgw2) >> + >> +check multinode_nbctl ls-add sw0 >> +check multinode_nbctl lsp-add sw0 sw0-port1 >> +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 >> 10.0.0.3" >> + >> +check multinode_nbctl lr-add lr0 >> +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24 >> +check multinode_nbctl lsp-add-router-port sw0 sw0-lr0 lr0-sw0 >> + >> +check multinode_nbctl ls-add public >> +check multinode_nbctl lsp-add public external_client >> +check multinode_nbctl lsp-set-addresses external_client "50:55:00:00:00:03 >> 172.20.0.99" >> +check multinode_nbctl lrp-add lr0 lr0-public 00:11:22:00:ff:01 >> 172.20.0.100/24 >> +check multinode_nbctl lsp-add-router-port public public-lr0 lr0-public >> +check multinode_nbctl lrp-set-gateway-chassis lr0-public $vgw_name 10 >> + >> +m_as $compute /data/create_fake_vm.sh sw0-port1 sw0p1 50:54:00:00:00:03 >> 1342 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a >> +m_as $client /data/create_fake_vm.sh external_client external >> 50:55:00:00:00:03 1342 172.20.0.99 24 172.20.0.100 1001::3/64 1001::a >> + >> +check multinode_nbctl --wait=hv sync >> +m_wait_for_ports_up sw0-port1 >> +m_wait_for_ports_up external_client >> + >> +# The distributed gateway port is claimed by the replicated gateway chassis. >> +m_wait_row_count Port_Binding 1 logical_port=cr-lr0-public >> +vgw_uuid=$(m_central_as ovn-sbctl --bare --columns _uuid find Chassis >> name=$vgw_name) >> +m_wait_column "$vgw_uuid" Port_Binding chassis logical_port=cr-lr0-public >> + >> +# The two controllers share a single Chassis record and a single binding for >> +# the chassisredirect port. Since both nodes are configured identically and >> +# both resolve to the same chassis, each of them finds the records already >> +# holding the values it was about to write and its transaction is a no-op, >> so >> +# once the setup has settled no transaction touching those tables is >> expected. >> +# If the nodes fought over a record, they would add one transaction per main >> +# loop iteration to the southbound database log. >> +sb_txn_count() { >> + m_central_as grep -ac "\"$1\":" /etc/ovn/ovnsb_db.db >> +} >> + >> +chassis_txns=$(sb_txn_count Chassis) >> +pb_txns=$(sb_txn_count Port_Binding) >> +sleep 5 >> +check test $(($(sb_txn_count Chassis) - chassis_txns)) -le 1 >> +check test $(($(sb_txn_count Port_Binding) - pb_txns)) -le 1 >> + >> +cr_logs=$(($(cr_lport_log_count $vgw1) + $(cr_lport_log_count $vgw2))) >> +# Claim on both >> +check test $((cr_logs - vgw1_cr_logs - vgw2_cr_logs)) -le 2 >> + >> +check multinode_nbctl --wait=hv sync >> + >> +# North-south connectivity through the replicated gateway, with the >> underlay ECMP >> +# route in place on both ends. >> +check_ns_connectivity() { >> + M_NS_CHECK_EXEC([$compute], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 >> 172.20.0.99 | FORMAT_PING], \ >> +[0], [dnl >> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >> +]) >> + >> + M_NS_CHECK_EXEC([$compute], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 >> 172.20.0.100 | FORMAT_PING], \ >> +[0], [dnl >> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >> +]) >> + >> + M_NS_CHECK_EXEC([$client], [external], [ping -q -c 3 -i 0.3 -w 2 >> 10.0.0.3 | FORMAT_PING], \ >> +[0], [dnl >> +3 packets transmitted, 3 received, 0% packet loss, time 0ms >> +]) >> +} >> + >> +# The traffic towards the replicated gateway must be tunneled to the >> anycast encap IP. >> +M_START_TCPDUMPS([$compute], [-c 2 -neei eth1 host $vgw_encap_ip and udp >> port 6081], [ch1_underlay]) >> + >> +check_ns_connectivity >> + >> +OVS_WAIT_UNTIL([test -s ch1_underlay.tcpdump]) >> +AT_CHECK([grep -q "$vgw_encap_ip" ch1_underlay.tcpdump]) >> + >> +# Each of the two replicated gateway nodes on its own must be able to carry >> the >> +# traffic: pin the underlay route to a single nexthop at a time. >> +for nexthop in $vgw1_ip $vgw2_ip; do >> + for c in $compute $client; do >> + check m_as $c ip route replace $vgw_encap_ip/32 via $nexthop dev >> eth1 >> + done >> + >> + check_ns_connectivity >> +done >> + >> +# Back to ECMP over both gateway nodes. >> +for c in $compute $client; do >> + check m_as $c ip route replace $vgw_encap_ip/32 \ >> + nexthop via $vgw1_ip dev eth1 \ >> + nexthop via $vgw2_ip dev eth1 >> +done >> + >> +check_ns_connectivity >> + >> +AT_CLEANUP _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
