Several ovn-controller instances may run with the same chassis name and the same encapsulation IP, an anycast address that is reachable over the underlay through ECMP. OVN then sees the whole group of nodes as a single chassis: a single Chassis record, a single Encap record and a single binding for every port placed on it, while the underlay decides which node of the group terminates a given tunnel flow.
There is no coordination and no leader election between the nodes of the group while writing to SBDB: every ``ovn-controller`` independently does the full bookkeeping for that one chassis, and it is the idempotence of those writes that keeps them from fighting each other. The flip side is that the nodes must be configured identically in chassis-related params, and that nothing which relies on conntrack can be used, because the group shares no runtime state. Document all of this, and add a multinode test that deploys two gateway nodes as one virtual gateway, gives a hypervisor and a node that does not run OVN an underlay ECMP route to the anycast encapsulation IP, checks the north-south connectivity through each of the two nodes and checks that the controllers do not fight over the shared records. Signed-off-by: Alexandra Rukomoinikova <[email protected]> --- Documentation/automake.mk | 1 + Documentation/topics/index.rst | 1 + Documentation/topics/virtual-gateway.rst | 168 ++++++++++++++++++ NEWS | 5 + controller/ovn-controller.8.xml | 9 +- tests/multinode-macros.at | 18 +- tests/multinode.at | 216 ++++++++++++++++++++++- 7 files changed, 411 insertions(+), 7 deletions(-) create mode 100644 Documentation/topics/virtual-gateway.rst diff --git a/Documentation/automake.mk b/Documentation/automake.mk index 685ba9588..fcf63c784 100644 --- a/Documentation/automake.mk +++ b/Documentation/automake.mk @@ -25,6 +25,7 @@ DOC_SOURCE = \ Documentation/topics/testing.rst \ Documentation/topics/test-development.rst \ Documentation/topics/high-availability.rst \ + Documentation/topics/virtual-gateway.rst \ Documentation/topics/dynamic-routing/architecture.rst \ Documentation/topics/dynamic-routing/configuration-examples.rst \ Documentation/topics/dynamic-routing/index.rst \ diff --git a/Documentation/topics/index.rst b/Documentation/topics/index.rst index 0b676a98b..6749cb6d0 100644 --- a/Documentation/topics/index.rst +++ b/Documentation/topics/index.rst @@ -40,6 +40,7 @@ OVN dynamic-routing/index incremental-processing/index high-availability + virtual-gateway role-based-access-control vif-plug-providers/index testing diff --git a/Documentation/topics/virtual-gateway.rst b/Documentation/topics/virtual-gateway.rst new file mode 100644 index 000000000..7d39bfbe2 --- /dev/null +++ b/Documentation/topics/virtual-gateway.rst @@ -0,0 +1,168 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in OVN documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +================ +Virtual Gateway +================ + +A *virtual gateway* is a group of two or more physical nodes that present +themselves to OVN as a single chassis. Every node of the group runs its own +``ovn-controller``, but all of them are configured with the same chassis name +(``external_ids:system-id``) and with the same encapsulation IP +(``external_ids:ovn-encap-ip``). The encapsulation IP is an anycast address, +it is announced into the underlay (statically or by a routing protocol) by all +the nodes of the group, and the underlay load-balances the tunnel traffic +between them with ECMP. + +:: + + underlay with ECMP to 10.0.0.1 + + +-------------------+ +----------------------+ + | | nexthop via gw-1 | gateway node 1 | + | hypervisor +---------------------->| system-id: vgw | + | | | ovn-encap-ip: | + | ovn-encap-ip: | | 10.0.0.1 | + | 192.168.0.10 | +----------+-----------+ + | | | + | | nexthop via gw-2 +----------+-----------+ + | +---------------------->| gateway node 2 | + | | | system-id: vgw | + +-------------------+ | ovn-encap-ip: | + | 10.0.0.1 | + +----------+-----------+ + | + physical network | + -------------------------------------------------------+------------ + +From the point of view of the OVN databases there is a single chassis: a single +``Chassis`` record, a single ``Encap`` record, and a single binding for every +port that is placed on the virtual gateway (for example the chassisredirect +port of a distributed gateway port). ``ovn-northd`` and the other +``ovn-controller`` instances do not know, and do not need to know, how many +physical nodes are behind that chassis. A hypervisor keeps a single tunnel to +the anycast encapsulation IP, and the underlay decides which node of the group +terminates a given flow. + +There is no coordination and no leader election between the nodes of the +group while writing to SBDB: every ``ovn-controller`` independently does the +full bookkeeping for that one chassis, and it is the idempotence of those +writes that keeps them from fighting each other. +Two consequences follow from this. The columns that OVN derives from the +local node rather than from the configuration must be forced to the same value +on every node, More detailed information on this is provided in the +``Identical OVS and OVN configuration and features`` section below. + +Use case +-------- +This is useful when the capacity or the availability of a single gateway node +is not enough: the group scales horizontally, and losing one node only means +that the underlay stops using one of the ECMP nexthops, without any change in +the OVN databases and without the gateway having to be rescheduled to another +chassis. + +A typical deployment looks as follows: + +* Two (or more) gateway nodes are installed with the same OVN and OVS + versions and with an identical ``ovn-controller`` configuration. Each of + them configures the anycast encapsulation IP locally, e.g. on the loopback + interface, so that the kernel accepts the tunnel packets destined to it:: + + $ ovs-vsctl set open . external-ids:system-id=vgw + $ ovs-vsctl set open . external-ids:ovn-encap-ip=10.0.0.1 + $ ovs-vsctl set open . external-ids:ovn-encap-type=geneve + $ ip address add 10.0.0.1/32 dev lo + +* The anycast address is reachable in the underlay through the physical + addresses of all the nodes of the group. On the hypervisors, and on any + other node that has to reach the virtual gateway, this is an ECMP route:: + + $ ip route replace 10.0.0.1/32 \ + nexthop via 192.168.0.1 dev eth0 \ + nexthop via 192.168.0.2 dev eth0 + + In a real deployment the route is normally learned from the fabric, with the + gateway nodes advertising the anycast prefix over BGP. + +* The logical topology is configured as usual, referring to the chassis by its + single name, e.g.:: + + $ ovn-nbctl lrp-set-gateway-chassis lr0-public vgw 10 + +Requirements and restrictions +----------------------------- + +The nodes of a virtual gateway are not independent chassis, they are one +chassis that happens to run on several machines. Everything that OVN stores +per chassis is therefore shared, and everything that OVN computes per chassis +must produce the same result on every node of the group. This leads to the +following requirements. + +Identical OVS and OVN configuration and features +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +All the nodes of the group must run the same OVS and OVN versions and must +expose the same set of features. ``ovn-controller`` reports the capabilities +of the local OVS instance in the ``Chassis`` record (``other_config``, for +example the supported datapath interface types, and the various +``ovn-chassis-feature`` entries), and ``ovn-northd`` generates the logical +flows for the chassis based on those values. If the nodes disagree, the +``Chassis`` record will keep flapping between the values reported by each node, +``ovn-northd`` will keep recomputing, and the node whose capabilities do not +match the ones currently published may not be able to process the flows it +receives. + +For the same reason, every chassis scoped configuration option has to be +identical on all the nodes, including , for example: ``external_ids:hostname``, +``external_ids:ovn-bridge-mappings`` and all others. + +Only stateless traffic is supported +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The nodes of a virtual gateway do not share any runtime state, and in +particular they do not share conntrack. The underlay picks one of them per +flow, and nothing guarantees that the two directions of a connection, or the +packets of a connection before and after a change of the ECMP nexthops (a node +being added, removed, rebooted or simply an ECMP rehash), are handled by the +same node. Any feature that needs the connection state to be on the node that +processes the packet is therefore not supported on a virtual gateway: + +* load balancers, +* stateful NAT, that is ``snat`` and ``dnat_and_snat`` (unless configured as + stateless), and NAT with ``ct-commit-all`` enabled, +* stateful ACLs, and anything else that relies on conntrack, for example + load balancer hairpinning or ECMP routes with + ``options:ecmp_symmetric_reply`` set. + +Only stateless forwarding, e.g. plain routing and switching, stateless NAT and +stateless ACLs, behaves correctly, because every packet can be handled +independently by any node of the group. + +Example +------- + +The ``ovn multinode virtual gateway - anycast encap ip with underlay ECMP`` +test in ``tests/multinode.at`` sets up a complete example of a two node +virtual gateway with an underlay ECMP route to the anycast encapsulation IP, +a hypervisor with a VM behind the gateway and an external node that does not +run OVN at all. diff --git a/NEWS b/NEWS index 0c7e3fda1..4955fb084 100644 --- a/NEWS +++ b/NEWS @@ -108,6 +108,11 @@ OVN v26.09.0 - xxx xx xxxx "options:dynamic-routing-advertise-prefixes" key that contains an explicit list of prefixes to be advertised via this port. If it is set, only these prefixes will be announced. + - Documented the "virtual gateway" setup, in which several ovn-controller + instances share one chassis name and one anycast encapsulation IP + reachable over the underlay through ECMP, so that they appear to OVN as + a single chassis. See Documentation/topics/virtual-gateway.rst for the + requirements and the restrictions. OVN v26.03.0 - xxx xx xxxx -------------------------- diff --git a/controller/ovn-controller.8.xml b/controller/ovn-controller.8.xml index 8121f7045..745d08f2e 100644 --- a/controller/ovn-controller.8.xml +++ b/controller/ovn-controller.8.xml @@ -79,7 +79,14 @@ local OVN "etc" directory or via the <code>-n</code> command-line option. The following precedence is used: first, the command-line option is read; if not present, the <code>system-id-override</code> file is read; if not - present, then the name configured in the database is used.</dd> + present, then the name configured in the database is used. + <p> + Several <code>ovn-controller</code> instances running on different + nodes may share the same <code>system-id</code>, so that they appear + to OVN as a single chassis backed by an anycast encapsulation IP. + See the "Virtual Gateway" document for the requirements and the + restrictions of such a setup. + </p></dd> <dt><code>external_ids:hostname</code></dt> <dd>The hostname to use in the Chassis table.</dd> diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at index ade4d167c..3d969bc1e 100644 --- a/tests/multinode-macros.at +++ b/tests/multinode-macros.at @@ -297,23 +297,30 @@ multinode_setup_northd() { m_as $c ovn-sbctl set-connection p${REMOTE_PROT}:6642 } -# multinode_setup_controller NODE ENCAP_IP REMOTE_IP [ENCAP_TYPE] +# multinode_setup_controller NODE SYSTEM_ID ENCAP_IP REMOTE_IP [ENCAP_TYPE] # -# Sets up controller on specified node. +# Sets up controller on specified node. SYSTEM_ID defaults to NODE; passing a +# different value (or the same value on more than one node) allows a test to +# control the chassis name the node registers with. multinode_setup_controller() { c=$1 + system_id=${2:-$1} encap_ip=$3 remote_ip=$4 encap_type=${5:-"geneve"} + start_controller=${6:-"yes"} # Cleanup existing one m_as $c /usr/share/openvswitch/scripts/ovs-ctl stop m_as $c /usr/share/ovn/scripts/ovn-ctl stop_controller m_as $c sh -c "rm -f /etc/openvswitch/*.db" - m_as $c /usr/share/openvswitch/scripts/ovs-ctl start --system-id=$c - echo "Using ${CONTROLLER_SSL_ARGS} for ovn-controller". - m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller ${CONTROLLER_SSL_ARGS} + m_as $c /usr/share/openvswitch/scripts/ovs-ctl start --system-id=$system_id + if [[ "$start_controller" = "yes" ]]; then + echo "Using ${CONTROLLER_SSL_ARGS} for ovn-controller". + m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller \ + ${CONTROLLER_SSL_ARGS} + fi m_as $c ovs-vsctl set open . external_ids:ovn-encap-ip=$encap_ip m_as $c ovs-vsctl set open . external-ids:ovn-encap-type=$encap_type @@ -322,6 +329,7 @@ multinode_setup_controller() { m_as $c ovs-vsctl set open . external-ids:ovn-remote-probe-interval=180000 m_as $c ovs-vsctl set open . external-ids:ovn-bridge-datapath-type=system + # Add back br-ex which was removed when removing ovs conf.db m_as $c ovs-vsctl --if-exists del-br br-ex m_as $c ovs-vsctl add-br br-ex diff --git a/tests/multinode.at b/tests/multinode.at index d70d28201..011f3d221 100644 --- a/tests/multinode.at +++ b/tests/multinode.at @@ -9,7 +9,7 @@ check_fake_multinode_setup cleanup_multinode_resources # Test East-West switching -check multinode_nbctl ls-add sw0 + check multinode_nbctl lsp-add sw0 sw0-port1 check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 10.0.0.3 1000::3" check multinode_nbctl lsp-add sw0 sw0-port2 @@ -5349,3 +5349,217 @@ AT_CHECK([cat ch2_eth2.tcpdump], [0], [dnl ]) AT_CLEANUP + +AT_SETUP([ovn multinode virtual gateway]) + +check_fake_multinode_setup +cleanup_multinode_resources + +# Two gateway nodes are deployed as one and the same "virtual" gateway: both +# ovn-controllers register with the same chassis name and with the same anycast +# encap IP. The encap IP is reachable over the underlay via ECMP through the +# physical addresses of both gateway nodes, so a geneve tunnel to the virtual +# gateway may land on either of them. +# +# underlay (eth1, 170.168.0.0/16) +# ┌───────────────────┐ ┌──────────────────┐ +# │ ovn-chassis-1 │ 1.1.1.1/32 │ ovn-gw-1 │ +# │ (compute node) ├──── nexthop via gw-1 eth1 ────────┤ ovn-virtual-gw │ +# │ │ │ encap 1.1.1.1 │ +# │ sw0p1 10.0.0.3 ├──── nexthop via gw-2 eth1 ───┐ └────────┬─────────┘ +# └───────────────────┘ │ │ +# │ ┌────────┴─────────┐ +# ┌───────────────────┐ └────┤ ovn-gw-2 │ +# │ ovn-chassis-2 │ 1.1.1.1/32 │ ovn-virtual-gw │ +# │ (external client) ├──── nexthop via gw-1 eth1 ────────┤ encap 1.1.1.1 │ +# │ ├──── nexthop via gw-2 eth1 ────────┤ │ +# │ │ └────────┬─────────┘ +# │ eth2 172.20.0.50 ├───────── provider network (eth2, br-ex) ────┘ +# └───────────────────┘ lr0-public 172.20.0.100 +# +# On ovn-chassis-2 (aka client) we have client port in public switch for checking +# internal connectivity, public switch is connected to dgp port that claimed by +# virtual chassis. Compute and client node both have underlay route for virtul gw ip (1.1.1.1) + +vgw1=ovn-gw-1 +vgw2=ovn-gw-2 +compute=ovn-chassis-1 +client=ovn-chassis-2 +vgw_name=ovn-virtual-gw +vgw_encap_ip=1.1.1.1 + +m_underlay_ip() { + m_as $1 ip -4 addr show eth1 | grep inet | awk '{print $2}' | cut -d'/' -f1 +} + +central_ip=$(m_as ovn-central-az1 ip -4 addr show eth1 | grep inet | awk '{print $2}' | cut -d'/' -f1) +vgw1_ip=$(m_underlay_ip $vgw1) +vgw2_ip=$(m_underlay_ip $vgw2) +compute_ip=$(m_underlay_ip $compute) + +m_as $compute ip link del sw0p1-p +m_as $client ip link del sw0p2-p +m_as $client ip link del sw1p1-p + +# Bring up both gateway nodes as the very same chassis: same chassis name and +# same encap IP, which is not part of the underlay address space. +for c in $vgw1 $vgw2; do + multinode_setup_controller $c $vgw_name $vgw_encap_ip $central_ip geneve no + on_exit "m_as $c ip addr del $vgw_encap_ip/32 dev lo" + check m_as $c ip addr add $vgw_encap_ip/32 dev lo + check m_as $c ovs-vsctl set open . external-ids:hostname=$vgw_name + check m_as $c ovs-vsctl set open . external-ids:ovn-bridge-mappings=public:br-ex +done + +for c in $vgw1 $vgw2; do + m_as $c /usr/share/ovn/scripts/ovn-ctl start_controller ${CONTROLLER_SSL_ARGS} +done + +# Both gateway nodes must end up sharing a single chassis record with a single +# encap record. +m_wait_row_count Chassis 1 name=$vgw_name +m_wait_row_count Encap 1 chassis_name=$vgw_name +m_check_column "$vgw_encap_ip" Encap ip chassis_name=$vgw_name + +# ECMP route to the virtual gateway encap IP in the underlay, both on the +# compute node and on the external client. +for c in $compute $client; do + on_exit "m_as $c ip route del $vgw_encap_ip/32" + check m_as $c ip route replace $vgw_encap_ip/32 \ + nexthop via $vgw1_ip dev eth1 \ + nexthop via $vgw2_ip dev eth1 +done + + +# The anycast encap IP is reachable over the underlay from both nodes. +AT_CHECK([m_as $compute ping -q -c 3 -i 0.3 -w 2 $vgw_encap_ip | FORMAT_PING], [0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) +AT_CHECK([m_as $client ping -q -c 3 -i 0.3 -w 2 $vgw_encap_ip | FORMAT_PING], [0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + +cr_lport_log_count() { + m_as $1 sh -c "cat /var/log/ovn/ovn-controller.log 2>/dev/null | \ + grep -ac 'Claiming lport cr-lr0-public'" +} + +vgw1_cr_logs=$(cr_lport_log_count $vgw1) +vgw2_cr_logs=$(cr_lport_log_count $vgw2) + +check multinode_nbctl ls-add sw0 +check multinode_nbctl lsp-add sw0 sw0-port1 +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 10.0.0.3" + +check multinode_nbctl lr-add lr0 +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24 +check multinode_nbctl lsp-add-router-port sw0 sw0-lr0 lr0-sw0 + +check multinode_nbctl ls-add public +check multinode_nbctl lsp-add public external_client +check multinode_nbctl lsp-set-addresses external_client "50:55:00:00:00:03 172.20.0.99" +check multinode_nbctl lrp-add lr0 lr0-public 00:11:22:00:ff:01 172.20.0.100/24 +check multinode_nbctl lsp-add-router-port public public-lr0 lr0-public +check multinode_nbctl lrp-set-gateway-chassis lr0-public $vgw_name 10 + +m_as $compute /data/create_fake_vm.sh sw0-port1 sw0p1 50:54:00:00:00:03 1342 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a +m_as $client /data/create_fake_vm.sh external_client external 50:55:00:00:00:03 1342 172.20.0.99 24 172.20.0.100 1001::3/64 1001::a + +check multinode_nbctl --wait=hv sync +m_wait_for_ports_up sw0-port1 +m_wait_for_ports_up external_client + +# The distributed gateway port is claimed by the virtual gateway chassis. +m_wait_row_count Port_Binding 1 logical_port=cr-lr0-public +vgw_uuid=$(m_central_as ovn-sbctl --bare --columns _uuid find Chassis name=$vgw_name) +m_wait_column "$vgw_uuid" Port_Binding chassis logical_port=cr-lr0-public + +check multinode_nbctl --wait=hv sync +m_wait_for_ports_up sw0-port1 + +# The distributed gateway port is claimed by the virtual gateway chassis. +m_wait_row_count Port_Binding 1 logical_port=cr-lr0-public +vgw_uuid=$(m_central_as ovn-sbctl --bare --columns _uuid find Chassis name=$vgw_name) +m_wait_column "$vgw_uuid" Port_Binding chassis logical_port=cr-lr0-public + +# The two controllers share a single Chassis record and a single binding for +# the chassisredirect port. Since both nodes are configured identically and +# both resolve to the same chassis, each of them finds the records already +# holding the values it was about to write and its transaction is a no-op, so +# once the setup has settled no transaction touching those tables is expected. +# If the nodes fought over a record, they would add one transaction per main +# loop iteration to the southbound database log. +sb_txn_count() { + m_central_as grep -ac "\"$1\":" /etc/ovn/ovnsb_db.db +} + +chassis_txns=$(sb_txn_count Chassis) +pb_txns=$(sb_txn_count Port_Binding) +sleep 5 +check test $(($(sb_txn_count Chassis) - chassis_txns)) -le 1 +check test $(($(sb_txn_count Port_Binding) - pb_txns)) -le 1 + +cr_logs=$(($(cr_lport_log_count $vgw1) + $(cr_lport_log_count $vgw2))) +# Claim on both +check test $((cr_logs - vgw1_cr_logs - vgw2_cr_logs)) -le 2 + +check multinode_nbctl --wait=hv sync + +# North-south connectivity through the virtual gateway, with the underlay ECMP +# route in place on both ends. +check_ns_connectivity() { + M_NS_CHECK_EXEC([$compute], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 172.20.0.99 | FORMAT_PING], \ +[0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + + M_NS_CHECK_EXEC([$compute], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 172.20.0.100 | FORMAT_PING], \ +[0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + + M_NS_CHECK_EXEC([$client], [external], [ping -q -c 3 -i 0.3 -w 2 10.0.0.3 | FORMAT_PING], \ +[0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) +} + +# The traffic towards the virtual gateway must be tunneled to the anycast encap IP. +M_START_TCPDUMPS([$compute], [-c 2 -neei eth1 host $vgw_encap_ip and udp port 6081], [ch1_underlay]) + +check_ns_connectivity + +OVS_WAIT_UNTIL([test -s ch1_underlay.tcpdump]) +AT_CHECK([grep -q "$vgw_encap_ip" ch1_underlay.tcpdump]) + +# Each of the two virtual gateway nodes on its own must be able to carry the +# traffic: pin the underlay route to a single nexthop at a time. +for nexthop in $vgw1_ip $vgw2_ip; do + for c in $compute $client; do + check m_as $c ip route replace $vgw_encap_ip/32 via $nexthop dev eth1 + done + + check_ns_connectivity +done + +# Back to ECMP over both gateway nodes. +for c in $compute $client; do + check m_as $c ip route replace $vgw_encap_ip/32 \ + nexthop via $vgw1_ip dev eth1 \ + nexthop via $vgw2_ip dev eth1 +done + +check_ns_connectivity + +# Restore the original chassis configuration of the nodes used by this test. +# Connect the chassis back to the original northd and remove northd per chassis. +for i in 1 2; do + chassis="ovn-chassis-$i" + ip=$(m_as $chassis ip -4 addr show eth1 | grep inet | awk '{print $2}' | cut -d'/' -f1) + + multinode_cleanup_ic $chassis + multinode_setup_controller $chassis $chassis $ip "170.168.0.2" + multinode_cleanup_northd $chassis +done + +AT_CLEANUP -- 2.48.1 _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
