On 14/07/2021 14:50, Ilya Maximets wrote: > Main documentation for the service model and tutorial with the use case > and configuration examples. > > Acked-by: Dumitru Ceara <[email protected]> > Signed-off-by: Ilya Maximets <[email protected]> > --- > Documentation/automake.mk | 1 + > Documentation/ref/ovsdb.7.rst | 62 ++++++++++++-- > Documentation/topics/index.rst | 1 + > Documentation/topics/ovsdb-relay.rst | 124 +++++++++++++++++++++++++++ > NEWS | 3 + > ovsdb/ovsdb-server.1.in | 27 +++--- > 6 files changed, 200 insertions(+), 18 deletions(-) > create mode 100644 Documentation/topics/ovsdb-relay.rst > > diff --git a/Documentation/automake.mk b/Documentation/automake.mk > index bc30f94c5..213d9c867 100644 > --- a/Documentation/automake.mk > +++ b/Documentation/automake.mk > @@ -52,6 +52,7 @@ DOC_SOURCE = \ > Documentation/topics/networking-namespaces.rst \ > Documentation/topics/openflow.rst \ > Documentation/topics/ovs-extensions.rst \ > + Documentation/topics/ovsdb-relay.rst \ > Documentation/topics/ovsdb-replication.rst \ > Documentation/topics/porting.rst \ > Documentation/topics/record-replay.rst \ > diff --git a/Documentation/ref/ovsdb.7.rst b/Documentation/ref/ovsdb.7.rst > index e4f1bf766..980ba29e7 100644 > --- a/Documentation/ref/ovsdb.7.rst > +++ b/Documentation/ref/ovsdb.7.rst > @@ -121,13 +121,14 @@ schema checksum from a schema or database file, > respectively. > Service Models > ============== > > -OVSDB supports three service models for databases: **standalone**, > -**active-backup**, and **clustered**. The service models provide different > -compromises among consistency, availability, and partition tolerance. They > -also differ in the number of servers required and in terms of performance. > The > -standalone and active-backup database service models share one on-disk > format, > -and clustered databases use a different format, but the OVSDB programs work > -with both formats. ``ovsdb(5)`` documents these file formats. > +OVSDB supports four service models for databases: **standalone**, > +**active-backup**, **relay** and **clustered**. The service models provide > +different compromises among consistency, availability, and partition > tolerance. > +They also differ in the number of servers required and in terms of > performance. > +The standalone and active-backup database service models share one on-disk > +format, and clustered databases use a different format, but the OVSDB > programs > +work with both formats. ``ovsdb(5)`` documents these file formats. Relay > +databases have no on-disk storage. > > RFC 7047, which specifies the OVSDB protocol, does not mandate or specify > any particular service model. > @@ -406,6 +407,50 @@ following consequences: > that the client previously read. The OVSDB client library in Open vSwitch > uses this feature to avoid servers with stale data. > > +Relay Service Model > +------------------- > + > +A **relay** database is a way to scale out read-mostly access to the > +existing database working in any service model including relay. > + > +Relay database creates and maintains an OVSDB connection with another OVSDB > +server. It uses this connection to maintain an in-memory copy of the remote > +database (a.k.a. the ``relay source``) keeping the copy up-to-date as the > +database content changes on the relay source in the real time. > + > +The purpose of relay server is to scale out the number of database clients. > +Read-only transactions and monitor requests are fully handled by the relay > +server itself. For the transactions that request database modifications, > +relay works as a proxy between the client and the relay source, i.e. it > +forwards transactions and replies between them. > + > +Compared to the clustered and active-backup models, relay service model > +provides read and write access to the database similarly to a clustered > +database (and even more scalable), but with generally insignificant > performance > +overhead of an active-backup model. At the same time it doesn't increase > +availability that needs to be covered by the service model of the relay > source. > + > +Relay database has no on-disk storage and therefore cannot be converted to > +any other service model. > + > +If there is already a database started in any service model, to start a relay > +database server use ``ovsdb-server relay:<DB_NAME>:<relay source>``, where > +``<DB_NAME>`` is the database name as specified in the schema of the database > +that existing server runs, and ``<relay source>`` is an OVSDB connection > method > +(see `Connection Methods`_ below) that connects to the existing database > +server. ``<relay source>`` could contain a comma-separated list of > connection > +methods, e.g. to connect to any server of the clustered database. > +Multiple relay servers could be started for the same relay source. > + > +Since the way relays handle read and write transactions is very similar > +to the clustered model where "cluster" means "set of relay servers connected > +to the same relay source", "follower" means "relay server" and the "leader" > +means "relay source", same consistency consequences as for the clustered > +model applies to relay as well (See `Understanding Cluster Consistency`_ > +above). > + > +Open vSwitch 2.16 introduced support for relay service model. > + > Database Replication > ==================== > > @@ -414,7 +459,8 @@ Replication, in this context, means to make, and keep > up-to-date, a read-only > copy of the contents of a database (the ``replica``). One use of replication > is to keep an up-to-date backup of a database. A replica used solely for > backup would not need to support clients of its own. A set of replicas that > do > -serve clients could be used to scale out read access to the primary database. > +serve clients could be used to scale out read access to the primary database, > +however `Relay Service Model`_ is more suitable for that purpose. > > A database replica is set up in the same way as a backup server in an > active-backup pair, with the difference that the replica is never promoted to > diff --git a/Documentation/topics/index.rst b/Documentation/topics/index.rst > index 0036567eb..d8ccbd757 100644 > --- a/Documentation/topics/index.rst > +++ b/Documentation/topics/index.rst > @@ -44,6 +44,7 @@ OVS > openflow > bonding > networking-namespaces > + ovsdb-relay > ovsdb-replication > dpdk/index > windows > diff --git a/Documentation/topics/ovsdb-relay.rst > b/Documentation/topics/ovsdb-relay.rst > new file mode 100644 > index 000000000..50a3c6d07 > --- /dev/null > +++ b/Documentation/topics/ovsdb-relay.rst > @@ -0,0 +1,124 @@ > +.. > + Copyright 2021, Red Hat, Inc. > + > + Licensed under the Apache License, Version 2.0 (the "License"); you may > + not use this file except in compliance with the License. You may obtain > + a copy of the License at > + > + http://www.apache.org/licenses/LICENSE-2.0 > + > + Unless required by applicable law or agreed to in writing, software > + distributed under the License is distributed on an "AS IS" BASIS, > WITHOUT > + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See > the > + License for the specific language governing permissions and limitations > + under the License. > + > + Convention for heading levels in Open vSwitch documentation: > + > + ======= Heading 0 (reserved for the title in a document) > + ------- Heading 1 > + ~~~~~~~ Heading 2 > + +++++++ Heading 3 > + ''''''' Heading 4 > + > + Avoid deeper levels because they do not render well. > + > +=============================== > +Scaling OVSDB Access With Relay > +=============================== > + > +Open vSwitch 2.16 introduced support for OVSDB Relay mode with the goal to > +increase database scalability for a big deployments. Mainly, OVN (Open > Virtual > +Network) Southbound Database deployments. This document describes the main > +concept and provides the configuration examples. > + > +What is OVSDB Relay? > +-------------------- > + > +Relay is a database service model in which one ``ovsdb-server`` (``relay``) > +connects to another standalone or clustered database server > +(``relay source``) and maintains in-memory copy of its data, receiving > +all the updates via this OVSDB connection. Relay server handles all the > +read-only requests (monitors and transactions) on its own and forwards all > the > +transactions that requires database modifications to the relay source. > + > +Why is this needed? > +------------------- > + > +Some OVN deployment could have hundreds or even thousands of nodes. On each > of > +these nodes there is an ovn-controller, which is connected to the > +OVN_Southbound database that is served by a standalone or clustered OVSDB. > +Standalone database is handled by a single ovsdb-server process and clustered > +could consist of 3 to 5 ovsdb-server processes. For the clustered database, > +higher number of servers may significantly increase transaction latency due > +to necessity for these servers to reach consensus. So, in the end limited > +number of ovsdb-server processes serves ever growing number of clients and > this > +leads to performance issues. > + > +Read-only access could be scaled up with OVSDB replication on top of > +active-backup service model, but ovn-controller is a read-mostly client, not > +a read-only, i.e. it needs to execute write transactions from time to time. > +Here relay service model comes into play. > + > +2-Tier Deployment > +----------------- > + > +Solution for the scaling issue could look like a 2-tier deployment, where > +a set of relay servers is connected to the main database cluster > +(OVN_Southbound) and clients (ovn-conrtoller) connected to these relay > +servers:: > + > + 172.16.0.1 > + +--------------------+ +----+ ovsdb-relay-1 +--+---+ client-1 > + | | | | > + | Clustered | | +---+ client-2 > + | Database | | ... > + | | | +---+ client-N > + | 10.0.0.2 | | > + | ovsdb-server-2 | | 172.16.0.2 > + | + + | +----+ ovsdb-relay-2 +--+---+ client-N+1 > + | | | | | | > + | | + +---+ +---+ client-N+2 > + | | 10.0.0.1 | | ... > + | | ovsdb-server-1 | | +---+ client-2N > + | | + | | > + | | | | | > + | + + | + ... ... ... ... ... > + | ovsdb-server-3 | | > + | 10.0.0.3 | | +---+ client-KN-1 > + | | | 172.16.0.K | > + +--------------------+ +----+ ovsdb-relay-K +--+---+ client-KN > + > +In practice, the picture might look a bit more complex, because all relay > +servers might connect to any member of a main cluster and clients might > +connect to any relay server of their choice. > + > +Assuming that servers of a main cluster started like this:: > + > + $ ovsdb-server --remote=ptcp:6642:10.0.0.1 ovn-sb-1.db > + > +The same for other two servers. In this case relay servers could be > +started like this:: > + > + $ REMOTES=tcp:10.0.0.1:6642,tcp:10.0.0.2:6642,tcp:10.0.0.3:6642 > + $ ovsdb-server --remote=ptcp:6642:172.16.0.1 relay:OVN_Southbound:$REMOTES > + $ ... > + $ ovsdb-server --remote=ptcp:6642:172.16.0.K relay:OVN_Southbound:$REMOTES > + > +Every relay server could connect to any of the cluster members of their > choice, > +fairness of load distribution is achieved by shuffling remotes. > + > +For the actual clients, they could be configured to connect to any of the > +relay servers. For ovn-controllers the configuration could look like this:: > + > + $ REMOTES=tcp:172.16.0.1:6642,...,tcp:172.16.0.K:6642 > + $ ovs-vsctl set Open_vSwitch . external-ids:ovn-remote=$REMOTES > + > +Setup like this allows the system to serve ``K * N`` clients while having > only > +``K`` actual connections on the main clustered database keeping it in a > +stable state. > + > +It's also possible to create multi-tier deployments by connecting one set > +of relay servers to another (smaller) set of relay servers, or even create > +tree-like structures with the cost of increased latency for write > transactions, > +because they will be forwarded multiple times. > diff --git a/NEWS b/NEWS > index 6cdccc715..aed7d811b 100644 > --- a/NEWS > +++ b/NEWS > @@ -7,6 +7,9 @@ Post-v2.15.0 > limits in addition to the previously configurable byte rate settings. > This is not supported in the userspace datapath yet. > - OVSDB: > + * Introduced new database service model - "relay". Targeted to scale > out > + read-mostly access (ovn-controller) to existing databases. > + For more information: ovsdb(7) and > Documentation/topics/ovsdb-relay.rst > * New command line options --record/--replay for ovsdb-server and > ovsdb-client to record and replay all the incoming transactions, > monitors, etc. More datails in > Documentation/topics/record-replay.rst. > diff --git a/ovsdb/ovsdb-server.1.in b/ovsdb/ovsdb-server.1.in > index fdd52e8f6..dac0f02cb 100644 > --- a/ovsdb/ovsdb-server.1.in > +++ b/ovsdb/ovsdb-server.1.in > @@ -10,6 +10,7 @@ ovsdb\-server \- Open vSwitch database server > .SH SYNOPSIS > \fBovsdb\-server\fR > [\fIdatabase\fR]\&... > +[\fIrelay:schema_name:remote\fR]\&... > [\fB\-\-remote=\fIremote\fR]\&... > [\fB\-\-run=\fIcommand\fR] > .so lib/daemon-syn.man > @@ -35,12 +36,15 @@ For an introduction to OVSDB and its implementation in > Open vSwitch, > see \fBovsdb\fR(7). > .PP > Each OVSDB file may be specified on the command line as \fIdatabase\fR. > -If none is specified, the default is \fB@DBDIR@/conf.db\fR. The database > -files must already have been created and initialized using, for > -example, \fBovsdb\-tool\fR's \fBcreate\fR, \fBcreate\-cluster\fR, or > -\fBjoin\-cluster\fR command. > +Relay databases may be specified on the command line as > +\fIrelay:schema_name:remote\fR. For a detailed description of relay database > +argument, see \fBovsdb\fR(7). > +If none of database files or relay databases is specified, the default is > +\fB@DBDIR@/conf.db\fR. The database files must already have been created and > +initialized using, for example, \fBovsdb\-tool\fR's \fBcreate\fR, > +\fBcreate\-cluster\fR, or \fBjoin\-cluster\fR command. > .PP > -This OVSDB implementation supports standalone, active-backup, and > +This OVSDB implementation supports standalone, active-backup, relay and > clustered database service models, as well as database replication. > See the Service Models section of \fBovsdb\fR(7) for more information. > .PP > @@ -50,7 +54,9 @@ successfully join a cluster (if the database file is > freshly created > with \fBovsdb\-tool join\-cluster\fR) or connect to a cluster that it > has already joined. Use \fBovsdb\-client wait\fR (see > \fBovsdb\-client\fR(1)) to wait until the server has successfully > -joined and connected to a cluster. > +joined and connected to a cluster. The same is true for relay databases. > +Same commands could be used to wait for a relay database to connect to > +the relay source (remote). > .PP > In addition to user-specified databases, \fBovsdb\-server\fR version > 2.9 and later also always hosts a built-in database named > @@ -243,10 +249,11 @@ not list remotes added indirectly because they were > read from the > database by configuring a > \fBdb:\fIdb\fB,\fItable\fB,\fIcolumn\fR remote. > . > -.IP "\fBovsdb\-server/add\-db \fIdatabase\fR" > -Adds the \fIdatabase\fR to the running \fBovsdb\-server\fR. The database > -file must already have been created and initialized using, for example, > -\fBovsdb\-tool create\fR. > +.IP "\fBovsdb\-server/add\-db \fIdatabase\fR > +Adds the \fIdatabase\fR to the running \fBovsdb\-server\fR. \fIdatabase\fR > +could be a database file or a relay description in the following format: > +\fIrelay:schema_name:remote\fR. The database file must already have been > +created and initialized using, for example, \fBovsdb\-tool create\fR. > . > .IP "\fBovsdb\-server/remove\-db \fIdatabase\fR" > Removes \fIdatabase\fR from the running \fBovsdb\-server\fR. \fIdatabase\fR >
Acked-by: Mark D. Gray <[email protected]> _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
