On 6/25/21 3:35 PM, Dumitru Ceara wrote: > On 6/12/21 4:00 AM, Ilya Maximets wrote: >> Main documentation for the service model and tutorial with the use case >> and configuration examples. >> >> Signed-off-by: Ilya Maximets <[email protected]> >> --- > > I left a few minor comments below. With them addressed: > > Acked-by: Dumitru Ceara <[email protected]> > > Thanks! > >> Documentation/automake.mk | 1 + >> Documentation/ref/ovsdb.7.rst | 62 ++++++++++++-- >> Documentation/topics/index.rst | 1 + >> Documentation/topics/ovsdb-relay.rst | 124 +++++++++++++++++++++++++++ >> NEWS | 3 + >> ovsdb/ovsdb-server.1.in | 27 +++--- >> 6 files changed, 200 insertions(+), 18 deletions(-) >> create mode 100644 Documentation/topics/ovsdb-relay.rst >> >> diff --git a/Documentation/automake.mk b/Documentation/automake.mk >> index bc30f94c5..213d9c867 100644 >> --- a/Documentation/automake.mk >> +++ b/Documentation/automake.mk >> @@ -52,6 +52,7 @@ DOC_SOURCE = \ >> Documentation/topics/networking-namespaces.rst \ >> Documentation/topics/openflow.rst \ >> Documentation/topics/ovs-extensions.rst \ >> + Documentation/topics/ovsdb-relay.rst \ >> Documentation/topics/ovsdb-replication.rst \ >> Documentation/topics/porting.rst \ >> Documentation/topics/record-replay.rst \ >> diff --git a/Documentation/ref/ovsdb.7.rst b/Documentation/ref/ovsdb.7.rst >> index e4f1bf766..a5b8a9c33 100644 >> --- a/Documentation/ref/ovsdb.7.rst >> +++ b/Documentation/ref/ovsdb.7.rst >> @@ -121,13 +121,14 @@ schema checksum from a schema or database file, >> respectively. >> Service Models >> ============== >> >> -OVSDB supports three service models for databases: **standalone**, >> -**active-backup**, and **clustered**. The service models provide different >> -compromises among consistency, availability, and partition tolerance. They >> -also differ in the number of servers required and in terms of performance. >> The >> -standalone and active-backup database service models share one on-disk >> format, >> -and clustered databases use a different format, but the OVSDB programs work >> -with both formats. ``ovsdb(5)`` documents these file formats. >> +OVSDB supports four service models for databases: **standalone**, >> +**active-backup**, **relay** and **clustered**. The service models provide >> +different compromises among consistency, availability, and partition >> tolerance. >> +They also differ in the number of servers required and in terms of >> performance. >> +The standalone and active-backup database service models share one on-disk >> +format, and clustered databases use a different format, but the OVSDB >> programs >> +work with both formats. ``ovsdb(5)`` documents these file formats. Relay >> +databases has no on-disk storage. > > s/has/have
OK. > >> >> RFC 7047, which specifies the OVSDB protocol, does not mandate or specify >> any particular service model. >> @@ -406,6 +407,50 @@ following consequences: >> that the client previously read. The OVSDB client library in Open vSwitch >> uses this feature to avoid servers with stale data. >> >> +Relay Service Model >> +------------------- >> + >> +A **relay** database is a way to scale out read-mostly access to the >> +existing database working in any service model including relay. >> + >> +Relay database creates and maintains an OVSDB connection with other OVSDB >> +server. It uses this connection to maintain in-memory copy of the remote >> +database (a.k.a. the ``relay source``) keeping the copy up-to-date as the >> +database content changes on relay source in the real time. >> + >> +The purpose of relay server is to scale out the number of database clients. >> +Read-only transactions and monitor requests are fully handled by the relay >> +server itself. For the transactions that requests database modifications, > > s/requests/request OK. > >> +relay works as a proxy between the client and the relay source, i.e. it >> +forwards transactions and replies between them. >> + >> +Compared to a clustered and active-backup models, relay service model >> provides > > s/Compared to a/Compared to the OK. > >> +read and write access to the database similarly to a clustered database (and >> +even more scalable), but with generally insignificant performance overhead >> of > > Joke: citation needed :) > >> +an active-backup model. At the same time it doesn't increase availability >> that >> +needs to be covered by the service model of the relay source. >> + >> +Relay database has no on-disk storage and therefore cannot be converted to >> +any other service model. >> + >> +If there is already a database started in any service model, to start a >> relay >> +database server use ``ovsdb-server relay:<DB_NAME>:<relay source>``, where >> +``<DB_NAME>`` is the database name as specified in the schema of the >> database >> +that existing server runs, and ``<relay source>`` is an OVSDB connection >> method >> +(see `Connection Methods`_ below) that connects to the existing database >> +server. ``<relay source>`` could contain a comma-separated list of >> connection >> +methods, e.g. to connect to any server of the clustered database. >> +Multiple relay servers could be started for the same relay source. >> + >> +Since the way how relay handles read and write transactions is very similar >> +to the clustered model where "cluster" means "set or relay servers connected >> +to the same relay source", "follower" means "relay server" and the "leader" >> +means "relay source", same consistency consequences as for the clustered >> +model applies to relay as well (See `Understanding Cluster Consistency`_ >> +above). >> + >> +Open vSwitch 2.16 introduced support for relay service model. >> + >> Database Replication >> ==================== >> >> @@ -414,7 +459,8 @@ Replication, in this context, means to make, and keep >> up-to-date, a read-only >> copy of the contents of a database (the ``replica``). One use of >> replication >> is to keep an up-to-date backup of a database. A replica used solely for >> backup would not need to support clients of its own. A set of replicas >> that do >> -serve clients could be used to scale out read access to the primary >> database. >> +serve clients could be used to scale out read access to the primary >> database, >> +however `Relay Service Model`_ is more suitable for that purpose. >> >> A database replica is set up in the same way as a backup server in an >> active-backup pair, with the difference that the replica is never promoted >> to >> diff --git a/Documentation/topics/index.rst b/Documentation/topics/index.rst >> index 0036567eb..d8ccbd757 100644 >> --- a/Documentation/topics/index.rst >> +++ b/Documentation/topics/index.rst >> @@ -44,6 +44,7 @@ OVS >> openflow >> bonding >> networking-namespaces >> + ovsdb-relay >> ovsdb-replication >> dpdk/index >> windows >> diff --git a/Documentation/topics/ovsdb-relay.rst >> b/Documentation/topics/ovsdb-relay.rst >> new file mode 100644 >> index 000000000..40d294c55 >> --- /dev/null >> +++ b/Documentation/topics/ovsdb-relay.rst >> @@ -0,0 +1,124 @@ >> +.. >> + Copyright 2021, Red Hat, Inc. >> + >> + Licensed under the Apache License, Version 2.0 (the "License"); you >> may >> + not use this file except in compliance with the License. You may >> obtain >> + a copy of the License at >> + >> + http://www.apache.org/licenses/LICENSE-2.0 >> + >> + Unless required by applicable law or agreed to in writing, software >> + distributed under the License is distributed on an "AS IS" BASIS, >> WITHOUT >> + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See >> the >> + License for the specific language governing permissions and >> limitations >> + under the License. >> + >> + Convention for heading levels in Open vSwitch documentation: >> + >> + ======= Heading 0 (reserved for the title in a document) >> + ------- Heading 1 >> + ~~~~~~~ Heading 2 >> + +++++++ Heading 3 >> + ''''''' Heading 4 >> + >> + Avoid deeper levels because they do not render well. >> + >> +=============================== >> +Scaling OVSDB Access With Relay >> +=============================== >> + >> +Open vSwitch 2.16 introduced support for OVSDB Relay mode with the goal to >> +increase database scalability for a big deployments. Mainly, OVN (Open >> Virtual >> +Network) Southbound Database deployments. This document describes the main >> +concept and provides the configuration examples. >> + >> +What is OVSDB Relay? >> +-------------------- >> + >> +Relay is a database service model in which one ``ovsdb-server`` (``relay``) >> +connects to another standalone or clustered database server >> +(``relay source``) and maintains in-memory copy of its data, receiving >> +all the updates via this OVSDB connection. Relay server handles all the >> +read-only requests (monitors and transactions) on its own and forwards all >> the >> +transactions that requires database modifications to the relay source. >> + >> +Why is this needed? >> +------------------- >> + >> +Some OVN deployment could have hundreds or even thousands nodes, on each of >> +these nodes there is an ovn-controller, which is connected to the >> +OVN_Southbound database that is served by a standalone or clustered OVSDB. >> +Standalone database is handled by a single ovsdb-server process and >> clustered >> +could consist of 3 to 5 ovsdb-server processes. For the clustered database, >> +higher number of servers may significantly increase transaction latency due >> +to necessity for these servers to reach consensus. So, in the end limited >> +number of ovsdb-server processes serves ever growing number of clients and >> this >> +leads to performance issues. >> + >> +Read-only access could be scaled up with OVSDB replication on top of >> +active-backup service model, but ovn-controller is a read-mostly client, not >> +a read-only, i.e. it needs to execute write transactions from time to time. >> +Here relay service model comes into play. >> + >> +2-Tier Deployment >> +----------------- >> + >> +Solution for the scaling issue could look like a 2-tier deployment, where >> +a set of relay servers is connected to the main database cluster >> +(OVN_Southbound) and clients (ovn-conrtoller) connected to these relay >> +servers:: >> + >> + 172.16.0.1 >> + +--------------------+ +----+ ovsdb-relay-1 +--+---+ client-1 >> + | | | | >> + | Clustered | | +---+ client-2 >> + | Database | | ... >> + | | | +---+ client-N >> + | 10.0.0.2 | | >> + | ovsdb-server-2 | | 172.16.0.2 >> + | + + | +----+ ovsdb-relay-2 +--+---+ client-N+1 >> + | | | | | | >> + | | + +---+ +---+ client-N+2 >> + | | 10.0.0.1 | | ... >> + | | ovsdb-server-1 | | +---+ client-2N >> + | | + | | >> + | | | | | >> + | + + | + ... ... ... ... ... >> + | ovsdb-server-3 | | >> + | 10.0.0.3 | | +---+ client-KN-1 >> + | | | 172.16.0.K | >> + +--------------------+ +----+ ovsdb-relay-K +--+---+ client-KN >> + >> +In practice, the picture might look a bit more complex, because all relay >> +servers might connect to any member of a main cluster and clients might >> +connect to any relay server of their choice. >> + >> +Assuming that servers of a main cluster started like this:: >> + >> + $ ovsdb-server --remote=ptcp:10.0.0.1:6642 ovn-sb-1.db >> + >> +The same for other two servers. In this case relay servers could be >> +started like this:: >> + >> + $ REMOTES=tcp:10.0.0.1:6642,tcp:10.0.0.2:6642,tcp:10.0.0.3:6642 >> + $ ovsdb-server --remote=ptcp:172.16.0.1:6642 relay:OVN_Southbound:$REMOTES >> + $ ... >> + $ ovsdb-server --remote=ptcp:172.16.0.K:6642 relay:OVN_Southbound:$REMOTES I also fixed the way passive connections configured (port should go before the ip). >> + >> +Every relay server could connect to any of the cluster members of their >> choice, >> +fairness of load distribution is achieved by shuffling remotes. >> + >> +For the actual clients, they could be configured to connect to any of the >> +relay servers. For ovn-controllers the configuration could look like this:: >> + >> + $ REMOTES=tcp:172.16.0.1:6642,...,tcp:172.16.0.K:6642 >> + $ ovs-vsctl set Open_vSwitch . external-ids:ovn-remote=$REMOTES >> + >> +Setup like this allows the system to serve ``K * N`` clients while having >> only >> +``K`` actual connections on the main clustered database keeping it in a >> +stable state. >> + >> +It's also possible to create multi-tier deployments by connecting one set >> +of relay servers to another (smaller) set of relay servers, or even create >> +tree-like structures by the cost of increased latency for write >> transactions, > > s/by/with OK. > >> +because they will be forwarded multiple times. >> diff --git a/NEWS b/NEWS >> index ebba17b22..391b0abba 100644 >> --- a/NEWS >> +++ b/NEWS >> @@ -1,6 +1,9 @@ >> Post-v2.15.0 >> --------------------- >> - OVSDB: >> + * Introduced new database service model - "relay". Targeted to scale >> out >> + read-mostly access (ovn-controller) to existing databases. >> + For more information: ovsdb(7) and >> Documentation/topics/ovsdb-relay.rst >> * New command line options --record/--replay for ovsdb-server and >> ovsdb-client to record and replay all the incoming transactions, >> monitors, etc. More datails in >> Documentation/topics/record-replay.rst. >> diff --git a/ovsdb/ovsdb-server.1.in b/ovsdb/ovsdb-server.1.in >> index fdd52e8f6..dac0f02cb 100644 >> --- a/ovsdb/ovsdb-server.1.in >> +++ b/ovsdb/ovsdb-server.1.in >> @@ -10,6 +10,7 @@ ovsdb\-server \- Open vSwitch database server >> .SH SYNOPSIS >> \fBovsdb\-server\fR >> [\fIdatabase\fR]\&... >> +[\fIrelay:schema_name:remote\fR]\&... >> [\fB\-\-remote=\fIremote\fR]\&... >> [\fB\-\-run=\fIcommand\fR] >> .so lib/daemon-syn.man >> @@ -35,12 +36,15 @@ For an introduction to OVSDB and its implementation in >> Open vSwitch, >> see \fBovsdb\fR(7). >> .PP >> Each OVSDB file may be specified on the command line as \fIdatabase\fR. >> -If none is specified, the default is \fB@DBDIR@/conf.db\fR. The database >> -files must already have been created and initialized using, for >> -example, \fBovsdb\-tool\fR's \fBcreate\fR, \fBcreate\-cluster\fR, or >> -\fBjoin\-cluster\fR command. >> +Relay databases may be specified on the command line as >> +\fIrelay:schema_name:remote\fR. For a detailed description of relay >> database >> +argument, see \fBovsdb\fR(7). >> +If none of database files or relay databases is specified, the default is >> +\fB@DBDIR@/conf.db\fR. The database files must already have been created >> and >> +initialized using, for example, \fBovsdb\-tool\fR's \fBcreate\fR, >> +\fBcreate\-cluster\fR, or \fBjoin\-cluster\fR command. >> .PP >> -This OVSDB implementation supports standalone, active-backup, and >> +This OVSDB implementation supports standalone, active-backup, relay and >> clustered database service models, as well as database replication. >> See the Service Models section of \fBovsdb\fR(7) for more information. >> .PP >> @@ -50,7 +54,9 @@ successfully join a cluster (if the database file is >> freshly created >> with \fBovsdb\-tool join\-cluster\fR) or connect to a cluster that it >> has already joined. Use \fBovsdb\-client wait\fR (see >> \fBovsdb\-client\fR(1)) to wait until the server has successfully >> -joined and connected to a cluster. >> +joined and connected to a cluster. The same is true for relay databases. >> +Same commands could be used to wait for a relay database to connect to >> +the relay source (remote). >> .PP >> In addition to user-specified databases, \fBovsdb\-server\fR version >> 2.9 and later also always hosts a built-in database named >> @@ -243,10 +249,11 @@ not list remotes added indirectly because they were >> read from the >> database by configuring a >> \fBdb:\fIdb\fB,\fItable\fB,\fIcolumn\fR remote. >> . >> -.IP "\fBovsdb\-server/add\-db \fIdatabase\fR" >> -Adds the \fIdatabase\fR to the running \fBovsdb\-server\fR. The database >> -file must already have been created and initialized using, for example, >> -\fBovsdb\-tool create\fR. >> +.IP "\fBovsdb\-server/add\-db \fIdatabase\fR >> +Adds the \fIdatabase\fR to the running \fBovsdb\-server\fR. \fIdatabase\fR >> +could be a database file or a relay description in the following format: >> +\fIrelay:schema_name:remote\fR. The database file must already have been >> +created and initialized using, for example, \fBovsdb\-tool create\fR. >> . >> .IP "\fBovsdb\-server/remove\-db \fIdatabase\fR" >> Removes \fIdatabase\fR from the running \fBovsdb\-server\fR. \fIdatabase\fR >> > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
