kocolosk commented on a change in pull request #651: URL: https://github.com/apache/couchdb-documentation/pull/651#discussion_r609929193
########## File path: rfcs/018-sharded-changes-feeds.md ########## @@ -0,0 +1,238 @@ +--- +name: Formal RFC +about: Submit a formal Request For Comments for consideration by the team. +title: 'High Throughput Parallel _changes Feed' +labels: rfc, discussion +assignees: '' + +--- + +# Introduction + +This proposal is designed to improve indexing throughput, reduce hot spots for +write-intensive workloads, and offer a horizontally-scalable API for consumers +to process the change capture feed for an individual database in CouchDB 4.0. + +## Abstract + +The current implementation on `main` writes all changes feed entries for a given +database into a single `?DB_CHANGES` subspace in FoundationDB. The view indexing +system (c.f. [RFC 008](008-map-indexes.md#index-building)) uses a single worker +for each design document that processes all the entries for that changes feed. +High throughput writers can overwhelm that indexer and ensure that it will never +bring the view up-to-date. The previous RFC mentions parallelizing the build as +a future optimization. Well, here we are. + +The parallelization technique proposed herein shards the changes feed itself +into multiple subspaces. This reduces the write load on any single underlying +FoundationDB storage server. We also introduce a new external API for accessing +these individual shards directly to ensure that consumers can scale out to keep +up with write-intensive workloads without needing to build their own system to +farm out changes from a single feed to multiple workers. + +Shard counts on a database can vary over time as needed, but previous entries +are not re-sharded. We sketch how an indexer can process the individual sharded +feeds in parallel without sacrificing the isolation semantics of the secondary +index (i.e., that it observes the state of the underlying database as it existed +as some specific sequence). Sequence numbers are globally unique and totally +ordered across shards. + +## Requirements Language + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", +"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this +document are to be interpreted as described in +[RFC 2119](https://www.rfc-editor.org/rfc/rfc2119.txt). + +## Terminology + +**changes shard**: a subspace in FoundationDB into which some portion of the +changes feed entries for that database are written. It is not directly related +to the underlying storage server shards in FoundationDB itself. + +--- + +# Detailed Description + +## Data Model + +The existing `?DB_CHANGES` subspace will be deprecated (i.e. renamed as +`?DB_CHANGES_DEPRECATED`) and a new `?DB_CHANGES` subspace will be created. This +subspace will contain an additional nested level with the individual shard +identifiers. Within each shard the data model is unchanged from before. + +## Routing + +Documents will be routed to shards using a configurable hashing scheme. The +default scheme will use consistent hashing on the partition key, so that a) all +updates to a given document will land in the same shard, and b) documents from +the same partition in a partitioned database will also be colocated. This +simplifies matters for a consumer processing the individual shard feeds in +parallel, as it can ignore the possibility of observing out-of-order updates to +the same document from different shards, and it furthermore allows the +computation of per-partition statistics (e.g. windowing functions over meter +readings in the canonical IoT device use case for partitions). + +## Resharding + +The shard count for a database can change over time. When the shard count +changes, a new set of `ShardIds` in the `?DB_CHANGES` subspace is created, and +all future updates to that database will be routed to those new subspaces. +Consumers of the shard-level API will receive a notification that a resharding +event has occurred once they reach the end of the updates committed to the +previous subspace. They MUST re-connect to the new endpoints once they receive Review comment: Yes that's correct -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
