flimzy commented on a change in pull request #285: Couchdb 3326 clustered purge URL: https://github.com/apache/couchdb-documentation/pull/285#discussion_r201071231
########## File path: src/cluster/purging.rst ########## @@ -0,0 +1,173 @@ +.. Licensed under the Apache License, Version 2.0 (the "License"); you may not +.. use this file except in compliance with the License. You may obtain a copy of +.. the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +.. WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +.. License for the specific language governing permissions and limitations under +.. the License. + +.. _cluster/purging: + +=============== +Clustered Purge +=============== +The primary purpose of clustered purge is to clean databases that have multiple +deleted tombstones or single documents that contain large numbers of conflicts. +But it can also be used to purge any document (deleted or non-deleted) with any +number of revisions. + +Clustered Purge is designed to maintain eventual consistency and prevent +unnecessary invalidation of secondary indexes. For this, every database keeps +track of a certain number of historical purges requested in the database, as +well as its current ``purge_seq``. Internal replications and secondary indexes +process database's purges and periodically update their corresponding purge +checkpoint documents to report ``purge_seq`` processed by them. To ensure +eventual consistency, the database will remove store historical purge requests +only after they have been processed by internal replication jobs and secondary +indexes. + +Internal Structures +==================================== +To enable internal replication of purge information between nodes and secondary +indexes, two internal purge trees were added to a database file to track +historical purges. + +.. code-block:: text + + purge_tree: UUID -> {PurgeSeq, DocId, Revs} + purge_seq_tree: PurgeSeq -> {UUID, DocId, Revs} + +Each interactive request to ``_purge API``, creates an ordered set of pairs on +increasing ``purge_seq`` and purge_request, where purge_request is a tuple that +contains docid and list of revisions. For each purge_request uuid is generated. +A purge request is added to internal purge trees: +a tuple ``{UUID -> {PurgeSeq, DocId, Revs}}`` is added to ``purge_tree``, +a tuple is ``{PurgeSeq -> {UUID, DocId, Revs}}`` added ``to purge_seq_tree``. + +Compaction of Purges +==================================== + +During the compaction of the database the oldest purge requests are to be +removed to store only ``purged_infos_limit`` number of purges in the database. +But in order to keep the database consistent with indexes and other replicas, +we can only remove purge requests that have already been processed by indexes +and internal replications jobs. Thus, occasionally purge trees may store +purges in the number greater than ``purged_infos_limit``. If the number of Review comment: I suggest rewording this sentence for clarity: > Thus, occasionally purge trees may store more than ``purged_infos_limit`` purges. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
