tonysun83 commented on a change in pull request #123: Documention for the
scheduling replicator
URL:
https://github.com/apache/couchdb-documentation/pull/123#discussion_r110234057
##########
File path: src/replication/replicator.rst
##########
@@ -155,98 +203,192 @@ the following order:
.. code-block:: javascript
{
- "_id": "doc_A",
- "source": "http://myserver.com:5984/foo";,
- "target": "http://user:pass@localhost:5984/bar";
+ "_id": "my_rep",
+ "source": "http://user:pass@localhost:5984/foo";,
+ "target": "http://user:pass@localhost:5984/bar";,
+ "create_target": true,
+ "continuous": true
}
and
.. code-block:: javascript
{
- "_id": "doc_B",
- "source": "http://myserver.com:5984/foo";,
- "target": "http://user:pass@localhost:5984/bar";
- }
-
-Both describe exactly the same replication (only their ``_ids`` differ). In
this
-case document ``doc_A`` triggers the replication, getting updated by CouchDB
-with the fields ``_replication_state``, ``_replication_state_time`` and
-``_replication_id``, just like it was described before. Document ``doc_B``
-however, is only updated with one field, the ``_replication_id`` so it will
-look like this:
-
-.. code-block:: javascript
-
- {
- "_id": "doc_B",
- "source": "http://myserver.com:5984/foo";,
+ "_id": "my_rep_dup",
+ "source": "http://user:pass@localhost:5984/foo";,
"target": "http://user:pass@localhost:5984/bar";,
- "_replication_id": "c0ebe9256695ff083347cbf95f93e280"
+ "create_target": true,
+ "continuous": true
}
-While document ``doc_A`` will look like this:
-
-.. code-block:: javascript
-
- {
- "_id": "doc_A",
- "source": "http://myserver.com:5984/foo";,
- "target": "http://user:pass@localhost:5984/bar";,
- "_replication_id": "c0ebe9256695ff083347cbf95f93e280",
- "_replication_state": "triggered",
- "_replication_state_time": 1297974122
- }
+Both describe exactly the same replication (only their ``_ids``
+differ). In this case document ``my_rep`` triggers the
+replication. While ``my_rep_dup``` will fail. Inspecting
+``_scheduler/docs`` explains exactly why it failed:
+
+.. code-block:: json
+
+ {
+ "database": "_replicator",
+ "doc_id": "my_rep_dup",
+ "error_count": 1,
+ "id": null,
+ "info": "Replication
`a81a78e822837e66df423d54279c15fe+continuous+create_target` specified by
document `my_rep_dup` already started, triggered by document `my_rep` from db
`_replicator`",
+ "last_updated": "2017-04-05T21:41:51Z",
+ "source": "http://adm:*****@localhost:5984/foo/";,
+ "start_time": "2017-04-05T21:41:51Z",
+ "state": "failed",
+ "target": "http://adm:*****@localhost:5984/bar/";
+ }
-Note that both document get exactly the same value for the ``_replication_id``
-field. This way you can identify which documents refer to the same replication
-
-you can for example define a view which maps replication IDs to document IDs.
+Notice the state for this replication is ``failed``. Unlike
+``crashing``, ``failed`` state is terminal. As long as both documents
+are present replicator will not retry to run this replication. Another
+reason for could be malformed documents. For example if worker process
+count is specified as a string (``"worker_processes": "a few"``)
+instead of an integer.
+
+Replication states
+==================
+
+Replication jobs during their life-cycle pass through various
+states. This is a diagram of all the states and transitions
+between them:
+
+.. figure:: ../../images/replication-state-diagram.svg
+ :align: center
+ :alt: Replication state diagram
+
+ Replication state diagram
+
+Green and red shapes represent replication job states.
+
+Yellow shapes represent external APIs, that's how users interact with
+the replicator. Writing documents to ``_replicator`` is the preferred
+way of creating replications, but posting to the ``_replicate`` HTTP
+endpoint is also supported.
+
+.. note:: Replications created through the ``_replicate`` endpoint
+ will not survive cluster node restart where they are running, and
+ once completed they will be removed from the system, so their
+ completion state is preserved in ``_scheduler/jobs`` endpoint.
+
+White shapes indicate internal API boundaries and point to how the
+replicator is structured internally. There are two stages in the
+processing: the first is where replication documents are parsed and
+become replication jobs, and the second is the scheduler. The
+scheduler runs replication jobs, periodically stopping and starting
+some. Jobs posted via the ``_replicate`` endpoint bypass the first
+component and go straight to the scheduler.
+
+States descriptions
+-------------------
+
+Before explaining the details of each state, it is worth noticing that
+color and shape of each state in the diagram:
+
+`Green` vs `red` partitions states into "healthy" and "unhealthy",
+respectively. Unhealthy states indicate something has gone wrong and
+it might need user's attention.
+
+`Rectangle` vs `oval` separates "terminal" states from "non-terminal"
+ones. Terminal states are those which will not transition to other
+states any more. Informally, jobs in a terminal state will not be
+retried and don't consume memory or CPU resources.
+
+ * ``Initializing``: Indicates replicator has noticed the change
+ from the replication document. Jobs should transition quickly
+ through this state. Being stuck here for a while could mean there
+ is an internal error.
+
+ * ``Failed``: Replication document could not be processed and turned
+ into a valid replication job for the scheduler. This state is
+ terminal and requires user intervention to fix the problem. Typical
+ reasons for ending up in this state is a malformed document. For
+ example specifying an integer for a parameter which accepts a
+ boolean. Another reason could be specifying a duplicate
+ replication. A duplicate replication is a replication with
+ identical parameters but a different document ID.
+
+ * ``Error``: Replication document update could not be turned into a
+ replication job. Unlike the ``Failed`` state, this one is
+ temporary, and replicator will keep retrying periodically. There is
+ an exponential backoff applied in case of consecutive failures.
+ The main reason this state exists is to handle filtered
+ replications with custom user functions. Filter function content is
+ needed in order to calculate the replication ID. A replication job
+ could not be created until the function code is retrieved. Because
+ retrieval happens over the network, temporary failures have to be
+ handled.
+
+ * ``Running``: Replication job is running normally. This means,
+ there might be a change feed open, and if changes are noticed, they
+ would be processed and posted to the target. Job is still
+ considered ``Running`` even if its workers are currently not
+ streaming changes from source to target and are just waiting on the
+ change feed.
+
+ * ``Pending``: Replication job is not running and is waiting its
+ turn. This state is reached then more than ``replicator.max_jobs``
+ replication have been added to the scheduler. In that case
+ scheduler will periodically stop some and start other jobs, trying
+ to give each one a fair chance at making progress.
+
+ * ``Crashing``: Replication job has been successfully added to the
+ replication scheduler. However an error was encountered during the
+ last run. Error could be a network failure, a missing source
+ database, a permissions error, etc. Repeated consecutive crashes
+ result in an exponential backoff. This state is considered temporary
+ (non-terminal) and replication jobs will be periodically
+ retried. Maximum backoff interval is around a day or so.
+
+ * ``Completed``: This is a terminal, successful state for
+ non-continuous replications. Once in this state the replication is
+ "forgotten" by the scheduler and it doesn't consume any more CPU or
+ memory resorces. Continuous replication jobs will never reach this
+ state.
+
+Compatibility Mode
+==================
+
+Previous version of CouchDB replicator wrote state updates back to
+replication documents. In case were user code programmatically read
+those states, there is compatibility mode enabled via
+``replicator.update_docs = true`` configuration setting. In this mode
+replicator will continue to write state updates to the
+documents.
Canceling replications
======================
-To cancel a replication simply ``DELETE`` the document which triggered the
-replication. The Couch log will show you an entry like the following:
-
-.. code-block:: text
-
- [Thu, 17 Feb 2011 20:16:29 GMT] [info] [<0.125.0>] Stopped replication
`c0ebe9256695ff083347cbf95f93e280+continuous+create_target` because replication
document `doc_A` was deleted
-
-.. note::
- You need to ``DELETE`` the document that triggered the replication.
- ``DELETE``-ing another document that describes the same replication
- but did not trigger it, will not cancel the replication.
+To cancel a replication simply ``DELETE`` the document which triggered
+the replication. To update a replication, for example, change the
+number of worker or the source, simply update the document with new
+data. If there is extra application-specific data in the replication
+documents, that data is ignored by the replicator.
Server restart
==============
-When CouchDB is restarted, it checks its ``_replicator`` database and
-restarts any replication that is described by a document that either has
-its ``_replication_state`` field set to ``triggered`` or it doesn't have
-yet the ``_replication_state`` field set.
-
-.. note::
- Continuous replications always have a ``_replication_state`` field
- with the value ``triggered``, therefore they're always restarted
- when CouchDB is restarted.
-
-Updating Documents in the Replicator Database
-=============================================
+When CouchDB is restarted, it checks its ``_replicator`` databases and
+restarts replications described by documents if they are not already in
+in a ``completed`` or ``failed`` state. If they are, they are ignored.
-Once the replicator has started work on a job defined in the ``_replicator``
-database, modifying the replication document is no longer allowed. Attempting
-to do this will result in the following response
+Clustering
+==========
-.. code-block:: javascript
-
- {
- "error": "forbidden",
- "reason": "Only the replicator can edit replication documents that are
in the triggered state."
- }
+In a cluster, replication jobs are balanced evenly among all the nodes
+nodes such that a replication job runs on only one node at a time.
-The way to accomplish this is to first delete the old version and then insert
-the new one.
+Every time there is a cluster membership change, that is when nodes
+are added or removed, as it happens in a rolling reboot, replicator
+application will notice the change, rescan all the document and
+running replication, and re-evaluate their cluster placement in light
+of the new set of live nodes. This mechanism also provides
+replication fail-over in case on node fails. Replication jobs started
+form replication documents (but not those started from a ``_replicate``
Review comment:
from
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
