[ https://issues.apache.org/jira/browse/COUCHDB-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983014#comment-15983014 ]
ASF subversion and git services commented on COUCHDB-3324: ---------------------------------------------------------- Commit d6a3e05a3c3de207117eb05d7f45dbc7fcc8fb55 in couchdb's branch refs/heads/63012-scheduler from [~vatamane] [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=d6a3e05 ] Refactor utils into 3 modules Over the years utils accumulated a lot of functionality. Clean up a bit by separating it into specific modules according to semantics: - couch_replicator_docs : Handle read and writing to replicator dbs. It includes updating state fields, parsing options from documents, and making sure replication VDU design document is in sync. - couch_replicator_filters : Fetch and manipulate replication filters. - couch_replicator_ids : Calculate replication IDs. Handles versioning and Pretty formatting of IDs. Filtered replications using user filter functions incorporate a filter code hash into the calculation, in that case call couch_replicator_filters module to fetch the filter from the source. Jira: COUCHDB-3324 > Scheduling Replicator > --------------------- > > Key: COUCHDB-3324 > URL: https://issues.apache.org/jira/browse/COUCHDB-3324 > Project: CouchDB > Issue Type: New Feature > Reporter: Nick Vatamaniuc > > Improve CouchDB replicator > * Allow running a large number of replication jobs > * Improve API with a focus on ease of use and performance. Avoid updating > replication document with transient state updates. Instead create a proper > API for querying replication states. At the same time provide a compatibility > mode to let users keep existing behavior (of getting updates in documents). > * Improve network resource usage and performance. Multiple connection to the > same cluster could share socket connections > * Handle rate limiting on target and source HTTP endpoints. Let replication > request auto-discover rate limit capacity based on a proven algorithm such as > Additive Increase / Multiplicative Decrease feedback control loop. > * Improve performance by avoiding repeatedly retrying failing replication > jobs. Instead use exponential backoff. > * Improve recovery from long (but temporary) network failure. Currently if > replications jobs fail to start 10 times in a row they will not be retried > anymore. This is not always desirable. In case of a long enough DNS (or other > network) failure replication jobs will effectively stop until they are > manually restarted. > * Better handling of filtered replications: Failing to fetch filters could > block couch replicator manager, lead to message queue backups and memory > exhaustion. Also, when replication filter code changes update replication > accordingly (replication job ID should change in that case) > * Provide better metrics to introspect replicator behavior. -- This message was sent by Atlassian JIRA (v6.3.15#6346)