Hi Bob, The replicator will dynamically choose _bulk_docs batch sizes based on the number of documents that are ready to be transmitted to the source. It’s possible to set an upper bound on the size of the batch, but at this time it’s not possible to set a lower bound.
It sounds like what’s happening here is that the replicator is faster than the filter function, and that it’s constantly waiting for the next document to pass the filter. One sanity check you might try is to request the filtered _changes feed directly and see what kind of throughput you get. The filtered _changes feed sets the upper bound on the replication throughput you can achieve with that filter. If the _changes feed is fast but the replication is slow, the next thing you should try to do is minimize the replication-related resource consumption on the source — e.g., is the replication mediated by the server hosting the source database, and if so do you have an opportunity to mediate the replication on a different server? Cheers, Adam > On Jul 24, 2015, at 10:50 PM, Bob Hathaway <[email protected]> wrote: > > Our couchdb replication runs fast without filters to a couch instance. But > from that > couch instance to another a filter is doing a simple check and sync'ing about > 10x slower. > Looking at couch debug logging, the bulk_docs Content-length is 10x smaller > with the filter. > With filter the _bulk_docs rate per minute appears identical to the number of > docs sync'd. > Without the filter, the _bulk_docs rate per minute is 10x less than docs > sync'd. > > It would appear the couch replication protocol groups 10 docs in a _bulk_docs > POST without a replication filter > but with the filter there is no grouping and bulk_docs appears to only > contain a single doc. > > Does couchdb filter replication not group docs and send 1 doc in each > _bulk_docs call to the target host? > > Is there some configuration which would allow the filter replication to group > docs to speed up replication? > > -- > Robert Hathaway > President & Chief Software Architect > SOA Object Systems, LLC > office: 201-408-5828 > cell: 201-390-7602 > email: [email protected] >
