replication performance improvements
------------------------------------
Key: COUCHDB-160
URL: https://issues.apache.org/jira/browse/COUCHDB-160
Project: CouchDB
Issue Type: Improvement
Components: Database Core
Affects Versions: 0.9
Reporter: Adam Kocoloski
Priority: Minor
I wrote some code to speed up CouchDB's replication process by parallelizing
document requests and using _bulk_docs to write changes to the target. I
tested the speedup as follows:
* 1000 document DB, 1022 update_seq, ~450 KB after compaction
* local and remote machines have ~45 ms latency
* timed requests using timer:tc(couch_rep, replicate, [<<"source">>,
<<"target">>]
* all replications are "from scratch"
trunk:
local-local 115
local-remote 145
remote-remote 173
remote-local 146
db size after replication: 1.8 MB
patch:
local-local 1.83
local-remote 38
remote-remote 64
remote-local 35
db size after replication: 453 KB
I'll attach the patch as an update to this issue. It might be worth exposing
the "batch size" (currently 100 docs) as a configurable parameter. Comments
welcome. Best,
Adam
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
