Hi again, I put the new replicator through its paces over the past few
days and am pretty happy with it. Summary of the changes:
http://github.com/kocolosk/couchdb/commits/otpify-replication
* Only one copy of a given source->target replication can run at any
given time. Other POSTs to _replicate with the same request will be
added as listeners to the ongoing replication. When that replication
finishes, we respond to the original requester and check for new
updates to include in the response to the others. This is Option 3 in
the previous thread on dev@
* Replications that terminate with an abnormal reason (other than
shutdown) will be restarted by the supervisor, but the listeners on
the original request are just given an {error, reason} response. They
can track the results of the retry by re-POSTing.
* Attachments are streamed directly to disk in the case of a pull
replication. This feature was made possible by some bleeding-edge
updates to ibrowse courtesy of Chandru Mullaparthi. Is still requires
quite a lot of memory in the ibrowse processes, but we're working on
that. The memory footprint in the CouchDB processes is negligible.
Testing between two EC2 nodes indicated that pulling 400MB of
attachments was approximately 20x faster than pushing them (push still
inlines in the JSON).
* Memory utilization has decreased. All the tests posted by Jeff
Hinrichs in COUCHDB-270 pass, although the tests with 20MB JSON
documents still use over a gig of memory in ibrowse processes.
* The parallel asynchronous GET requests we used in pull replications
have been temporarily rolled back out. I have a branch that includes
them, but I want to make sure there are no regressions in the memory
utilization before merging that feature back in. As a result, you'll
find that push replications will be significantly faster than pulls if
there are no large attachments hanging around.
* Replication updates show up in the Futon status window.
Side note: I redefined couch_util:should_flush() so that it considers
both the process' memory usage as well as any associated reference-
counted binaries in calculating its response. The code to find those
binaries relies on a part of process_info() that "may be changed or
removed without prior notice" and isn't terribly well documented, but
we ignore binary memory usage at our peril. With the old definition
should_flush() would come back false even when the replicator had
grabbed 2GB worth of binaries off the disk!
If there are no objections I'd like to add this code to trunk later
today. Best,
Adam
