On Sat, Mar 7, 2009 at 10:18 AM, Adam Kocoloski <[email protected]> wrote: > Hi again, I put the new replicator through its paces over the past few days > and am pretty happy with it. Summary of the changes: > > http://github.com/kocolosk/couchdb/commits/otpify-replication > > * Only one copy of a given source->target replication can run at any given > time. Other POSTs to _replicate with the same request will be added as > listeners to the ongoing replication. When that replication finishes, we > respond to the original requester and check for new updates to include in > the response to the others. This is Option 3 in the previous thread on dev@ > > * Replications that terminate with an abnormal reason (other than shutdown) > will be restarted by the supervisor, but the listeners on the original > request are just given an {error, reason} response. They can track the > results of the retry by re-POSTing. > > * Attachments are streamed directly to disk in the case of a pull > replication. This feature was made possible by some bleeding-edge updates > to ibrowse courtesy of Chandru Mullaparthi. Is still requires quite a lot > of memory in the ibrowse processes, but we're working on that. The memory > footprint in the CouchDB processes is negligible. Testing between two EC2 > nodes indicated that pulling 400MB of attachments was approximately 20x > faster than pushing them (push still inlines in the JSON). > > * Memory utilization has decreased. All the tests posted by Jeff Hinrichs > in COUCHDB-270 pass, although the tests with 20MB JSON documents still use > over a gig of memory in ibrowse processes. > > * The parallel asynchronous GET requests we used in pull replications have > been temporarily rolled back out. I have a branch that includes them, but I > want to make sure there are no regressions in the memory utilization before > merging that feature back in. As a result, you'll find that push > replications will be significantly faster than pulls if there are no large > attachments hanging around. > > * Replication updates show up in the Futon status window. > > Side note: I redefined couch_util:should_flush() so that it considers both > the process' memory usage as well as any associated reference-counted > binaries in calculating its response. The code to find those binaries > relies on a part of process_info() that "may be changed or removed without > prior notice" and isn't terribly well documented, but we ignore binary > memory usage at our peril. With the old definition should_flush() would > come back false even when the replicator had grabbed 2GB worth of binaries > off the disk! > > If there are no objections I'd like to add this code to trunk later today. > Best, > > Adam > >
+∞
