Chris Anderson has done some work here with the batch feature for
similar efficiency reasons, except that his feature is to buffer the
docs into memory and return success to the client immediately, it's
for things where losing some messages is acceptable (like logging), so
long as the transactions are fast and efficient as possible. You
instead want the client to wait and get the success or fail. it seems
the batch work is where to build this into.
-Damien
On May 22, 2009, at 4:08 AM, Brian Candler wrote:
On Wed, May 20, 2009 at 11:48:10AM -0400, Damien Katz wrote:
As for timing of fsync: ideally what I would like is for each write
operation to return some sort of checkpoint tag (which could just be
the
current file size). Then have a new HTTP operation "wait for sync
<tag>"
which blocks until the file has been fsync'd at or after that
position.
Just use the x-couch-full-commit=true http header.
That's not what I want (unless I've misunderstood what this header
does).
Suppose I have 100 concurrent client connections, each client is
committing
one document at roughly the same time. I don't one 100 fsync's; I
want each
client to be able to write a doc, and then be notified when the next
convenient fsync takes place, so they know their doc is on persistent
storage.
Setting the default commit interval down to (say) 200ms would then
give
clients a reasonably low latency for getting their docs committed to
disk,
without killing performance by fsyncing every write.
Otherwise, the only solution I can see is a batching proxy: it takes
PUT and
POST requests, builds them into a single _bulk_docs, and submits that
periodically with x-couch-full-commit:true before responding. But
that gets
very awkward when dealing with multiple databases, and/or multiple
users
with different credentials.
Regards,
Brian.
