Hey couchers, For the past week, I've been playing with the thought of implementing a sharding orchestrator for CouchDB instances. My use case is that I have a lot of data/documents that can't be stored on a single machine and I'd like to be able to query the data with fairly low latencies. Oh, there's a big write throughput constantly, too.
Querying could be done by rereducing the view results coming in from the various shards. AFAIK, this is how BigCouch does it too. Now, the rebalancing act of all this is pretty straight forward; each shard is given a generated view that displays all documents that don't belong there in the shard (and thus need to be transferred). The orchestrator will then periodically check if any documents should be transferred to another server, replicate those documents and purge* them on the source server. * Purging since I am not interested in keeping them on disk on the source database anymore, not even tombstones. All the document data should only reside on the new shard. My issue with the described solution is that there will be a moment when views will be showing incorrect information. As I see it, there are two problems that needs to be solved with this solution: 1. Purging of documents: According to the purge documentation<http://docs.couchdb.org/en/latest/api/database/misc.html#post--db-_purge>, purging documents will require rebuilding entire views. That ain't good. 2. Inconsistent views; While transferring documents from one shard to another there will be a moment when a document resides on two databases. 1) could possibly be solved by introducing an replica database of the source database so that views could be recreated offline. However, this makes replication way more tedious than I initially expected. 2) could be solved by simply locking view reads until the inconsistency is fixed. But hey, we're not bug fans of that kind of global locking. Possibly patching all map-functions in the views would also fix the issue of double documents. At the same time, that feels kind of like a hack. Now, I know BigCouch is around the corner. How does it cope with the above issues? Maybe it's introduced some new magic for this that currently is not in CouchDB? I've been trying to Google this, but so far haven't found too much. Feel free to point me in the right direction if you know of any material. Cheers, Jens
