So I've heard from both hosting providers that it is fine, but also managed to take both of their shared services "down" with only about ~100 users (200 continuous filtered replications). I'm only now at the point where I have tooling to build out arbitrary large tests on my local machine to see the stats for myself, but as I understand it the issue is that every replication needs at least one couchjs process to do its filtering for it.
So rather than inactive users mostly just taking up disk space, they're instead costing a full-fledged process worth of memory and system resources, each, all the time. As I understand it, this isn't much better on BigCouch either since the data is scattered ± evenly on each machine, so while the *computation* is spread, each node in the cluster still needs k*numberOfUsers couchjs processes running. So it's "scalable" in the sense that traditional databases are scalable: vertically, by buying machines with more and more memory. Again, I am still working on getting a better feel for the costs involved, but the basic theory with a master-to-N hub is not a great start: every change needs to be processed by every N replications. So if a user writes a document that ends up in the master database, every other user's filter function needs to process that change coming out of master. Even when N users are generating 0 (instead of M) changes, it's not doing M*N work but there's still always 2*N open connections and supporting processes providing a nasty baseline for large values of N. After another day considering my options, I see two (complementary) ways to mitigate this more simply: 1. Have developers expect the need to scale out via intermediary databases with intermediary filters, arrangement tailored manually for the application at hand. This burns more storage space, but is consistent with CouchDB's "disks are cheap" philosophy. Assuming it's possible to combine replication filters for a given application, this should make it more like M*log(N) cost. 2. Implement a lesser (more "eventual") form of "continuous" replication — replications request a priority and let CouchDB "continuously" replicate without actually holding every database open (catch up every few seconds/minutes/hours as available resources permit). This would lower the baseline resources required. Both of these currently require extra tooling and infrastructure not easily handled from a design document. I didn't expect to need a full set of maintenance scripts and sharding strategies at a hundred not-very-active users — since continuous filtered replication is a fundamental pattern for communicating databases, it'd be nice if it were scalable, or at least cheaper. thanks, -natevw On Jan 29, 2013, at 4:55 PM, Stephen Bartell wrote: > Nathan, I'm actually in the process of setting up a multi-tenant environment > the canonical way, like you have. > > I've seen the replication overhead get pretty intense, but I figure that > scaling out to several couches is the way to go once the overhead becomes > unbearable. Actually I was hoping BigCouch would eventually be the answer. > > Why is this not the case for you? > > In one of those links you provided (JasonSmith@stackoverflow) said that db > per user is the only scalable way. It would be nice if he or someone here > could weight in on why/how thats the only scalable way. Especially in light > of Nathan claiming the exact opposite. > > sb > > On Jan 29, 2013, at 10:44 AM, Nathan Vander Wilt <[email protected]> > wrote: > >> # The problem >> >> It's a fairly common "complaint" that CouchDB's database model does not >> support fine-grained control over reads. The canonical solution is a >> database per user: >> http://wiki.apache.org/couchdb/PerDocumentAuthorization#Database_per_user >> http://stackoverflow.com/a/4731514/179583 >> >> This does not scale. >> >> 1. It complicates formerly simple backup/redundancy: now I need to make sure >> N replications stay working, N databases have correct permissions, instead >> of just one "main" database. Okay, write some scripts, deploy some cronjobs, >> can be made to work... >> >> 2. ...however, if data needs to be shared between users, this model >> *completely falls apart*. Bi-directional continuous filtered replication >> between a "hub" and each user database is extremely resource intensive. >> >> I naïvely followed the Best Practices and ended up with a system that can >> barely support 100 users to a machine due to replication overhead. Now if I >> want to continue doing it "The Right Way" I need to cobble together some >> sort of rolling replication hack at best. >> >> It's apparent the real answer for CouchDB security, right now, is to hide >> the database underneath some middleware boilerplate crap running as DB root. >> This is a well-explored pattern, by which I mean the database ends up with >> as many entry points as a sewer system has grates. >> >> >> # An improvement? >> >> What if CouchDB let you define virtual databases, that shared the underlying >> document data when possible, that updated incrementally (when queried) >> rather than continuously, that could even internally be implemented in a >> fanout fashion? >> >> - virtual databases would basically be part of the internal b-tree key >> hierarchy, sort of like multiple root nodes sharing the branches as much as >> possible >> - sharing the underlying document data would almost halve the amount of disk >> needed versus a "master" database storing all the data which is then copied >> to each user >> - updating incrementally would put less continuous memory pressure on the >> system >> - haven't actually done the maths, so I may be missing something, but >> wouldn't fanning out changes internally from a master database through >> intermediate partitions reduce the processing load? >> >> Basically, rather than each time a user updates a document, copying it to a >> master database, then filtering every M updates through N instances of >> couchjs; instead internally CouchDB could build a tree of combined filters — >> say, master database filters to log(N) hidden partitions at the first level >> and accepted changes would trickle through only relevant further layers. (In >> a way, this is kind of at odds with the incremental nature — maybe it does >> make sense to pay an amortized cost on write rather than on reads.) >> >> >> # The urgency >> >> Maybe this *particular* solution isn't really a solution, but we need one: >> >> If replicating amongst per-user databases is the only correct way to >> implement document-level read permissions, CouchDB **NEEDS** built-in >> support for a scalable way of doing so. >> >> There are plenty of other feature requests I could troll the list with >> regarding CouchApps. But this one is key; everything else I've been able to >> work around behind a little reverse proxy here and in front of an external >> process there. Without scalable read-level security, I see no particular >> raison d'être for Apache CouchDB — if CouchDB can't support direct HTTP >> access in production in general, then it's just another centralized database. >> >> >> thanks, >> -natevw >
