Hey Nick, I run a server with several thousand databases. I found that replicating them continuously to a global DB for backup caused CPU usage to skyrocket. My current solution is to use rsync along with rsync.net to backup my database files via a scheduled cron job. Rsync is really efficient at only copying the differences in files. Rsync and cron, alternatively Rsync and a directory watching script, may meet your needs.
— Paul Okstad http://pokstad.com > On Mar 9, 2016, at 12:29 PM, Nick Wood <[email protected]> wrote: > > Hello, > > I'm looking to back up a CouchDB server with multiple databases. Currently > 1,400, but it fluctuates up and down throughout the day as new databases > are added and old ones deleted. ~10% of the databases are written to within > any 5 minute period of time. > > Goals > - Maintain a continual off-site snapshot of all databases, preferably no > older than a few seconds (or minutes) > - Be efficient with bandwidth (i.e. not copy the whole database file for > every backup run) > > My current solution watches the global _changes feed and fires up a > continuous replication to an off-site server whenever it sees a change. If > it doesn't see a change from a database for 10 minutes, it kills that > replication. This means I only have ~150 active replications running on > average at any given time. > > I thought this was a pretty clever approach, but I can't stabilize it. > Replications hang frequently with crashes in the log file. I haven't yet > tracked down the source of the crashes. I'm running the official 1.6.1 > docker image as of yesterday so I don't think it would be an erlang issue. > > Rather than keep banging my head against these stability issues, I thought > I'd ask to see if anyone else has come up with a clever backup solution > that meets the above goals? > > Nick
