Dear users, Is there any way to move database files physically without shutting-down whole couchdb engine?
My environment: Couchdb v1.2.0, Ubuntu server 12.04, >300 couch databases which totally consume more than 300GB of space. Largest database has size 25GB, about twenty databases of size 10-25GB, about twenty 5-10GB. All other are 1GB or less. Every database serves one customer. My task: Currently I am running out of free space on partition which hosts database_dir, and looking for way to move databases to another physical location (which is already connected and ready to use). I need to move if not all DBs - then at least part of them, to free some space for databases which will left at their current place. But I Can't stop couchdb for this maintenance more than for 15 minutes due to SLAs. 15 minutes is not enough to copy 300Gigs :(, that is why I am asking for any known method to move databases physically. What I already tried: * Rsync files from old location to new, to make copy as close to original as possible. Then stop engine for shorter time, rsync again and start with new database_dir. But rsync makes no sense, because files to large and continuously changing, so every run of rsync is almost equal to copy. * Replicate databases inside one couchdb instance: if we have /oldlocation/db1.couch - then I create symlink `ln -s /newlocation/db1.couch /oldlocation/localreplica_db1.couch`, and then PUT to _replicator database with request to replicate from db1 to localreplica_db1. My idea was to keep two copies of each db some time, and then restart couchdb with new database_dir. I will have the same names, but this method disregards "_changes" sequence, which is significant. * Upgrade couchdb to latest stable version (1.6.X, on test box) and retry replication - it doesn't keep _changes as well. * dd if=/dev/oldpartition of=/dev/newpartition is not applicable because files on old partition are constantly changing. The last chance for me to perform this movement is: * Create empty directory "fake_dir" at physical partition which hosts current database_dir * Move testdb.couch file (and associated _design/view directories) from database_dir to fake_dir. This will not affect database if corresponding filehandlers remains open. * Send http DELETE message to testdb. Couchdb will respond with "missing file" (of course, because we were moving it to another location), but this operation will close file handler for that database. File which was moved to another location will not be affected. * Downtime for partial database "testdb" begins now. * Copy testdb.couch and all associated views to new location * Check permissions to testdb.couch in this new location (should belong to same user which drives couchdb) * Create symlinks for couch file "ln -s /newlocation/testdb.couch /oldlocation/testdb.couch" and for all associated views * Send any GET message to testdb and ensure that it works. * Downtime for partial database "testdb" ends now * Repeat for every database that can be copied to new location in 15 minutes or less. Thus total maintenance can continue as long as partial database downtime doesn't exceed 15 minutes. I am going to move part of overall quantity of couch databases from one physical location to another this way, but before doing this - I decided to ask: Does anybody see any underwater rocks in that "method"? Can somebody give any advice how to move databases in better way? Methods, tips, tricks, techniques, scripts - everything appreciated. Thanks in advance. -- Pavlo
