Another thing you might think about is to modify the membase code (it is open-source after all) such that membase itself invokes the backup using the sqlite3_backup() interfaces. If the backup is being performed using the same database connection that is doing the writes, then the backup will know which pages of the original database file change, and only the changed pages will have to be rewritten. Hence, the backup doesn't need to start over again at the beginning as it does when a write occurs because of a separate process.
On Tue, Sep 13, 2011 at 2:39 PM, David Holtkamp <holtka...@gmail.com> wrote: > I was unaware of this. I thought it started to write to a Journal file > during the backup and then persisted it afterward. However, if this > scenario was happening, wouldn't the file size tend to jump around > sporadically around 200Mb instead of settling into one specific size? > I don't follow your reasoning here..... > > > > > On Tue, Sep 13, 2011 at 2:29 PM, Richard Hipp <d...@sqlite.org> wrote: > >> On Tue, Sep 13, 2011 at 2:23 PM, David Holtkamp <holtka...@gmail.com> > wrote: > >> > >>> Hello, > >>> I have been using Membase (a nosql database), which uses sqlite3 to > >>> persist data to disk and have run into some problems when attempting > >>> to backup the database files. Membase uses 4 different database files > >>> to store the data on each Node and in my particular case, each file is > >>> around 800 Mb. When I issue the backup statement: > >>> > >>> /opt/membase/bin/sqlite3 /mnt/data-store/default-data/default-1.mb > >>> '.backup /mnt/data-backup/mbfiles/default-1-backup.mb' > >>> > >> > >> When some other process writes to the database file, the backup > >> automatically starts over again from the beginning. (It has to do this, > >> since it has no way of knowing what parts of the file changed.) > Probably > >> membase is writing to the database a periodic intervals that are > >> approximately the same amount of time it takes you to copy 200MB. > >> > >> Is there any way to ask membase to stop persisting for long enough for > you > >> to complete your backup? > >> > >> > >>> > >>> The command executes and almost immediately copies 180Mb and then > >>> comes to a near standstill. It continues in tiny spurts until it > >>> reaches 260915200 bytes, and then the file stops growing but the > >>> process continues to consume CPU resources. I've been trying to get a > >>> successful backup for weeks now with no avail. > >>> > >>> If it matters, this system is set up on the Amazon Cloud via EC2 and > >>> is using EBS drives. Additionally, this is a production environment so > >>> the db is being accessed and restarts cannot be performed. > >>> > >>> Also here is a list of the files (I have set them and the destination > >>> directory to 777 permissions to fix some access issues) > >>> > >>> drwxrwxrwx 2 membase membase 4096 Sep 13 18:19 . > >>> drwxrwxrwx 4 membase root 4096 Aug 5 01:10 .. > >>> -rwxrwxrwx 1 membase membase 53248 Sep 13 18:19 default > >>> -rwxrwxrwx 1 membase membase 892068864 Sep 13 18:19 default-0.mb > >>> -rwxrwxrwx 1 membase membase 32768 Sep 13 18:19 default-0.mb-shm > >>> -rwxrwxrwx 1 membase membase 1858136 Sep 13 18:19 default-0.mb-wal > >>> -rwxrwxrwx 1 membase membase 892246016 Sep 13 18:19 default-1.mb > >>> -rwxrwxrwx 1 membase membase 32768 Sep 13 18:19 default-1.mb-shm > >>> -rwxrwxrwx 1 membase membase 1422168 Sep 13 18:19 default-1.mb-wal > >>> -rwxrwxrwx 1 membase membase 891885568 Sep 13 18:19 default-2.mb > >>> -rwxrwxrwx 1 membase membase 32768 Sep 13 18:19 default-2.mb-shm > >>> -rwxrwxrwx 1 membase membase 1322608 Sep 13 18:19 default-2.mb-wal > >>> -rwxrwxrwx 1 membase membase 892113920 Sep 13 18:18 default-3.mb > >>> -rwxrwxrwx 1 membase membase 32768 Sep 13 18:19 default-3.mb-shm > >>> -rwxrwxrwx 1 membase membase 1359288 Sep 13 18:19 default-3.mb-wal > >>> -rwxrwxrwx 1 ec2-user ec2-user 0 Sep 12 17:12 default-4.mb > >>> -rwxrwxrwx 1 ec2-user ec2-user 0 Sep 12 16:27 default.mb > >>> -rwxrwxrwx 1 membase membase 32768 Sep 13 18:19 default-shm > >>> -rwxrwxrwx 1 membase membase 1083664 Sep 13 18:19 default-wal > >>> > >>> Any help would be greatly appropriated. > >>> Thanks! > >>> David > >>> _______________________________________________ > >>> sqlite-users mailing list > >>> sqlite-users@sqlite.org > >>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > >>> > >> > >> > >> > >> -- > >> D. Richard Hipp > >> d...@sqlite.org > >> _______________________________________________ > >> sqlite-users mailing list > >> sqlite-users@sqlite.org > >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > >> > > > > > > > > -- > > David Holtkamp > > Crimson Moon Entertainment LLC > > Founder / CEO > > > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > -- D. Richard Hipp d...@sqlite.org _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
