More on this backup thread... It's nice to empty files from the trash can before doing a backup, since this helps reduce the number of files building up in the rsync'ed backup (if you do it the way I have described).
I couldn't find an easy way to get a list of "removed" files in the web GUI (correct me if I'm wrong). So here is some raw SQL for getting a list from the database (which I would then locate and delete using the GUI). The usual disclaimers apply: this is not futureproof, may not work for postgres, do not attempt to modify the database without using the API or GUI, etc. SELECT f.name AS "filename", u.name AS "user", d.name AS "enclosing directory" FROM Files f LEFT JOIN Users u ON (f.owner=u.id) LEFT JOIN Directories d ON (f.directory_id=d.id) WHERE f.removed=TRUE; One option to do this in the GUI might be an optional flattened view in the Files browser. Then the filters (removed,owned by others) could work on everything at once. While we are on the subject of a GUI wish-list: would a "File->New->Biosource/Sample/Extract..." menu be of use to anyone? At the moment one must go to the list page and then the New... button. cheers, Bob. Bob MacCallum writes: > > > I just added something to my backup script which should make it easy to > restore db and files to be consistent with each other (without a lot of > orphan > files). You simply need to keep a list of all the files present at the time > of mysqldumping. These can then be used in the restore using > rsync --files-from > > > > #!/bin/bash > > # today's date > DATE=`date -Idate` > # location of base2 files > USERFILES=/data/base2/files > # location to put backups - remember to keep off-site backups too > DEST=/somewhere/else > > # mkdir $DEST/mysql-dumps > # mkdir $DEST/files > > > # dump the db > mysqldump --single-transaction --quick --add-drop-table --databases base2 > base2dynamic | gzip -c > $DEST/mysql-dumps/base2-dbs-$DATE.sql.gz > > # dump the files (important: do not use --delete) > rsync -arv $USERFILES/ $DEST/files > > # dump the file manifest > (cd $USERFILES ; find . -type f) > $DEST/files/manifest-$DATE > > > > then to restore the files from 2008-01-01, use something like this: > > mv $USERFILES $USERFILES.old > rsync -a --files-from $DEST/files/manifest-2008-01-01 $DEST/files $USERFILES > > (restoring the mysql db is self-explanatory I hope) > > > > note that I haven't actually done a restore yet... > > cheers, > Bob. > > > Nicklas Nordborg writes: > > Bob MacCallum wrote: > > > Thanks for all the ideas on this. > > > > > > re: userfiles/db sync, would this work..? > > > > > > backup: > > > > > > 1. mysqldump to date-stamp-named files > > > 2. rsync -arv /data/base2/files/ /backup/base2/files > > > (note, no --delete option on purpose) > > > > > > restore: > > > > > > 1. undump whichever mysqldump is needed > > > 2. copy over all files that are older than the mysqldump > > > > > > > There is nothing in BASE that guarantees that the name of a deleted file > > is not reused by another file later on. However, it is very unlikely > > that this should happen with the current implementation. I think you > > need to upload a file, delete it, upload another within the same second > > AND the system must choose the same random filename for both files. Even > > if this happens it would only matter if you started a mysqldump after > > the upload of the first file, but before the deletion of it. > > > > For all practical reasons, I think you can assume that files will always > > have unique pathnames, and if rsync is just adding new files to the > > backup directory you should be able to restore the file system to any > > point in time. > > > > > The one problem I see is that the restore can bring back files that had > been > > > deleted in BASE. I doubt these would interfere with the running of BASE > > > however. A small program could be written to delete any files that are > not > > > known about by the db (unless there is already something to do this). > > > > No there is no such program. > > > > > Alternatively, one can add the --delete option to rsync and live with > only > > > being able to restore the latest dump (which is OK for us for now). > > > > /Nicklas > > > > ------------------------------------------------------------------------- > > This SF.net email is sponsored by: Microsoft > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > _______________________________________________ > > The BASE general discussion mailing list > > basedb-users@lists.sourceforge.net > > unsubscribe: send a mail with subject "unsubscribe" to > > [EMAIL PROTECTED] > > -- > Bob MacCallum | VectorBase Developer | Kafatos/Christophides Groups | > Division of Cell and Molecular Biology | Imperial College London | > Phone +442075941945 | Email [EMAIL PROTECTED] > > ------------------------------------------------------------------------- > Check out the new SourceForge.net Marketplace. > It's the best place to buy or sell services for > just about anything Open Source. > http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace > _______________________________________________ > The BASE general discussion mailing list > basedb-users@lists.sourceforge.net > unsubscribe: send a mail with subject "unsubscribe" to > [EMAIL PROTECTED] -- Bob MacCallum | VectorBase Developer | Kafatos/Christophides Groups | Division of Cell and Molecular Biology | Imperial College London | Phone +442075941945 | Email [EMAIL PROTECTED] ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Register now and save $200. Hurry, offer ends at 11:59 p.m., Monday, April 7! Use priority code J8TLD2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone _______________________________________________ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject "unsubscribe" to [EMAIL PROTECTED]
