so an update on this situation. we managed to replicate the problem. took a san snapshot, got an empty db. here's where it gets tricky. we dug around the log files, and found error messages that matched these:
http://mail-archives.apache.org/mod_mbox/couchdb-user/200812.mbox/%[email protected]%3e on a hunch, we ran a compaction... and voila! all docs are back. again, this is on .91 (we're moving to .10 tonight/tomorrow), but any insight as to what could have caused this would be appreciated. thanks, alex On Mon, Oct 19, 2009 at 11:10 PM, Alex P <[email protected]> wrote: > we're using xfs, but thanks for the offer. > > looks like someone discarded the snapshot that we had, so i can't compare > the database. the only new piece of information that i have is that ebs > snapshots are differential, and clearing out all history seems to have > 'resolved' the issue. not an answer i'm happy with, but it's no longer > replicatable. > > > On Mon, Oct 19, 2009 at 10:35 PM, Dusty Doris <[email protected]>wrote: > >> On Mon, Oct 19, 2009 at 5:37 PM, Alex P <[email protected]> wrote: >> > thanks! i'll definitely look into that. >> >> If your are using ext3 I can also help you with that. You need to use >> LVM2 to create volumes and then you can do something like >> >> $ dmsetup suspend /dev/ebs/couchvol0 >> $ ebs_snapshot >> $ dmsetup resume /dev/ebs/couchvol0 >> >> If you'd like more info on that let me know and I'll add some more >> details. >> >> >> >> > >> > On Mon, Oct 19, 2009 at 4:32 PM, Glenn Rempe <[email protected]> wrote: >> > >> >> Not sure if this will help at all, but I am also using EBS to store a >> large >> >> cdb DB. >> >> It is recommended that the filesystem that you want to snapshot on EBS >> is >> >> frozen (or unmounted) if that is possible to help ensure a coherent EBS >> >> snapshot. >> >> >> >> For example, on my mysql and couchdb volumes I use the XFS filesystem >> which >> >> has an XFS freeze command which I use to temporarily lock the FS and >> flush >> >> it to disk, and then request the snapshot. You can do this in a script >> and >> >> it takes only a few seconds of disk downtime. I believe writes are >> cached >> >> during this period. I have been using this with mysql but have not yet >> set >> >> it up for my couchdb volume. >> >> >> >> See : >> >> >> >> >> >> >> http://developer.amazonwebservices.com/connect/entry.jspa?categoryID=100&externalID=1663 >> >> http://alestic.com/2009/09/ec2-consistent-snapshot >> >> >> >> HTH >> >> >> >> On Mon, Oct 19, 2009 at 1:55 PM, Chris Anderson <[email protected]> >> wrote: >> >> >> >> > On Mon, Oct 19, 2009 at 12:53 PM, Alex P <[email protected]> >> wrote: >> >> > > a simple copy seems to work just fine. db shows up with the right >> seq >> >> > number >> >> > > etc. >> >> > >> >> > cool. totally wanna learn about the SAN roundtrip. :) >> >> > >> >> > > >> >> > > On Mon, Oct 19, 2009 at 2:37 PM, Paul Davis < >> >> [email protected] >> >> > >wrote: >> >> > > >> >> > >> On Mon, Oct 19, 2009 at 3:35 PM, Alex P <[email protected]> >> wrote: >> >> > >> > cdb catalogs databases purely on file name and presence in the >> >> > >> appropriate >> >> > >> > directory, right? so if i copy db1 to db1_copy, i should see >> >> db1_copy >> >> > >> show >> >> > >> > up in futon? >> >> > >> >> >> > >> Yepper. Make sure the permissions are correct after copying >> though. >> >> > >> >> >> > >> Paul Davis >> >> > >> >> >> > > >> >> > >> >> > >> >> > >> >> > -- >> >> > Chris Anderson >> >> > http://jchrisa.net >> >> > http://couch.io >> >> > >> >> >> >> >> >> >> >> -- >> >> Glenn Rempe >> >> >> >> email : [email protected] >> >> voice : (415) 894-5366 or (415)-89G-LENN >> >> twitter : @grempe >> >> contact info : http://www.rempe.us/contact.html >> >> pgp : http://www.rempe.us/gnupg.txt >> >> >> > >> > >
