Alex, Are any of the beam files used by the running CouchDB node on the EBS volume? I've seen these errors before and they're generally (always?) a result of the Erlang VM being unable to read from its current working directory.
Paul Davis On Thu, Oct 22, 2009 at 2:28 AM, Alex P <[email protected]> wrote: > so an update on this situation. we managed to replicate the problem. took a > san snapshot, got an empty db. here's where it gets tricky. we dug around > the log files, and found error messages that matched these: > > http://mail-archives.apache.org/mod_mbox/couchdb-user/200812.mbox/%[email protected]%3e > > on a hunch, we ran a compaction... and voila! all docs are back. > > again, this is on .91 (we're moving to .10 tonight/tomorrow), but any > insight as to what could have caused this would be appreciated. > > thanks, > alex > > On Mon, Oct 19, 2009 at 11:10 PM, Alex P <[email protected]> wrote: > >> we're using xfs, but thanks for the offer. >> >> looks like someone discarded the snapshot that we had, so i can't compare >> the database. the only new piece of information that i have is that ebs >> snapshots are differential, and clearing out all history seems to have >> 'resolved' the issue. not an answer i'm happy with, but it's no longer >> replicatable. >> >> >> On Mon, Oct 19, 2009 at 10:35 PM, Dusty Doris <[email protected]>wrote: >> >>> On Mon, Oct 19, 2009 at 5:37 PM, Alex P <[email protected]> wrote: >>> > thanks! i'll definitely look into that. >>> >>> If your are using ext3 I can also help you with that. You need to use >>> LVM2 to create volumes and then you can do something like >>> >>> $ dmsetup suspend /dev/ebs/couchvol0 >>> $ ebs_snapshot >>> $ dmsetup resume /dev/ebs/couchvol0 >>> >>> If you'd like more info on that let me know and I'll add some more >>> details. >>> >>> >>> >>> > >>> > On Mon, Oct 19, 2009 at 4:32 PM, Glenn Rempe <[email protected]> wrote: >>> > >>> >> Not sure if this will help at all, but I am also using EBS to store a >>> large >>> >> cdb DB. >>> >> It is recommended that the filesystem that you want to snapshot on EBS >>> is >>> >> frozen (or unmounted) if that is possible to help ensure a coherent EBS >>> >> snapshot. >>> >> >>> >> For example, on my mysql and couchdb volumes I use the XFS filesystem >>> which >>> >> has an XFS freeze command which I use to temporarily lock the FS and >>> flush >>> >> it to disk, and then request the snapshot. You can do this in a script >>> and >>> >> it takes only a few seconds of disk downtime. I believe writes are >>> cached >>> >> during this period. I have been using this with mysql but have not yet >>> set >>> >> it up for my couchdb volume. >>> >> >>> >> See : >>> >> >>> >> >>> >> >>> http://developer.amazonwebservices.com/connect/entry.jspa?categoryID=100&externalID=1663 >>> >> http://alestic.com/2009/09/ec2-consistent-snapshot >>> >> >>> >> HTH >>> >> >>> >> On Mon, Oct 19, 2009 at 1:55 PM, Chris Anderson <[email protected]> >>> wrote: >>> >> >>> >> > On Mon, Oct 19, 2009 at 12:53 PM, Alex P <[email protected]> >>> wrote: >>> >> > > a simple copy seems to work just fine. db shows up with the right >>> seq >>> >> > number >>> >> > > etc. >>> >> > >>> >> > cool. totally wanna learn about the SAN roundtrip. :) >>> >> > >>> >> > > >>> >> > > On Mon, Oct 19, 2009 at 2:37 PM, Paul Davis < >>> >> [email protected] >>> >> > >wrote: >>> >> > > >>> >> > >> On Mon, Oct 19, 2009 at 3:35 PM, Alex P <[email protected]> >>> wrote: >>> >> > >> > cdb catalogs databases purely on file name and presence in the >>> >> > >> appropriate >>> >> > >> > directory, right? so if i copy db1 to db1_copy, i should see >>> >> db1_copy >>> >> > >> show >>> >> > >> > up in futon? >>> >> > >> >>> >> > >> Yepper. Make sure the permissions are correct after copying >>> though. >>> >> > >> >>> >> > >> Paul Davis >>> >> > >> >>> >> > > >>> >> > >>> >> > >>> >> > >>> >> > -- >>> >> > Chris Anderson >>> >> > http://jchrisa.net >>> >> > http://couch.io >>> >> > >>> >> >>> >> >>> >> >>> >> -- >>> >> Glenn Rempe >>> >> >>> >> email : [email protected] >>> >> voice : (415) 894-5366 or (415)-89G-LENN >>> >> twitter : @grempe >>> >> contact info : http://www.rempe.us/contact.html >>> >> pgp : http://www.rempe.us/gnupg.txt >>> >> >>> > >>> >> >> >
