I have described a generalized repair algorithm which should work even for CouchDB's that have been restarted multiple times leaving orphaned data in the middle of the file (and valid headers after it)
Adam Kocoloski has offered to write the "for Node in BtreeNodes" part and Volker Mische and I are working on collecting the optimal set of BtreeNodes. http://wiki.couchone.com/page/recovery-tool (see the last section "Generalized algorithm") If you want to write any of this, also write it along side us and we will compare notes and help each other out on IRC. Chris On Aug 9, 2010, at 11:53 AM, Filipe David Manana wrote: > For the record (and people not on IRC), the code at: > > http://github.com/fdmanana/couchdb/commits/db_repair > > is working for at least simple cases. Use > couch_db_repair:repair(DbNameAsString). > There's one TODO: update the reduce values for the by_seq and by_id BTrees. > > If anyone wants to give some help on this, your welcome. > > On Mon, Aug 9, 2010 at 6:12 PM, Mikeal Rogers <[email protected]>wrote: > >> I'm starting to create a bunch of test db files that expose this bug under >> different conditions like multiple restarts, across compaction, variances >> in >> updates the might cause conflict, etc. >> >> http://github.com/mikeal/couchtest >> >> The README outlines what was done to the db's and what needs to be >> recovered. >> >> -Mikeal >> >> On Mon, Aug 9, 2010 at 9:33 AM, Filipe David Manana <[email protected] >>> wrote: >> >>> On Mon, Aug 9, 2010 at 5:22 PM, Robert Newson <[email protected] >>>> wrote: >>> >>>> Doesn't this bit; >>>> >>>> - Db#db{waiting_delayed_commit=nil}; >>>> + Db; >>>> + % Db#db{waiting_delayed_commit=nil}; >>>> >>>> revert the bug fix? >>>> >>> >>> That's intentional, for my local testing. >>> That patch isn't obviously anything close to final, it's too experimental >>> yet. >>> >>>> >>>> B. >>>> >>>> On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <[email protected]> wrote: >>>>> Hi All, >>>>> >>>>> Filipe jumped in to start working on the recovery tool, but he isn't >>> done >>>> yet. >>>>> >>>>> Here's the current patch: >>>>> >>>>> http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz >>>>> >>>>> it is not done and very early, but any help on this is greatly >>>> appreciated. >>>>> >>>>> The current state is (in Filipe's words): >>>>> - i can detect that a file needs repair >>>>> - and get the last btree roots from it >>>>> - "only" missing: get last db seq num >>>>> - write new header >>>>> - and deal with the local docs btree (if exists) >>>>> >>>>> Thanks! >>>>> Jan >>>>> -- >>>>> >>>>> >>>> >>> >>> >>> >>> -- >>> Filipe David Manana, >>> [email protected] >>> >>> "Reasonable men adapt themselves to the world. >>> Unreasonable men adapt the world to themselves. >>> That's why all progress depends on unreasonable men." >>> >> > > > > -- > Filipe David Manana, > [email protected] > > "Reasonable men adapt themselves to the world. > Unreasonable men adapt the world to themselves. > That's why all progress depends on unreasonable men."
