[ZODB-Dev] Recovering from BTree corruption
Hi guys. It seems that one of our customers has a corrupted BTree. I would love for someone to provide some insight on how we can recover the data. we have two databases: 1 for resources and 1 for 'content'. resources contain lots of very big files. The system is configured to have a mount point at /plone/resources is a subclass of BTreeFolder, using internal data struct of OOBTree. anytime I iterate over the keys I get POSKeyError. anytime I iterate over the values the same. if I run BTree.check() on the data structure's tree attribute (the OOBTree itself) I get a POSKeyError. Running the utils.checkbtrees doesnt say this btree has a problem. While debugging this I had a conversation with sidnei about mounted databases. He recalled that if your using a mounted database you should not pack. If for some reason your mounted database had a cross reference to another database and somehow you had a dangling reference to the other database it would cause POSKeyError. Is there any other ways of testing consistency of FileStorage other than: - checkbtrees.py - fstest.py And any ideas how I can salvage the data? This BTree, of course, had the most valuable data. cheers -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Recovering from BTree corruption
On Sep 10, 2007, at 10:34 AM, Alan Runyan wrote: Hi guys. It seems that one of our customers has a corrupted BTree. I would love for someone to provide some insight on how we can recover the data. we have two databases: 1 for resources and 1 for 'content'. resources contain lots of very big files. The system is configured to have a mount point at /plone/resources is a subclass of BTreeFolder, using internal data struct of OOBTree. Does the BTree folder live in the content database or the resources database. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Recovering from BTree corruption
Alan Runyan wrote at 2007-9-10 09:34 -0500: ... While debugging this I had a conversation with sidnei about mounted databases. He recalled that if your using a mounted database you should not pack. If for some reason your mounted database had a cross reference to another database and somehow you had a dangling reference to the other database it would cause POSKeyError. BTrees are actually directed acyclic graphs (DAGs) with two node types tree (internal node) and bucket (leaf). Beside its children, a tree contains a link to its leftmost bucket. Beside its keys/values, a bucket contains a link to the next bucket. When you iterate over keys or values, the leftmost bucket is accessed via the root's leftmost bucket link and then all buckets are visited via the next bucket links. Your description seems to indicate that you have lost a next bucket link. If you are lucky, then the tree access structure (the children links of the tree nodes) is still intact -- or if not, is at least partially intact. Then, you will be able to recover large parts of your tree. You have two options: * reconstruct the tree from its pickles. This is the way, the checking of BTrees works. * Determine the last key (LK) before you get the POSKeyError; then use the tree structure to access the next available key. You may need to try ever larger values above LK to skip a potentially damanged part of the tree. I would start with the second approach and switch to the first one when it becomes too tedious. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Recovering from BTree corruption
On Sep 10, 2007, at 10:34 AM, Alan Runyan wrote: Hi guys. It seems that one of our customers has a corrupted BTree. I would love for someone to provide some insight on how we can recover the data. we have two databases: 1 for resources and 1 for 'content'. resources contain lots of very big files. The system is configured to have a mount point at /plone/resources is a subclass of BTreeFolder, using internal data struct of OOBTree. And, as you said in another node, the BTree folder actually loves in the resources database. anytime I iterate over the keys I get POSKeyError. anytime I iterate over the values the same. if I run BTree.check() on the data structure's tree attribute (the OOBTree itself) I get a POSKeyError. Running the utils.checkbtrees doesnt say this btree has a problem. While debugging this I had a conversation with sidnei about mounted databases. He recalled that if your using a mounted database you should not pack. If for some reason your mounted database had a cross reference to another database and somehow you had a dangling reference to the other database it would cause POSKeyError. Cross database references are inherently weak. A reference from a foreign database doesn't prevent an object from being treated as garbage. So, if the only reference to an object is from a foreign database, then the object is considered garbage. It doesn't sound like this is what's affecting you. The cross-database reference is to the BTree. It sounds like the internal references are within database. Is there any other ways of testing consistency of FileStorage other than: - checkbtrees.py - fstest.py There's an fsrefs script that checks internal references I believe. And any ideas how I can salvage the data? This BTree, of course, had the most valuable data. Possibly, there's a backup that has data records for the missing OIDs. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
