Re: [ZODB-Dev] Recovering from BTree corruption

[Jim Fulton]

On Sep 11, 2007, at 10:27 AM, Alan Runyan wrote:

And, as you said in another node, the BTree folder actually loves in
the resources database.

Correct the BTree is in /plone/resources/files to be exact.


Cross database references are inherently weak.  A reference from a
foreign database doesn't prevent an object from being treated as
garbage.  So, if the only reference to an object is from a foreign
database, then the object is considered garbage.  It doesn't sound
like this is what's affecting you.  The cross-database reference is
to the BTree.  It sounds like the internal references are within
database.

Well.  Someone could have 'copy/pasted' a file from the content  
database
into the resources/files database.  That could have been one issue.

:(

BTW, I assume you mean cut/paste aka move.

  - checkbtrees.py
  - fstest.py

There's an fsrefs script that checks internal references I believe.

fsrefs.py shows loads of problems in both the data.fs and the  
resources.fs.
probably > 200 entries per database. i.e.

oid 0xD87110L BTrees._OOBTree.OOBucket
last updated: 2007-09-04 14:43:37.687332, tid=0x37020D3A0CC9DCCL
refers to invalid objects:
        oid ('\x00\x00\x00\x00\x00\xb0+f', None) missing: '<unknown>'
        oid ('\x00\x00\x00\x00\x00\xb0N\xbc', None) missing:  
'<unknown>'
        oid ('\x00\x00\x00\x00\x00\xb0N\xbd', None) missing:  
'<unknown>'
        oid ('\x00\x00\x00\x00\x00\xd7\xb1\xa0', None) missing:  
'<unknown>'
        oid ('\x00\x00\x00\x00\x00\xc5\xe8:', None) missing:  
'<unknown>'
        oid ('\x00\x00\x00\x00\x00\xc3\xc6l', None) missing:  
'<unknown>'
        oid ('\x00\x00\x00\x00\x00\xc3\xc6m', None) missing:  
'<unknown>'
        oid ('\x00\x00\x00\x00\x00\xcahC', None) missing: '<unknown>'
        oid ('\x00\x00\x00\x00\x00\xaf\x07\xc1', None) missing:  
'<unknown>'

Interesting. I wonder if these are actually cross-database references.

My questions are:

 - I imagine if there are 'invalid' references this is considered  
"corruption"
   or "inconsistency"?

I consider this inconsistency. The file structure is intact, but the  
data isn't what it should be.  Not that it matters to the end user  
what we call it.


  - How do I tell if something is a reference to another database?

I don't know how to do this with fsrefs.  I'm not 100% sure that  
fsrefs recognizes cross-database references.


  - Having these invalid references, is this common to  ZODB  
applications?

No.


Possibly, there's a backup that has data records for the missing  
OIDs.

Going to ask hosting company to pull up backups for the past few  
weeks.
But how i'm going to find this other than "seeing if the folder  
allows me
to iterate over the items" is not throwing POSKeyError.  Does that  
sound
like a decent litmus test?

Well. there's also fsrefs.

I'll try to make some time in the next few days to look at this issue.

I'll look at fsrefs a bit more closely to:

  - make sure it understands cross-database references, and

  - Make sure it reports whether missing references are local or  
remote.

I'd like to decide what to do next based on this investigation.  In  
particular, I want to be sure if the problems you are having are  
actually due to cross-database reference issues.

I'll also look at writing a tool that might be able to recover lost  
objects from backup databases.  The idea is that a tool would scan a  
database for missing oids save the list to files, separating  
references to different databases.  Then there'd be another tool that  
would read this list and a list of old database files and scan the  
files looking for oids in the list and extracting records if they are  
found.

I do suspect we need to do something about cross-database  
references.  My long-term plan is to:

- Add an option to file storages to skip garbage collection when  
packing.

- Add a multi-database garbage-collection protocol and tool

In the short term, It might be good to have a mechanism for limiting  
which objects can have cross-database reference to them to limit the  
chance of inadvertent cross-datavase references via move.  This would  
need to be fleshed out though, which takes time.  Perhaps something  
can be done at the zope or plone level in the code for moving objects  
to make sure that objects aren't moved between databases.

Jim

--
Jim Fulton                      [EMAIL PROTECTED]                Python 
Powered!
CTO                             (540) 361-1714                  
http://www.python.org
Zope Corporation        http://www.zope.com             http://www.zope.org



_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev