On Sep 11, 2007, at 10:27 AM, Alan Runyan wrote:

And, as you said in another node, the BTree folder actually loves in
the resources database.

Correct the BTree is in /plone/resources/files to be exact.

Cross database references are inherently weak.  A reference from a
foreign database doesn't prevent an object from being treated as
garbage.  So, if the only reference to an object is from a foreign
database, then the object is considered garbage.  It doesn't sound
like this is what's affecting you.  The cross-database reference is
to the BTree.  It sounds like the internal references are within

Well. Someone could have 'copy/pasted' a file from the content database
into the resources/files database.  That could have been one issue.


BTW, I assume you mean cut/paste aka move.

  - checkbtrees.py
  - fstest.py

There's an fsrefs script that checks internal references I believe.

fsrefs.py shows loads of problems in both the data.fs and the resources.fs.
probably > 200 entries per database. i.e.

oid 0xD87110L BTrees._OOBTree.OOBucket
last updated: 2007-09-04 14:43:37.687332, tid=0x37020D3A0CC9DCCL
refers to invalid objects:
        oid ('\x00\x00\x00\x00\x00\xb0+f', None) missing: '<unknown>'
oid ('\x00\x00\x00\x00\x00\xb0N\xbc', None) missing: '<unknown>' oid ('\x00\x00\x00\x00\x00\xb0N\xbd', None) missing: '<unknown>' oid ('\x00\x00\x00\x00\x00\xd7\xb1\xa0', None) missing: '<unknown>' oid ('\x00\x00\x00\x00\x00\xc5\xe8:', None) missing: '<unknown>' oid ('\x00\x00\x00\x00\x00\xc3\xc6l', None) missing: '<unknown>' oid ('\x00\x00\x00\x00\x00\xc3\xc6m', None) missing: '<unknown>'
        oid ('\x00\x00\x00\x00\x00\xcahC', None) missing: '<unknown>'
oid ('\x00\x00\x00\x00\x00\xaf\x07\xc1', None) missing: '<unknown>'

Interesting. I wonder if these are actually cross-database references.

My questions are:

- I imagine if there are 'invalid' references this is considered "corruption"
   or "inconsistency"?

I consider this inconsistency. The file structure is intact, but the data isn't what it should be. Not that it matters to the end user what we call it.

  - How do I tell if something is a reference to another database?

I don't know how to do this with fsrefs. I'm not 100% sure that fsrefs recognizes cross-database references.

- Having these invalid references, is this common to ZODB applications?


Possibly, there's a backup that has data records for the missing OIDs.

Going to ask hosting company to pull up backups for the past few weeks. But how i'm going to find this other than "seeing if the folder allows me to iterate over the items" is not throwing POSKeyError. Does that sound
like a decent litmus test?

Well. there's also fsrefs.

I'll try to make some time in the next few days to look at this issue.

I'll look at fsrefs a bit more closely to:

  - make sure it understands cross-database references, and

- Make sure it reports whether missing references are local or remote.

I'd like to decide what to do next based on this investigation. In particular, I want to be sure if the problems you are having are actually due to cross-database reference issues.

I'll also look at writing a tool that might be able to recover lost objects from backup databases. The idea is that a tool would scan a database for missing oids save the list to files, separating references to different databases. Then there'd be another tool that would read this list and a list of old database files and scan the files looking for oids in the list and extracting records if they are found.

I do suspect we need to do something about cross-database references. My long-term plan is to:

- Add an option to file storages to skip garbage collection when packing.

- Add a multi-database garbage-collection protocol and tool

In the short term, It might be good to have a mechanism for limiting which objects can have cross-database reference to them to limit the chance of inadvertent cross-datavase references via move. This would need to be fleshed out though, which takes time. Perhaps something can be done at the zope or plone level in the code for moving objects to make sure that objects aren't moved between databases.


Jim Fulton                      mailto:[EMAIL PROTECTED]                Python 
CTO                             (540) 361-1714                  
Zope Corporation        http://www.zope.com             http://www.zope.org

For more information about ZODB, see the ZODB Wiki:

ZODB-Dev mailing list  -  ZODB-Dev@zope.org

Reply via email to