[ZODB-Dev] Re: AW: diploma thesis: ZODB Indexing
Jim Fulton wrote: On Sep 5, 2007, at 9:39 AM, Christian Theune wrote: [snip] I also have the feeling that our goal for ad-hoc querying would be incompatible with your envisioned framework for defining collections and indexes. I guess I have no idea what you are talking about. I assumed you meant something along the lines of what people expect of relational databases. In the relational world, people define tables and indexes in order to be able to do indexed ad-hoc queries. Maybe you are talking about something else. It is interesting to compare with XML databases. Some XML databases like MonetDB or eXist offer XPath queries into the database without anyone having to pre-define indexes. Basically these databases tend to index the entire tree structure. I'd suggest reading the eXist papers that are about. I'd also take a look at MonetDB, as at its core it's a general database system which has a RDB and XML db built on top. http://monetdb.cwi.nl/ I'm not sure how many of these ideas can be translated to work for Python structures - XPath is rather specific to XML, after all. But if anyone wants to talk about my ideas on all this, let me know. :) Regards, Martijn ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Recovering from BTree corruption
And, as you said in another node, the BTree folder actually loves in
the resources database.
Correct the BTree is in /plone/resources/files to be exact.
Cross database references are inherently weak. A reference from a
foreign database doesn't prevent an object from being treated as
garbage. So, if the only reference to an object is from a foreign
database, then the object is considered garbage. It doesn't sound
like this is what's affecting you. The cross-database reference is
to the BTree. It sounds like the internal references are within
database.
Well. Someone could have 'copy/pasted' a file from the content database
into the resources/files database. That could have been one issue.
- checkbtrees.py
- fstest.py
There's an fsrefs script that checks internal references I believe.
fsrefs.py shows loads of problems in both the data.fs and the resources.fs.
probably 200 entries per database. i.e.
oid 0xD87110L BTrees._OOBTree.OOBucket
last updated: 2007-09-04 14:43:37.687332, tid=0x37020D3A0CC9DCCL
refers to invalid objects:
oid ('\x00\x00\x00\x00\x00\xb0+f', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xb0N\xbc', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xb0N\xbd', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xd7\xb1\xa0', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xc5\xe8:', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xc3\xc6l', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xc3\xc6m', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xcahC', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xaf\x07\xc1', None) missing: 'unknown'
My questions are:
- I imagine if there are 'invalid' references this is considered corruption
or inconsistency?
- How do I tell if something is a reference to another database?
- Having these invalid references, is this common to ZODB applications?
Possibly, there's a backup that has data records for the missing OIDs.
Going to ask hosting company to pull up backups for the past few weeks.
But how i'm going to find this other than seeing if the folder allows me
to iterate over the items is not throwing POSKeyError. Does that sound
like a decent litmus test?
--
Alan Runyan
Enfold Systems, Inc.
http://www.enfoldsystems.com/
phone: +1.713.942.2377x111
fax: +1.832.201.8856
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Recovering from BTree corruption
Alan Runyan wrote at 2007-9-11 09:27 -0500:
...
oid 0xD87110L BTrees._OOBTree.OOBucket
last updated: 2007-09-04 14:43:37.687332, tid=0x37020D3A0CC9DCCL
refers to invalid objects:
oid ('\x00\x00\x00\x00\x00\xb0+f', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xb0N\xbc', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xb0N\xbd', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xd7\xb1\xa0', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xc5\xe8:', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xc3\xc6l', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xc3\xc6m', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xcahC', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\xaf\x07\xc1', None) missing: 'unknown'
Looks as if the OOBucket has lost quite some value links (as
only a single one links to the next bucket).
My questions are:
- I imagine if there are 'invalid' references this is considered corruption
or inconsistency?
I depends on your preferences.
...
- Having these invalid references, is this common to ZODB applications?
No.
At least not for ZODB applications that do not use inter database
references.
Possibly, there's a backup that has data records for the missing OIDs.
Going to ask hosting company to pull up backups for the past few weeks.
But how i'm going to find this other than seeing if the folder allows me
to iterate over the items is not throwing POSKeyError. Does that sound
like a decent litmus test?
You can also run fsrefs on it. When you do not get missing ...,
then the backup does not have you POSKeyError (but may lack quite
a few newer modifications).
--
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
