On Sat, Jan 30, 2010 at 6:56 AM, Shane Hathaway <sh...@hathawaymix.org>wrote:
> Rudá Porto Filgueiras wrote:
>
>> How can I repair or verify a RelStorage database?
>> I´m using MySQL Adapter, there is any script like fsrecover.py
>>
>> Follow the failure when RelStorage try to load some objects:
>>
>> 2010-01-30T04:25:14 ERROR ZODB.Connection Couldn't load state for
>> 0x04ef0f59
>> Traceback (most recent call last):
>> File
>> "/usr/local/zope-agecom/plone/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
>> line 847, in setstate
>> self._setstate(obj)
>> File
>> "/usr/local/zope-agecom/plone/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
>> line 888, in _setstate
>> p, serial = self._storage.load(obj._p_oid, self._version)
>> File
>> "/usr/local/zope-agecom/plone/eggs/RelStorage-1.4.0b1-py2.4.egg/relstorage/storage.py",
>> line 432, in load
>> raise POSKeyError(oid)
>> POSKeyError: 0x04ef0f59
>>
>
> To diagnose this, I would:
>
> - Examine the logs in detail. There should be a log entry giving details
> about that specific POSKeyError.
>
Follow error from november 2009 (there are some repeated (five) errors like
that on the date/time).
550308-2009-11-28T05:31:51 ERROR ZODB.Connection Couldn't load state for
0x03ebd4
550309-Traceback (most recent call last):
550310- File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
line 797, in setstate
550311- self._setstate(obj)
550312- File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
line 855, in _setstate
550313- self._reader.setGhostState(obj, p)
550314- File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/serialize.py",
line 604, in setGhostState
550315- state = self.getState(pickle)
550316- File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/serialize.py",
line 597, in getState
550317- return unpickler.load()
550318- File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/serialize.py",
line 471, in _persistent_load
550319- return self.load_oid(reference)
550320- File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/serialize.py",
line 537, in load_oid
550321- return self._conn.get(oid)
550322- File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
line 225, in get
550323- p, serial = self._storage.load(oid, self._version)
550324- File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
line 1196, in load
550325- return self._storage.load(oid, self._base_version)
550326- File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZEO/ClientStorage.py",
line 727, in load
550327- return self.loadEx(oid, version)[:2]
550328- File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZEO/ClientStorage.py",
line 750, in loadEx
550329- data, tid, ver = self._server.loadEx(oid, version)
550330- File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZEO/ServerStub.py",
line 196, in loadEx
550331- return self.rpc.call("loadEx", oid, version)
550332- File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZEO/zrpc/connection.py",
line 650, in call
550333- raise inst # error raised by server
550334:POSKeyError: 0x06294e
This errros is today errors, there are repeated (10) errors like that:
835744:2010-01-30T09:26:47 WARNING relstorage POSKeyError on oid 1793877: no
tid found; Current transaction is 253262639700997905; Recent object tids: []
835745-------
835746-2010-01-30T09:26:47 ERROR ZODB.Connection Couldn't load state for
0x1b5f55
835747-Traceback (most recent call last):
835748- File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
line 847, in setstate
835749- self._setstate(obj)
835750- File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
line 888, in _setstate
835751- p, serial = self._storage.load(obj._p_oid, self._version)
835752- File
"/usr/local/zope/agecom-virtual/eggs/RelStorage-1.4.0b1-py2.4.egg/relstorage/storage.py",
line 432, in load
835753: raise POSKeyError(oid)
835754:POSKeyError: 0x1b5f55
835756-2010-01-30T09:26:48 INFO Archetypes
ESC[00mESC[01;32m/usr/local/zope/agecom-virtual/parts/plone/Archetypes/UIDCatalog.py[106]:getObject
835757-ESC[00mUIDCatalogBrains getObject raised an error:
835758- Traceback (most recent call last):
835759-
835760- File
"/usr/local/zope/agecom-virtual/parts/plone/Archetypes/UIDCatalog.py", line
87, in getObject
835761- path = self.getPath()
835762-
835763- File
"/usr/local/zope/agecom-virtual/parts/zope2/lib/python/Products/ZCatalog/CatalogBrains.py",
line 33, in getPath
835764- return self.aq_parent.getpath(self.data_record_id_)
835765-
835766- File
"/usr/local/zope/agecom-virtual/parts/zope2/lib/python/Products/ZCatalog/ZCatalog.py",
line 608, in getpath
835767- return self._catalog.paths[rid]
835768-
835769- File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
line 847, in setstate
835770- self._setstate(obj)
835771-
835772- File
"/usr/local/zope/agecom-virtual/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py",
line 888, in _setstate
835773- p, serial = self._storage.load(obj._p_oid, self._version)
835774-
835775- File
"/usr/local/zope/agecom-virtual/eggs/RelStorage-1.4.0b1-py2.4.egg/relstorage/storage.py",
line 432, in load
835776: raise POSKeyError(oid)
835777-
835778:POSKeyError: 0x1b5f55
> - Shut down MySQL and back up the database as-is.
>
> I discover the problem after restore database on another MySQL/Zope server
and try to do Plone migration 2.5 -> 3.2.
> - Run the MySQL database verification and repair tools. It's possible that
> the object you need is there, but MySQL can't find it due to corrupted
> indexes or similar.
>
>
I will do that, but it must afert 02:00 am. :-)
Did you think PostgreSQL is more resistent to such failures than MySQL?
> - Run a pack in dry-run mode, which will fill the object_refs table so I
> can find out exactly what object(s) refer to that OID (note that 0x04ef0f59
> is 82775897 in decimal).
>
>
Yes, I know that, there is a function to do this conversion on ZODB package.
Is
>>> from ZODB.utils import U64
>>> U64('0x1b5f55')
3492595808898266421L
> - Use both SQL and "zopectl debug" to examine the breakage.
>
On SQL I need to inspect current_object, object_ref and object_state to find
these "ghost" objects and who is trying to load then?
>
> Running a pack in dry-run mode performs verification without actually
> deleting anything. There is no fsrecover-like script, but I could be
> convinced to create one if the need arises.
>
>
I don't know how dificult is, but I'm sure this will be a high value for now
and the future. :-)
For me it can be something like zodbconvert (moving from the same or another
Storage type) but doing these corrections when it find transactions or lod
object failure.
I try to use ZODB convert to moving from MySQL to FileStorage, but it
generates a failure, but I do that to discover the exactly transaction and
date time that generates other failure.
After that I correct the problem by creating a "stub dict" pickle
object_state to be able to delete the "father" content becouse before that
if I try to list the folder contents it generates POSKeyError exception.
I expect not need to do this again, but know that happen again, could be a
server problem, and I have backup.
But how I can restore old database and apply all good objects created after
the corrupted transaction?
Anyway, thanks for your support.
> Shane
>
--
Rudá Porto Filgueiras
http://python-blog.blogspot.com
http://twitter.com/rudaporto
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
