Re: [ZODB-Dev] Repair corrupted RelStorage database

2010-02-01 Thread Rudá Porto Filgueiras
On Sat, Jan 30, 2010 at 2:51 PM, Rudá Porto Filgueiras rud...@gmail.comwrote:

 On Sat, Jan 30, 2010 at 6:56 AM, Shane Hathaway sh...@hathawaymix.orgwrote:


 - Run the MySQL database verification and repair tools.  It's possible
 that the object you need is there, but MySQL can't find it due to corrupted
 indexes or similar.


 I will do that, but it must afert 02:00 am. :-)


Only to update:

After check and recover all MYISAM tables:  myisamchk -c *.MYI  and
myisamchk -r *.MYI, zodbconvert.py runs fine and read all objects and
trasactions from MySQL and export to FileStorage file with 22GB.

I think it solve these integrity problems, but I will run the migration
process to see what will happen.
Shane, thanks again by your support.

Cheers,

-- 
Rudá Porto Filgueiras
http://python-blog.blogspot.com
http://twitter.com/rudaporto
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Repair corrupted RelStorage database

2010-01-30 Thread Shane Hathaway
Rudá Porto Filgueiras wrote:
 How can I repair or verify a RelStorage database?
 I´m using MySQL Adapter, there is any script like fsrecover.py
 
 Follow the failure when RelStorage try to load some objects:
 
 2010-01-30T04:25:14 ERROR ZODB.Connection Couldn't load state for 0x04ef0f59
 Traceback (most recent call last):
   File 
 /usr/local/zope-agecom/plone/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py,
  
 line 847, in setstate
 self._setstate(obj)
   File 
 /usr/local/zope-agecom/plone/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py,
  
 line 888, in _setstate
 p, serial = self._storage.load(obj._p_oid, self._version)
   File 
 /usr/local/zope-agecom/plone/eggs/RelStorage-1.4.0b1-py2.4.egg/relstorage/storage.py,
  
 line 432, in load
 raise POSKeyError(oid)
 POSKeyError: 0x04ef0f59

To diagnose this, I would:

- Examine the logs in detail.  There should be a log entry giving 
details about that specific POSKeyError.

- Shut down MySQL and back up the database as-is.

- Run the MySQL database verification and repair tools.  It's possible 
that the object you need is there, but MySQL can't find it due to 
corrupted indexes or similar.

- Run a pack in dry-run mode, which will fill the object_refs table so I 
can find out exactly what object(s) refer to that OID (note that 
0x04ef0f59 is 82775897 in decimal).

- Use both SQL and zopectl debug to examine the breakage.

Running a pack in dry-run mode performs verification without actually 
deleting anything.  There is no fsrecover-like script, but I could be 
convinced to create one if the need arises.

Shane
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Repair corrupted RelStorage database

2010-01-30 Thread Rudá Porto Filgueiras
On Sat, Jan 30, 2010 at 6:56 AM, Shane Hathaway sh...@hathawaymix.orgwrote:

 Rudá Porto Filgueiras wrote:

 How can I repair or verify a RelStorage database?
 I´m using MySQL Adapter, there is any script like fsrecover.py

 Follow the failure when RelStorage try to load some objects:

 2010-01-30T04:25:14 ERROR ZODB.Connection Couldn't load state for
 0x04ef0f59
 Traceback (most recent call last):
  File
 /usr/local/zope-agecom/plone/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py,
 line 847, in setstate
self._setstate(obj)
  File
 /usr/local/zope-agecom/plone/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py,
 line 888, in _setstate
p, serial = self._storage.load(obj._p_oid, self._version)
  File
 /usr/local/zope-agecom/plone/eggs/RelStorage-1.4.0b1-py2.4.egg/relstorage/storage.py,
 line 432, in load
raise POSKeyError(oid)
 POSKeyError: 0x04ef0f59


 To diagnose this, I would:

 - Examine the logs in detail.  There should be a log entry giving details
 about that specific POSKeyError.



Follow error from november 2009 (there are some repeated (five) errors like
that on the date/time).

550308-2009-11-28T05:31:51 ERROR ZODB.Connection Couldn't load state for
0x03ebd4
550309-Traceback (most recent call last):
550310-  File
/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py,
line 797, in setstate
550311-self._setstate(obj)
550312-  File
/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py,
line 855, in _setstate
550313-self._reader.setGhostState(obj, p)
550314-  File
/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/serialize.py,
line 604, in setGhostState
550315-state = self.getState(pickle)
550316-  File
/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/serialize.py,
line 597, in getState
550317-return unpickler.load()
550318-  File
/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/serialize.py,
line 471, in _persistent_load
550319-return self.load_oid(reference)
550320-  File
/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/serialize.py,
line 537, in load_oid
550321-return self._conn.get(oid)
550322-  File
/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py,
line 225, in get
550323-p, serial = self._storage.load(oid, self._version)
550324-  File
/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py,
line 1196, in load
550325-return self._storage.load(oid, self._base_version)
550326-  File
/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZEO/ClientStorage.py,
line 727, in load
550327-return self.loadEx(oid, version)[:2]
550328-  File
/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZEO/ClientStorage.py,
line 750, in loadEx
550329-data, tid, ver = self._server.loadEx(oid, version)
550330-  File
/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZEO/ServerStub.py,
line 196, in loadEx
550331-return self.rpc.call(loadEx, oid, version)
550332-  File
/usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZEO/zrpc/connection.py,
line 650, in call
550333-raise inst # error raised by server
550334:POSKeyError: 0x06294e

This errros is today errors, there are repeated (10) errors like that:

835744:2010-01-30T09:26:47 WARNING relstorage POSKeyError on oid 1793877: no
tid found; Current transaction is 253262639700997905; Recent object tids: []
835745---
835746-2010-01-30T09:26:47 ERROR ZODB.Connection Couldn't load state for
0x1b5f55
835747-Traceback (most recent call last):
835748-  File
/usr/local/zope/agecom-virtual/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py,
line 847, in setstate
835749-self._setstate(obj)
835750-  File
/usr/local/zope/agecom-virtual/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py,
line 888, in _setstate
835751-p, serial = self._storage.load(obj._p_oid, self._version)
835752-  File
/usr/local/zope/agecom-virtual/eggs/RelStorage-1.4.0b1-py2.4.egg/relstorage/storage.py,
line 432, in load
835753:raise POSKeyError(oid)
835754:POSKeyError: 0x1b5f55
835756-2010-01-30T09:26:48 INFO Archetypes
ESC[00mESC[01;32m/usr/local/zope/agecom-virtual/parts/plone/Archetypes/UIDCatalog.py[106]:getObject
835757-ESC[00mUIDCatalogBrains getObject raised an error:
835758- Traceback (most recent call last):
835759-
835760-  File
/usr/local/zope/agecom-virtual/parts/plone/Archetypes/UIDCatalog.py, line
87, in getObject
835761-path = self.getPath()
835762-
835763-  File
/usr/local/zope/agecom-virtual/parts/zope2/lib/python/Products/ZCatalog/CatalogBrains.py,
line 33, in getPath
835764-return self.aq_parent.getpath(self.data_record_id_)