Re: [ZODB-Dev] Repair corrupted RelStorage database
Rudá Porto Filgueiras wrote: How can I repair or verify a RelStorage database? I´m using MySQL Adapter, there is any script like fsrecover.py Follow the failure when RelStorage try to load some objects: 2010-01-30T04:25:14 ERROR ZODB.Connection Couldn't load state for 0x04ef0f59 Traceback (most recent call last): File /usr/local/zope-agecom/plone/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py, line 847, in setstate self._setstate(obj) File /usr/local/zope-agecom/plone/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py, line 888, in _setstate p, serial = self._storage.load(obj._p_oid, self._version) File /usr/local/zope-agecom/plone/eggs/RelStorage-1.4.0b1-py2.4.egg/relstorage/storage.py, line 432, in load raise POSKeyError(oid) POSKeyError: 0x04ef0f59 To diagnose this, I would: - Examine the logs in detail. There should be a log entry giving details about that specific POSKeyError. - Shut down MySQL and back up the database as-is. - Run the MySQL database verification and repair tools. It's possible that the object you need is there, but MySQL can't find it due to corrupted indexes or similar. - Run a pack in dry-run mode, which will fill the object_refs table so I can find out exactly what object(s) refer to that OID (note that 0x04ef0f59 is 82775897 in decimal). - Use both SQL and zopectl debug to examine the breakage. Running a pack in dry-run mode performs verification without actually deleting anything. There is no fsrecover-like script, but I could be convinced to create one if the need arises. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] free space for FileStorage, and possible issues with repozo (was: Missing Content)
Michael Havard wrote: We're missing 16 days worth of content. Python 2.3.5, Zope 2.8.4, Plone 2.1.2, and a host of other products. ZEO configuration. Persistent disk caching is on. Yesterday everything was fine. All the content was there and accessible. I recall reading about a case that involved quietly running out of free space on the volume used for FileStorage. Something like … the most recent content was in RAM — and the running ZEO cluster seemed to work fine with that (no adverse effect on the end user, at the time) — but that content was not committed to disk (and so, was probably unrecoverable following a restart of the OS). … Thinking maybe someone restored from an older backup I went out to our daily backups and pulled the previous days data.fs backup. A restore didn't work. It still shows the outdated content. I recently had a similar experience. In Plone, using the backup script associated with collective.recipe.backup I: 1. restored to a less recent point in time (point A), without the desired effect 2. restored to a more recent point in time (point B), but did not find the content that had been created/modified between points A and B. More specifically: I found old versions of content, without the recent modifications. AFAIR the failed restoration occurred in the midst of http://n2.nabble.com/-tp4291640p4291640.html using ZODB 3.8.5 (greater than the version normally associated with Plone 3.3.3) with Zope 2.10.7 (less than the version normally associated with Plone 3.3.3). -- View this message in context: http://old.nabble.com/Missing-Content-tp4174116p27382995.html Sent from the Zope - ZODB-Dev mailing list archive at Nabble.com. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repair corrupted RelStorage database
On Sat, Jan 30, 2010 at 6:56 AM, Shane Hathaway sh...@hathawaymix.orgwrote: Rudá Porto Filgueiras wrote: How can I repair or verify a RelStorage database? I´m using MySQL Adapter, there is any script like fsrecover.py Follow the failure when RelStorage try to load some objects: 2010-01-30T04:25:14 ERROR ZODB.Connection Couldn't load state for 0x04ef0f59 Traceback (most recent call last): File /usr/local/zope-agecom/plone/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py, line 847, in setstate self._setstate(obj) File /usr/local/zope-agecom/plone/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py, line 888, in _setstate p, serial = self._storage.load(obj._p_oid, self._version) File /usr/local/zope-agecom/plone/eggs/RelStorage-1.4.0b1-py2.4.egg/relstorage/storage.py, line 432, in load raise POSKeyError(oid) POSKeyError: 0x04ef0f59 To diagnose this, I would: - Examine the logs in detail. There should be a log entry giving details about that specific POSKeyError. Follow error from november 2009 (there are some repeated (five) errors like that on the date/time). 550308-2009-11-28T05:31:51 ERROR ZODB.Connection Couldn't load state for 0x03ebd4 550309-Traceback (most recent call last): 550310- File /usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py, line 797, in setstate 550311-self._setstate(obj) 550312- File /usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py, line 855, in _setstate 550313-self._reader.setGhostState(obj, p) 550314- File /usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/serialize.py, line 604, in setGhostState 550315-state = self.getState(pickle) 550316- File /usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/serialize.py, line 597, in getState 550317-return unpickler.load() 550318- File /usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/serialize.py, line 471, in _persistent_load 550319-return self.load_oid(reference) 550320- File /usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/serialize.py, line 537, in load_oid 550321-return self._conn.get(oid) 550322- File /usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py, line 225, in get 550323-p, serial = self._storage.load(oid, self._version) 550324- File /usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py, line 1196, in load 550325-return self._storage.load(oid, self._base_version) 550326- File /usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZEO/ClientStorage.py, line 727, in load 550327-return self.loadEx(oid, version)[:2] 550328- File /usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZEO/ClientStorage.py, line 750, in loadEx 550329-data, tid, ver = self._server.loadEx(oid, version) 550330- File /usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZEO/ServerStub.py, line 196, in loadEx 550331-return self.rpc.call(loadEx, oid, version) 550332- File /usr/local/zope/agecom-virtual/eggs/ZODB3-3.7.3_polling-py2.4-linux-x86_64.egg/ZEO/zrpc/connection.py, line 650, in call 550333-raise inst # error raised by server 550334:POSKeyError: 0x06294e This errros is today errors, there are repeated (10) errors like that: 835744:2010-01-30T09:26:47 WARNING relstorage POSKeyError on oid 1793877: no tid found; Current transaction is 253262639700997905; Recent object tids: [] 835745--- 835746-2010-01-30T09:26:47 ERROR ZODB.Connection Couldn't load state for 0x1b5f55 835747-Traceback (most recent call last): 835748- File /usr/local/zope/agecom-virtual/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py, line 847, in setstate 835749-self._setstate(obj) 835750- File /usr/local/zope/agecom-virtual/eggs/ZODB3-3.8.4_polling-py2.4-linux-x86_64.egg/ZODB/Connection.py, line 888, in _setstate 835751-p, serial = self._storage.load(obj._p_oid, self._version) 835752- File /usr/local/zope/agecom-virtual/eggs/RelStorage-1.4.0b1-py2.4.egg/relstorage/storage.py, line 432, in load 835753:raise POSKeyError(oid) 835754:POSKeyError: 0x1b5f55 835756-2010-01-30T09:26:48 INFO Archetypes ESC[00mESC[01;32m/usr/local/zope/agecom-virtual/parts/plone/Archetypes/UIDCatalog.py[106]:getObject 835757-ESC[00mUIDCatalogBrains getObject raised an error: 835758- Traceback (most recent call last): 835759- 835760- File /usr/local/zope/agecom-virtual/parts/plone/Archetypes/UIDCatalog.py, line 87, in getObject 835761-path = self.getPath() 835762- 835763- File /usr/local/zope/agecom-virtual/parts/zope2/lib/python/Products/ZCatalog/CatalogBrains.py, line 33, in getPath 835764-return self.aq_parent.getpath(self.data_record_id_)
Re: [ZODB-Dev] free space for FileStorage, and possible issues with repozo (was: Missing Content)
On Sat, Jan 30, 2010 at 6:18 AM, grahamperrin g.j.per...@bton.ac.uk wrote: Michael Havard wrote: We're missing 16 days worth of content. Python 2.3.5, Zope 2.8.4, Plone 2.1.2, and a host of other products. ZEO configuration. Persistent disk caching is on. Yesterday everything was fine. All the content was there and accessible. I recall reading about a case that involved quietly running out of free space on the volume used for FileStorage. Something like … the most recent content was in RAM — and the running ZEO cluster seemed to work fine with that (no adverse effect on the end user, at the time) — but that content was not committed to disk (and so, was probably unrecoverable following a restart of the OS). If you remove an open file storage (on a Unix-like system), the file still exists, but it is no longer accessible from the original directory. New transactions are committed to this still existing file, not ram. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] free space for FileStorage, and possible issues with repozo (was: Missing Content)
On Sat, Jan 30, 2010 at 12:52:30PM -0500, Jim Fulton wrote: On Sat, Jan 30, 2010 at 6:18 AM, grahamperrin g.j.per...@bton.ac.uk wrote: Michael Havard wrote: We're missing 16 days worth of content. Python 2.3.5, Zope 2.8.4, Plone 2.1.2, and a host of other products. ZEO configuration. Persistent disk caching is on. Yesterday everything was fine. All the content was there and accessible. I recall reading about a case that involved quietly running out of free space on the volume used for FileStorage. Something like … the most recent content was in RAM — and the running ZEO cluster seemed to work fine with that (no adverse effect on the end user, at the time) — but that content was not committed to disk (and so, was probably unrecoverable following a restart of the OS). If you remove an open file storage (on a Unix-like system), the file still exists, but it is no longer accessible from the original directory. New transactions are committed to this still existing file, not ram. It is also possible to resurrect the deleted file as long as the process keeping it open doesn't quit: 1. find the pid and fd with lsof 2. cat /proc/$pid/fd/$fd /new/path (sadly, ln doesn't seem to work -- Invalid cross-device link -- so you need twice as much disk space) Marius Gedminas -- Only great masters of style can succeed in being obtuse. -- Oscar Wilde Most UNIX programmers are great masters of style. -- The Unnamed Usenetter signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
