[ZODB-Dev] How to fix CorruptedDataError: Error reading unknown oid. Found '' at 81036527?
I have a customer site (Plone 3.0.6) causing an error during packing: Module ZPublisher.Publish, line 119, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 42, in call_object Module string, line 3, in _facade Module AccessControl.requestmethod, line 64, in _curried Module App.ApplicationManager, line 431, in manage_pack Module ZODB.DB, line 624, in pack Module ZEO.ClientStorage, line 846, in pack Module ZEO.ServerStub, line 165, in pack Module ZEO.zrpc.connection, line 650, in call CorruptedDataError: Error reading unknown oid. Found '' at 81036527 ^C2011-07-14 13:36:22 INFO SignalHandler Caught signal SIGINT 2011-07-14 13:36:22 INFO Z2 Shutting down Using multi-zodb-gc reveals an error: plone@web1:~/poskeyerror-check$ bin/multi-zodb-gc storage.cfg zc.zodbdgc.Bad instance at 0x20578c0 plone@web1:~/poskeyerror-check$ bin/multi-zodb-check-refs storage.cfg Using fscover.py does not help here.. plone@web1:~/poskeyerror-check$ bin/python ./lib/python2.6/site-packages/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZODB/fsrecover.py Data.fs Data2.fs Recovering Data.fs into Data2.fs No handlers could be found for logger ZODB.FileStorage . 1 . 2 . 3 . 4 . 5 . 6 . 7 . 8 . 9 . 0 0 bytes removed during recovery Any more options? Andreas attachment: lists.vcf___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to fix CorruptedDataError: Error reading unknown oid. Found '' at 81036527?
On Thu, Jul 14, 2011 at 7:54 AM, Andreas Jung li...@zopyx.com wrote: I have a customer site (Plone 3.0.6) causing an error during packing: Module ZPublisher.Publish, line 119, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 42, in call_object Module string, line 3, in _facade Module AccessControl.requestmethod, line 64, in _curried Module App.ApplicationManager, line 431, in manage_pack Module ZODB.DB, line 624, in pack Module ZEO.ClientStorage, line 846, in pack Module ZEO.ServerStub, line 165, in pack Module ZEO.zrpc.connection, line 650, in call CorruptedDataError: Error reading unknown oid. Found '' at 81036527 ^C2011-07-14 13:36:22 INFO SignalHandler Caught signal SIGINT 2011-07-14 13:36:22 INFO Z2 Shutting down That error message seems rather weird. WTF SIGINT? Using multi-zodb-gc reveals an error: plone@web1:~/poskeyerror-check$ bin/multi-zodb-gc storage.cfg zc.zodbdgc.Bad instance at 0x20578c0 plone@web1:~/poskeyerror-check$ bin/multi-zodb-check-refs storage.cfg Using fscover.py does not help here.. plone@web1:~/poskeyerror-check$ bin/python ./lib/python2.6/site-packages/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZODB/fsrecover.py Data.fs Data2.fs Recovering Data.fs into Data2.fs No handlers could be found for logger ZODB.FileStorage . 1 . 2 . 3 . 4 . 5 . 6 . 7 . 8 . 9 . 0 0 bytes removed during recovery Any more options? Use multi-zodb-check-refs with the -r option to get a database of reverse references. Use that do figure out what's refering to the missing object and fix that object(s) so that it no longer does. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to fix CorruptedDataError: Error reading unknown oid. Found '' at 81036527?
Jim Fulton wrote:
2011-07-14 13:36:22 INFO Z2 Shutting down
That error message seems rather weird. WTF SIGINT?
Likely caused during the Zope shutdown amd unrelated).
Use multi-zodb-check-refs with the -r option to get a database of
reverse references.
Use that do figure out what's refering to the missing object and fix
that object(s) so that
it no longer does.
This gives me one reference:
plone@web1:~/poskeyerror-check$ cat Out
for x in refs.items(): print x
...
('1', BTrees.LOBTree.LOBTree object at 0xe7a850)
I followed the documentation at
http://pastebin.com/bL0CbBm2
So what do I do with the oid 0xe7a850 in this case?
Trying to load the referenced object gives me of course a PosKey error:
app = refs
p64
function p64 at 0x26ab2a8
app
{'Application': persistent broken OFS.Application.Application instance
'\x00\x00\x00\x00\x00\x00\x00\x01', 'ZGlobals': BTrees.OOBTree.OOBTree
object at 0x2a6f7d0}
app._p_jar.get(p64(0xe7a850))
Traceback (most recent call last):
File stdin, line 1, in module
File
/export/plone/poskeyerror-check/lib/python2.6/site-packages/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZODB/Connection.py,
line 248, in get
p, serial = self._storage.load(oid, '')
File
/export/plone/poskeyerror-check/lib/python2.6/site-packages/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZODB/FileStorage/FileStorage.py,
line 412, in load
pos = self._lookup_pos(oid)
File
/export/plone/poskeyerror-check/lib/python2.6/site-packages/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZODB/FileStorage/FileStorage.py,
line 403, in _lookup_pos
raise POSKeyError(oid)
ZODB.POSException.POSKeyError: 0xe7a850
Andreas
attachment: lists.vcf___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to fix CorruptedDataError: Error reading unknown oid. Found '' at 81036527?
On Thu, Jul 14, 2011 at 4:12 PM, Andreas Jung li...@zopyx.com wrote: I followed the documentation at http://pastebin.com/bL0CbBm2 Wow, looks like I didn't specify an expiration date for my paste ;) If anyone feels like putting that into a more permanent place, feel free to use it. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to fix CorruptedDataError: Error reading unknown oid. Found '' at 81036527?
On Thu, Jul 14, 2011 at 10:12 AM, Andreas Jung li...@zopyx.com wrote:
Jim Fulton wrote:
2011-07-14 13:36:22 INFO Z2 Shutting down
That error message seems rather weird. WTF SIGINT?
Likely caused during the Zope shutdown amd unrelated).
I find this a bit unlikely, unless there was also a system shut down.
FileStorage goes to great pains to prevent this sort of corruption.
I don't think it's possible for a process shut down alone to
cause this.
I could imagine this sort of corruption if the system was shut
down before data could be synced to physical disk.
Use multi-zodb-check-refs with the -r option to get a database of
reverse references.
Use that do figure out what's refering to the missing object and fix
that object(s) so that
it no longer does.
This gives me one reference:
plone@web1:~/poskeyerror-check$ cat Out
for x in refs.items(): print x
...
('1', BTrees.LOBTree.LOBTree object at 0xe7a850)
That's sort of interesting. So my guess is that the missing object is
a bucket and was the first bucket in the BTree, otherwise there'd also
be a reference from the previous bucket.
I followed the documentation at
http://pastebin.com/bL0CbBm2
I don't know where you got that. I didn't write it. :)
So what do I do with the oid 0xe7a850 in this case?
Well, ideally, you'd update it to not include a reference to the
broken object.
Trying to load the referenced object gives me of course a PosKey error:
Dang. I would want it to give you a broken object. I think there
might be a launchpad issue for that. Not sure if 3.10 has the fix
(not sure what you're using).
I wouldn't normally suggest what I'm about to suggest, but I think you
have enough ZODB foo. :)
I would try adding a new bucket with that oid. Maybe you can even
stuff it in the connection cache to avoid committing it to the
database. I'm not sure off hand what the steps required would be, but
I think you can figure it out. Once you get an object in the cache,
you should be able to load the container, modify it to not include the
subobject, and commit it.
You may also need to follow references to the container, using the
reverse-references database to get a handle on what was happening at
the application level.
Alternatively, if you think this was due to a non-graceful shutdown
while committing, there might be a transaction you can undo.
Jim
--
Jim Fulton
http://www.linkedin.com/in/jimfulton
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to fix CorruptedDataError: Error reading unknown oid. Found '' at 81036527?
On Thu, Jul 14, 2011 at 5:38 PM, Andreas Jung li...@zopyx.com wrote: For the sake of completeness: I found this https://mail.zope.org/pipermail/zodb-dev/2008-February/011606.html and will try it out . There's also instructions in my pastebin dump on how to do this - and quite a bit simpler than Chris version: from persistent import Persistent a = Persistent() a._p_oid = '\x00\x00\x00\x00\x00\xc9-w' a._p_jar = app._p_jar app._p_jar._register(a) app._p_jar._added[a._p_oid] = a transaction.commit() At Jarn we've used that trick many times to repair broken internals in the intid/keyreference data structures. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to fix CorruptedDataError: Error reading unknown oid. Found '' at 81036527?
On Thu, Jul 14, 2011 at 3:23 PM, Hanno Schlichting ha...@hannosch.eu wrote: On Thu, Jul 14, 2011 at 5:38 PM, Andreas Jung li...@zopyx.com wrote: For the sake of completeness: I found this https://mail.zope.org/pipermail/zodb-dev/2008-February/011606.html and will try it out . There's also instructions in my pastebin dump on how to do this - and quite a bit simpler than Chris version: from persistent import Persistent a = Persistent() a._p_oid = '\x00\x00\x00\x00\x00\xc9-w' a._p_jar = app._p_jar app._p_jar._register(a) app._p_jar._added[a._p_oid] = a transaction.commit() At Jarn we've used that trick many times to repair broken internals in the intid/keyreference data structures. Do you have any theories why objects are going away for you? The only time we've seen this is when we packed a multi-database incorrectly. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage zodbconvert, ConflictError on Zope2 startup
On Thu, Jul 14, 2011 at 9:32 PM, Sean Upton sdup...@gmail.com wrote: ZODB.POSException.ConflictError: database conflict error (oid 0x00, class persistent.mapping.PersistentMapping, serial this txn started with 0x038fa89e48c295cc 2011-07-13 14:22:17.053148, serial currently committed 0x038b077ae26bec77 2010-12-19 21:46:53.067557) Full traceback: http://pastie.org/2214036 Any ideas on what I'm doing wrong or what's going on? Where should I look into the RelStorage tables for clues? That's weird, you are getting a conflict error on inserting the root application object (oid 0x00). So for some reason the startup code cannot load the existing root object, thinks the database is empty and tries to start fresh. I'm not sure how that could happen, unless the ZODB connection strings are wrong or database permissions not quite right. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to fix CorruptedDataError: Error reading unknown oid. Found '' at 81036527?
On Thu, Jul 14, 2011 at 9:40 PM, Jim Fulton j...@zope.com wrote: On Thu, Jul 14, 2011 at 3:23 PM, Hanno Schlichting ha...@hannosch.eu wrote: At Jarn we've used that trick many times to repair broken internals in the intid/keyreference data structures. Do you have any theories why objects are going away for you? Not on a low-level. I know the culprit is five.intid which basically register zope.lifecycle event subscribers for all IPersistent objects to add and remove intid registrations. That code seems to get things wrong. But I've never dug into the code to figure out under what circumstances this happens. I very much believe this is wrong application code in five.intid. It does have to do some funky tricks with raising NotYet exceptions and handling those at transaction boundaries, as the intid is calculated from the p_oid. A new object doesn't have a p_oid until the object is added to the connection. So this is rather tricky code dealing with low-level assumptions. But I never got a reproducible case. So I've just fixed the invalid data whenever I hit it and ripped out five.intid from every project where possible. To my knowledge no such problems exist in zope.intid/zope.keyreference. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage zodbconvert, ConflictError on Zope2 startup
On Thu, Jul 14, 2011 at 1:43 PM, Hanno Schlichting ha...@hannosch.eu wrote: On Thu, Jul 14, 2011 at 9:32 PM, Sean Upton sdup...@gmail.com wrote: Full traceback: http://pastie.org/2214036 That's weird, you are getting a conflict error on inserting the root application object (oid 0x00). I am able to avoid this by commenting out cache-servers and cache-module-name in my zope.conf. I am now looking at how memcached integration seems to break for me (memcached is running on a unix socket with correct file permissions; I am using cache-servers in zope.conf, and I am not in my config file for zodbconvert). Sean ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
