Re: [ZODB-Dev] RelStorage zodbconvert, ConflictError on Zope2 startup
On Thu, Jul 14, 2011 at 3:28 PM, Sean Upton wrote: >>> Full traceback: http://pastie.org/2214036 > I am able to avoid this by commenting out cache-servers and > cache-module-name in my zope.conf. Looks like the ConflictError at start-up is self-inflicted: I never stopped memcached to purge it of traces of the old/previous relstorage database/instance I was running in my tests. I was running zodbconvert with --clear, but not specifying use of memcached for zodbconvert destination storage ZConfig values, only for my zope2 instance configurations. Lesson learned: purge memcached of any previous data via restart if you plan to replace an existing RelStorage storage with zodbconvert --clear. Sean ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage zodbconvert, ConflictError on Zope2 startup
On Thu, Jul 14, 2011 at 1:43 PM, Hanno Schlichting wrote: > On Thu, Jul 14, 2011 at 9:32 PM, Sean Upton wrote: >> Full traceback: http://pastie.org/2214036 > That's weird, you are getting a conflict error on inserting the root > application object (oid 0x00). I am able to avoid this by commenting out cache-servers and cache-module-name in my zope.conf. I am now looking at how memcached integration seems to break for me (memcached is running on a unix socket with correct file permissions; I am using cache-servers in zope.conf, and I am not in my config file for zodbconvert). Sean ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to fix "CorruptedDataError: Error reading unknown oid. Found '' at 81036527"?
On Thu, Jul 14, 2011 at 3:58 PM, Hanno Schlichting wrote: > On Thu, Jul 14, 2011 at 9:40 PM, Jim Fulton wrote: >> On Thu, Jul 14, 2011 at 3:23 PM, Hanno Schlichting wrote: >>> At Jarn we've used that trick many times to repair broken internals in >>> the intid/keyreference data structures. >> >> Do you have any theories why objects are going away for you? > > Not on a low-level. > > I know the culprit is five.intid which basically register > zope.lifecycle event subscribers for all IPersistent objects to add > and remove intid registrations. > > That code seems to get things wrong. But I've never dug into the code > to figure out under what circumstances this happens. I very much > believe this is wrong application code in five.intid. It does have to > do some funky tricks with raising NotYet exceptions and handling those > at transaction boundaries, as the intid is calculated from the p_oid. > A new object doesn't have a p_oid until the object is added to the > connection. > > So this is rather tricky code dealing with low-level assumptions. But > I never got a reproducible case. So I've just fixed the invalid data > whenever I hit it and ripped out five.intid from every project where > possible. To my knowledge no such problems exist in > zope.intid/zope.keyreference. Is the symptom you're seeing POSKeyError? Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to fix "CorruptedDataError: Error reading unknown oid. Found '' at 81036527"?
On Thu, Jul 14, 2011 at 9:40 PM, Jim Fulton wrote: > On Thu, Jul 14, 2011 at 3:23 PM, Hanno Schlichting wrote: >> At Jarn we've used that trick many times to repair broken internals in >> the intid/keyreference data structures. > > Do you have any theories why objects are going away for you? Not on a low-level. I know the culprit is five.intid which basically register zope.lifecycle event subscribers for all IPersistent objects to add and remove intid registrations. That code seems to get things wrong. But I've never dug into the code to figure out under what circumstances this happens. I very much believe this is wrong application code in five.intid. It does have to do some funky tricks with raising NotYet exceptions and handling those at transaction boundaries, as the intid is calculated from the p_oid. A new object doesn't have a p_oid until the object is added to the connection. So this is rather tricky code dealing with low-level assumptions. But I never got a reproducible case. So I've just fixed the invalid data whenever I hit it and ripped out five.intid from every project where possible. To my knowledge no such problems exist in zope.intid/zope.keyreference. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage zodbconvert, ConflictError on Zope2 startup
On Thu, Jul 14, 2011 at 9:32 PM, Sean Upton wrote: > ZODB.POSException.ConflictError: database conflict error (oid 0x00, > class persistent.mapping.PersistentMapping, serial this txn started > with 0x038fa89e48c295cc 2011-07-13 14:22:17.053148, serial currently > committed 0x038b077ae26bec77 2010-12-19 21:46:53.067557) > > Full traceback: http://pastie.org/2214036 > > Any ideas on what I'm doing wrong or what's going on? Where should I > look into the RelStorage tables for clues? That's weird, you are getting a conflict error on inserting the root application object (oid 0x00). So for some reason the startup code cannot load the existing root object, thinks the database is empty and tries to start fresh. I'm not sure how that could happen, unless the ZODB connection strings are wrong or database permissions not quite right. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to fix "CorruptedDataError: Error reading unknown oid. Found '' at 81036527"?
On Thu, Jul 14, 2011 at 3:23 PM, Hanno Schlichting wrote: > On Thu, Jul 14, 2011 at 5:38 PM, Andreas Jung wrote: >> For the sake of completeness: I found this >> >> https://mail.zope.org/pipermail/zodb-dev/2008-February/011606.html >> >> and will try it out . > > There's also instructions in my pastebin dump on how to do this - and > quite a bit simpler than Chris version: > > from persistent import Persistent > a = Persistent() > a._p_oid = '\x00\x00\x00\x00\x00\xc9-w' > a._p_jar = app._p_jar > app._p_jar._register(a) > app._p_jar._added[a._p_oid] = a > transaction.commit() > > At Jarn we've used that trick many times to repair broken internals in > the intid/keyreference data structures. Do you have any theories why objects are going away for you? The only time we've seen this is when we packed a multi-database incorrectly. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RelStorage zodbconvert, ConflictError on Zope2 startup
I'm a bit stumped with a problem with zodbconvert from RelStorage 1.5.0 (ZODB 3.9.5, Plone 4.0, Zope2 2.2.19). I get ConflictError on Zope2 startup on a just-converted ZODB. I tried converting with blobs and without; from FileStorage and from ClientStorage sources of the same database -- all to the same end when I try to use it. Has anyone seen anything like this at Zope2 startup? ZODB.POSException.ConflictError: database conflict error (oid 0x00, class persistent.mapping.PersistentMapping, serial this txn started with 0x038fa89e48c295cc 2011-07-13 14:22:17.053148, serial currently committed 0x038b077ae26bec77 2010-12-19 21:46:53.067557) Full traceback: http://pastie.org/2214036 Any ideas on what I'm doing wrong or what's going on? Where should I look into the RelStorage tables for clues? Thanks, Sean sdup...@gmail.com | sean.up...@hsc.utah.edu ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to fix "CorruptedDataError: Error reading unknown oid. Found '' at 81036527"?
On Thu, Jul 14, 2011 at 5:38 PM, Andreas Jung wrote: > For the sake of completeness: I found this > > https://mail.zope.org/pipermail/zodb-dev/2008-February/011606.html > > and will try it out . There's also instructions in my pastebin dump on how to do this - and quite a bit simpler than Chris version: from persistent import Persistent a = Persistent() a._p_oid = '\x00\x00\x00\x00\x00\xc9-w' a._p_jar = app._p_jar app._p_jar._register(a) app._p_jar._added[a._p_oid] = a transaction.commit() At Jarn we've used that trick many times to repair broken internals in the intid/keyreference data structures. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to fix "CorruptedDataError: Error reading unknown oid. Found '' at 81036527"?
Jim Fulton wrote: I wouldn't normally suggest what I'm about to suggest, but I think you have enough ZODB foo. :) I would try adding a new bucket with that oid. Maybe you can even stuff it in the connection cache to avoid committing it to the database. I'm not sure off hand what the steps required would be, but I think you can figure it out. Once you get an object in the cache, you should be able to load the container, modify it to not include the subobject, and commit it. For the sake of completeness: I found this https://mail.zope.org/pipermail/zodb-dev/2008-February/011606.html and will try it out . Andreas <>___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to fix "CorruptedDataError: Error reading unknown oid. Found '' at 81036527"?
On Thu, Jul 14, 2011 at 10:12 AM, Andreas Jung wrote:
>
>
> Jim Fulton wrote:
>>>
>>> 2011-07-14 13:36:22 INFO Z2 Shutting down
>>
>> That error message seems rather weird. WTF SIGINT?
>
> Likely caused during the Zope shutdown amd unrelated).
I find this a bit unlikely, unless there was also a system shut down.
FileStorage goes to great pains to prevent this sort of corruption.
I don't think it's possible for a process shut down alone to
cause this.
I could imagine this sort of corruption if the system was shut
down before data could be synced to physical disk.
>>
>> Use multi-zodb-check-refs with the -r option to get a database of
>> reverse references.
>> Use that do figure out what's refering to the missing object and fix
>> that object(s) so that
>> it no longer does.
>
> This gives me one reference:
>
> plone@web1:~/poskeyerror-check$ cat Out
for x in refs.items(): print x
> ...
> ('1', )
That's sort of interesting. So my guess is that the missing object is
a bucket and was the first bucket in the BTree, otherwise there'd also
be a reference from the previous bucket.
>
>
> I followed the documentation at
>
> http://pastebin.com/bL0CbBm2
I don't know where you got that. I didn't write it. :)
> So what do I do with the oid 0xe7a850 in this case?
Well, ideally, you'd update it to not include a reference to the
broken object.
> Trying to load the referenced object gives me of course a PosKey error:
Dang. I would want it to give you a broken object. I think there
might be a launchpad issue for that. Not sure if 3.10 has the fix
(not sure what you're using).
I wouldn't normally suggest what I'm about to suggest, but I think you
have enough ZODB foo. :)
I would try adding a new bucket with that oid. Maybe you can even
stuff it in the connection cache to avoid committing it to the
database. I'm not sure off hand what the steps required would be, but
I think you can figure it out. Once you get an object in the cache,
you should be able to load the container, modify it to not include the
subobject, and commit it.
You may also need to follow references to the container, using the
reverse-references database to get a handle on what was happening at
the application level.
Alternatively, if you think this was due to a non-graceful shutdown
while committing, there might be a transaction you can undo.
Jim
--
Jim Fulton
http://www.linkedin.com/in/jimfulton
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to fix "CorruptedDataError: Error reading unknown oid. Found '' at 81036527"?
On Thu, Jul 14, 2011 at 4:12 PM, Andreas Jung wrote: > I followed the documentation at > > http://pastebin.com/bL0CbBm2 Wow, looks like I didn't specify an expiration date for my paste ;) If anyone feels like putting that into a more permanent place, feel free to use it. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to fix "CorruptedDataError: Error reading unknown oid. Found '' at 81036527"?
Jim Fulton wrote:
2011-07-14 13:36:22 INFO Z2 Shutting down
That error message seems rather weird. WTF SIGINT?
Likely caused during the Zope shutdown amd unrelated).
Use multi-zodb-check-refs with the -r option to get a database of
reverse references.
Use that do figure out what's refering to the missing object and fix
that object(s) so that
it no longer does.
This gives me one reference:
plone@web1:~/poskeyerror-check$ cat Out
>>> for x in refs.items(): print x
...
('1', )
I followed the documentation at
http://pastebin.com/bL0CbBm2
So what do I do with the oid 0xe7a850 in this case?
Trying to load the referenced object gives me of course a PosKey error:
>>> app = refs
>>> p64
>>> app
{'Application': '\x00\x00\x00\x00\x00\x00\x00\x01'>, 'ZGlobals': object at 0x2a6f7d0>}
>>> app._p_jar.get(p64(0xe7a850))
Traceback (most recent call last):
File "", line 1, in
File
"/export/plone/poskeyerror-check/lib/python2.6/site-packages/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZODB/Connection.py",
line 248, in get
p, serial = self._storage.load(oid, '')
File
"/export/plone/poskeyerror-check/lib/python2.6/site-packages/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZODB/FileStorage/FileStorage.py",
line 412, in load
pos = self._lookup_pos(oid)
File
"/export/plone/poskeyerror-check/lib/python2.6/site-packages/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZODB/FileStorage/FileStorage.py",
line 403, in _lookup_pos
raise POSKeyError(oid)
ZODB.POSException.POSKeyError: 0xe7a850
Andreas
<>___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to fix "CorruptedDataError: Error reading unknown oid. Found '' at 81036527"?
On Thu, Jul 14, 2011 at 7:54 AM, Andreas Jung wrote: > I have a customer site (Plone 3.0.6) causing an error during packing: > > Module ZPublisher.Publish, line 119, in publish > Module ZPublisher.mapply, line 88, in mapply > Module ZPublisher.Publish, line 42, in call_object > Module , line 3, in _facade > Module AccessControl.requestmethod, line 64, in _curried > Module App.ApplicationManager, line 431, in manage_pack > Module ZODB.DB, line 624, in pack > Module ZEO.ClientStorage, line 846, in pack > Module ZEO.ServerStub, line 165, in pack > Module ZEO.zrpc.connection, line 650, in call > CorruptedDataError: Error reading unknown oid. Found '' at 81036527 > ^C2011-07-14 13:36:22 INFO SignalHandler Caught signal SIGINT > 2011-07-14 13:36:22 INFO Z2 Shutting down That error message seems rather weird. WTF SIGINT? > > Using multi-zodb-gc reveals an error: > > plone@web1:~/poskeyerror-check$ bin/multi-zodb-gc storage.cfg > > > plone@web1:~/poskeyerror-check$ bin/multi-zodb-check-refs storage.cfg > > > Using fscover.py does not help here.. > > plone@web1:~/poskeyerror-check$ bin/python > ./lib/python2.6/site-packages/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZODB/fsrecover.py > Data.fs Data2.fs > Recovering Data.fs into Data2.fs > No handlers could be found for logger "ZODB.FileStorage" > . 1 . 2 . 3 . 4 . 5 . 6 . 7 . 8 . 9 . 0 > 0 bytes removed during recovery > > Any more options? Use multi-zodb-check-refs with the -r option to get a database of reverse references. Use that do figure out what's refering to the missing object and fix that object(s) so that it no longer does. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] How to fix "CorruptedDataError: Error reading unknown oid. Found '' at 81036527"?
I have a customer site (Plone 3.0.6) causing an error during packing: Module ZPublisher.Publish, line 119, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 42, in call_object Module , line 3, in _facade Module AccessControl.requestmethod, line 64, in _curried Module App.ApplicationManager, line 431, in manage_pack Module ZODB.DB, line 624, in pack Module ZEO.ClientStorage, line 846, in pack Module ZEO.ServerStub, line 165, in pack Module ZEO.zrpc.connection, line 650, in call CorruptedDataError: Error reading unknown oid. Found '' at 81036527 ^C2011-07-14 13:36:22 INFO SignalHandler Caught signal SIGINT 2011-07-14 13:36:22 INFO Z2 Shutting down Using multi-zodb-gc reveals an error: plone@web1:~/poskeyerror-check$ bin/multi-zodb-gc storage.cfg plone@web1:~/poskeyerror-check$ bin/multi-zodb-check-refs storage.cfg Using fscover.py does not help here.. plone@web1:~/poskeyerror-check$ bin/python ./lib/python2.6/site-packages/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZODB/fsrecover.py Data.fs Data2.fs Recovering Data.fs into Data2.fs No handlers could be found for logger "ZODB.FileStorage" . 1 . 2 . 3 . 4 . 5 . 6 . 7 . 8 . 9 . 0 0 bytes removed during recovery Any more options? Andreas <>___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
