On Thu, Feb 09, 2012 at 02:26:10AM +0100, Kaweh Kazemi wrote: > Thanks a lot for the investigation, Marius. That was awesome. > > Seems to me, I need to upgrade to Python 2.7 and be done with the > problem then. :/
I wonder, though, if ZODB ought to use the 'noload' method of
cPickle.Unpickler, which is undocumented and broken on 2.6 (and gone on
3.x, AFAIU).
> > Incidentally, since you use cross-database references, please make sure
> > they continue to work after you pack your storage. I've lost data that
> > way (the ZODB garbage collector doesn't see references that exist in
> > other storages, and can assume objects are garbage when it shouldn't).
> > Packing with GC disabled ought to be safe.
>
> Cross-database references are only one-way in our implementation.
> "users" storage is referencing objects in the "game" storage - never
> vice versa / no circles. Also we make sure we never delete any objects
> in *any* storage, just "disable" them on an application level. This
> worked fine so far. Are you seeing any problems with this?
I don't use cross-database references. My only encounter with them was
on Zope 2.x: I uploaded a new version of a website from a .zexp file
into /temp_folder, tested it briefly there, then used the ZMI to move it
into /. It worked fine for a day, and was broken the next day. Turns
out /temp_folder in Zope 2 uses a different backend storage, and moving
the folder to root just created a cross-db reference, but didn't move
the objects across. Then the garbage collector for the /temp_folder
storage noticed that my site was not directly reachable from
/temp_folder's root dict, and collected it away as garbage.
I do now know if that bug is still present in any version of Zope or
ZODB; I just learned not to trust cross-db references. Once burned
twice shy, as they say. I'm not saying you will encounter any
garbage-collection related problems; I'm saying I would double-check to
be sure if I was in your position.
> Disabling GC during pack works (and was a workaround for a while), but
> the difference between non-GC (4G) and GC pack (1GB) are 3GB, which is
> quite a lot and growing too fast. We just don't need all the
> references from old object states to persist.
Packing always removes old object states.
Garbage collection also removes new object states for objects that are
no longer reachable.
Does zeopack consider cross-database references correctly when it checks
for reachability? I would hope so, but I don't really know. It would
only work if you're packing all the storages together, presumably.
Marius Gedminas
--
Well, there's a quantum computer that can factor 15, so those of you using
4-bit RSA should worry.
-- Bruce Schneier
signature.asc
Description: Digital signature
_______________________________________________ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
