Re: [ZODB-Dev] Estrange behaviour with BTreeFolder2
On Wed, Apr 01, 2009 at 11:46:01AM +0200, Santi Camps wrote:
I'm having memory problems and, after some debugging with gc, I see
some estrange behaviours with BTreeFolder2 (or with BTrees itself, I'm
not sure). I've a BTreeFolder2 with 4 objects, more o less.
Some of them are emails. Just accessing the container, about 700
mails are loaded in memory !!
from AccessControl.SecurityManagement import newSecurityManager
from Testing.makerequest import makerequest
user = app.beta.acl_users.getUser('manager').__of__(app.beta.acl_users)
newSecurityManager({}, user)
app = makerequest(app)
km = app.beta
km.portal_repository._tree
import gc
objects = gc.get_objects()
objects2 = [obj for obj in objects if getattr(obj, '__class__', None)]
(I'm a bit surprised that you get objects without a __class__ attribute
-- could you elaborate about those? Old-style class instances?
Extension types?)
emails = [obj for obj in objects2 if 'mail' in obj.__class__.__name__.lower()]
How many different classes are there with 'mail' in the (lowercased)
class name?
Do they all inherit from Persistent?
print len(emails)
697
Following gc references, I see this mails are referenced from a
OOBucket, that is referenced by another OOBucket, ...
I assume those mails are values rather than keys.
and some of them
are referenced from persistent.PickleCache (instead I've configured
cache-size = 0 in zope.conf). Seems that some buckets and its
referenced objects are readed during btree loading. So, the garbage
collector never clears this from memory, and after some hours of
intensive work, the server RAM is finished
Is that normal ? I'm making some mistake ? Anybody knows any way to solve
it ?
Your description would make perfect sense to me if your mail classes
weren't subclasses of Persistent (and the fix would be: make them
subclass Persistent).
Marius Gedminas
--
I used to (somewhat) sneer at people who describe themselves thus: I program
HTML. Then I tried to make a web-site look as I wanted it to (ie. not
grotesquely ugly), wow - perhaps they were onto something. Some think there is
a pile of broken mess in our desktop, but at least you know where you are with
a GtkHBox.
-- Michael Meeks
signature.asc
Description: Digital signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Estrange behaviour with BTreeFolder2
objects = gc.get_objects() objects2 = [obj for obj in objects if getattr(obj, '__class__', None)] (I'm a bit surprised that you get objects without a __class__ attribute -- could you elaborate about those? Old-style class instances? Extension types?) No, it's just a copy paste of a old memory debugging thread, sorry emails = [obj for obj in objects2 if 'mail' in obj.__class__.__name__.lower()] How many different classes are there with 'mail' in the (lowercased) class name? Do they all inherit from Persistent? Just one class, it inherit from Persistent, yes. Following other classes results in the same problem Following gc references, I see this mails are referenced from a OOBucket, that is referenced by another OOBucket, ... I assume those mails are values rather than keys. Yes, the keys are strings with the id's of the objects Your description would make perfect sense to me if your mail classes weren't subclasses of Persistent (and the fix would be: make them subclass Persistent). No, all them inherit from CMF PortalContent, and this from Persistent, so there is not the problem. I'm using RelStorage, but this shouldn't affect. I will try the same tests in a FileStorage to be sure Thanks for your help -- Santi Camps (Earcon S.L.) http://www.earcon.com http://www.kmkey.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Estrange behaviour with BTreeFolder2
I'm using RelStorage, but this shouldn't affect. I will try the same tests in a FileStorage to be sure Hi all It seems to be a Relstorage specific issue. The same tests using FileStorage doesn't produce memory garbage ... It could be something related to poll-interval parameter ? I've leaved default values there Thanks again -- Santi Camps (Earcon S.L.) http://www.earcon.com http://www.kmkey.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Estrange behaviour with BTreeFolder2
Hmm, I don't know how RelStorage could affect the pickle cache; the two have no direct interaction at all. Are you using any ZODB patches other than the polling patch? Shane Hi Shane No, just the Relstorage polling patch I'm not saying that pickle cache was affected, I'm saying that loading a BTreeFolder2 seems very affected. In a FileStorage system, objects contained in the BTreeFolder are not loaded when the BTreeFolder itself is loaded, but in RelStorage, a lot of contained objects are loaded just accessing the container ¿? After that, this objects are never removed from memory, but the question is why they are loaded. Any ideas ? Thanks -- Santi Camps (Earcon S.L.) http://www.earcon.com http://www.kmkey.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Estrange behaviour with BTreeFolder2
Santi Camps wrote: I'm not saying that pickle cache was affected, I'm saying that loading a BTreeFolder2 seems very affected. In a FileStorage system, objects contained in the BTreeFolder are not loaded when the BTreeFolder itself is loaded, but in RelStorage, a lot of contained objects are loaded just accessing the container ¿? After that, this objects are never removed from memory, but the question is why they are loaded. That should not happen, and I'm not aware of any reason why it would. There are automated tests that verify that does not happen, in fact, and they always pass. So I need a lot more help to reproduce this. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
