On Thu, Apr 22, 2010 at 1:19 PM, Jürgen Herrmann
<juergen.herrm...@xlhost.de> wrote:
> On Thu, 22 Apr 2010 12:54:55 -0400, Jim Fulton <j...@zope.com> wrote:
>> On Thu, Apr 22, 2010 at 9:00 AM, Jürgen Herrmann
>> <juergen.herrm...@xlhost.de> wrote:
>>> hi there!
>>> today i ran a check script which iterates over approx 150000 objects
>>> and does some sanity check calculations on these. during this loop
>>> i saw the zope process use up to about 4.5gb memory. the database
>>> has ~3.5million objects in it.
>>> i set the zodb cache size for the mount point in question to 10000
>>> objects. obviously this limit is not honoured during one transaction:
>> Right.
>>> so two questions here:
>>> - would the byte-limited zodb cache setting help here?
>> No. The limits aren't really limits. They are more like
>> suggestions. :)  In particular, they are only enforced at
>> transaction (or subtransaction) boundaries, when a connection is
>> opened or closed, or when applications call certain APIs explicitly.
> are these apis exposed somewhere? calling these periodically sounds
> cleaner than the aproach below.
>>> - if no - how can i iterate over a big set of objects without
>>>  forcing them to stay in the cache for the whole transaction?
>>>  after all i just need each object once during the iteration.
>> As you're iterating through the objects, if you know you aren't going
>> to need an object again or otherwise think it would be OK to free it
>> from memory, you can call _p_deactivate on it. If the object hasn't
>> been modified, then it's state will be released and it will become a
>> ghost.
> as my objects contain references to OOBTrees, do i also have to call
> _p_deactivate() on these explicitly, or is it enough to call it on the
> containing object?

You need to call it on any objects you want to go away.

Another option is to periodically call cacheGC() on the
connection object:


This will apply the limits, to the degree it can. As with _p_deactivate,
only unmodified objects will be removed from memory.  I'm assuming in this
discussion that you're only analyzing, not updating objects.


Jim Fulton
Zope-Dev maillist  -  Zope-Dev@zope.org
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope )

Reply via email to