[Zope-dev] zope2/zodb cache size question
hi there! today i did some sanity check calcs on a set of approx 15 objects in one of our zodb. for these calculations i had to iterate over all of these objects. during this loop memory footprint of the zope process grew to ~5gb and the cache summary showed for the zodb connection which was used: Connection at 24c10410263104 462834 -- XLhost.de - eXperts in Linux hosting ® XLhost.de GmbH Jürgen Herrmann, Geschäftsführer Boelckestrasse 21, 93051 Regensburg, Germany Geschäftsführer: Volker Geith, Jürgen Herrmann Registriert unter: HRB9918 Umsatzsteuer-Identifikationsnummer: DE245931218 Fon: +49 (0)800 XLHOSTDE [0800 95467833] Fax: +49 (0)800 95467830 WEB: http://www.XLhost.de IRC: #xlh...@irc.quakenet.org ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] zope2/zodb cache size question
hi there! today i ran a check script which iterates over approx 15 objects and does some sanity check calculations on these. during this loop i saw the zope process use up to about 4.5gb memory. the database has ~3.5million objects in it. i set the zodb cache size for the mount point in question to 1 objects. obviously this limit is not honoured during one transaction: connection active objects total objects ... Connection at 24c10410 263104 462834 ... so two questions here: - would the byte-limited zodb cache setting help here? - if no - how can i iterate over a big set of objects without forcing them to stay in the cache for the whole transaction? after all i just need each object once during the iteration. thanks in advance and best regards, jürgen herrmann -- XLhost.de - eXperts in Linux hosting ® XLhost.de GmbH Jürgen Herrmann, Geschäftsführer Boelckestrasse 21, 93051 Regensburg, Germany Geschäftsführer: Volker Geith, Jürgen Herrmann Registriert unter: HRB9918 Umsatzsteuer-Identifikationsnummer: DE245931218 Fon: +49 (0)800 XLHOSTDE [0800 95467833] Fax: +49 (0)800 95467830 WEB: http://www.XLhost.de IRC: #xlh...@irc.quakenet.org ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope2/zodb cache size question
On Thu, 22 Apr 2010 14:56:31 +0200, Jürgen Herrmann juergen.herrm...@xlhost.de wrote: hi there! today i did some sanity check calcs on a set of approx 15 objects in one of our zodb. for these calculations i had to iterate over all of these objects. during this loop memory footprint of the zope process grew to ~5gb and the cache summary showed for the zodb connection which was used: Connection at 24c10410263104 462834 please ignore this msg, i hit the send button and the mail was not finished yet. ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope2/zodb cache size question
On Thu, Apr 22, 2010 at 03:00:16PM +0200, Jürgen Herrmann wrote: today i ran a check script which iterates over approx 15 objects and does some sanity check calculations on these. during this loop i saw the zope process use up to about 4.5gb memory. the database has ~3.5million objects in it. i set the zodb cache size for the mount point in question to 1 objects. obviously this limit is not honoured during one transaction: connection active objects total objects ... Connection at 24c10410 263104 462834 ... so two questions here: - would the byte-limited zodb cache setting help here? - if no - how can i iterate over a big set of objects without forcing them to stay in the cache for the whole transaction? after all i just need each object once during the iteration. Use savepoints: for n, obj in enumerate(your_objects): perform_sanity_check(obj) if n % 1 == 0: transaction.savepoint() Real-world example: http://bazaar.launchpad.net/~schooltool-owners/schooltool/schooltool/annotate/head:/src/schooltool/generations/evolve26.py Marius Gedminas -- http://pov.lt/ -- Zope 3 consulting and development signature.asc Description: Digital signature ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope2/zodb cache size question
On Thu, Apr 22, 2010 at 9:00 AM, Jürgen Herrmann juergen.herrm...@xlhost.de wrote: hi there! today i ran a check script which iterates over approx 15 objects and does some sanity check calculations on these. during this loop i saw the zope process use up to about 4.5gb memory. the database has ~3.5million objects in it. i set the zodb cache size for the mount point in question to 1 objects. obviously this limit is not honoured during one transaction: Right. so two questions here: - would the byte-limited zodb cache setting help here? No. The limits aren't really limits. They are more like suggestions. :) In particular, they are only enforced at transaction (or subtransaction) boundaries, when a connection is opened or closed, or when applications call certain APIs explicitly. - if no - how can i iterate over a big set of objects without forcing them to stay in the cache for the whole transaction? after all i just need each object once during the iteration. As you're iterating through the objects, if you know you aren't going to need an object again or otherwise think it would be OK to free it from memory, you can call _p_deactivate on it. If the object hasn't been modified, then it's state will be released and it will become a ghost. Jim -- Jim Fulton ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope2/zodb cache size question
On Thu, 22 Apr 2010 12:54:55 -0400, Jim Fulton j...@zope.com wrote: On Thu, Apr 22, 2010 at 9:00 AM, Jürgen Herrmann juergen.herrm...@xlhost.de wrote: hi there! today i ran a check script which iterates over approx 15 objects and does some sanity check calculations on these. during this loop i saw the zope process use up to about 4.5gb memory. the database has ~3.5million objects in it. i set the zodb cache size for the mount point in question to 1 objects. obviously this limit is not honoured during one transaction: Right. so two questions here: - would the byte-limited zodb cache setting help here? No. The limits aren't really limits. They are more like suggestions. :) In particular, they are only enforced at transaction (or subtransaction) boundaries, when a connection is opened or closed, or when applications call certain APIs explicitly. are these apis exposed somewhere? calling these periodically sounds cleaner than the aproach below. - if no - how can i iterate over a big set of objects without forcing them to stay in the cache for the whole transaction? after all i just need each object once during the iteration. As you're iterating through the objects, if you know you aren't going to need an object again or otherwise think it would be OK to free it from memory, you can call _p_deactivate on it. If the object hasn't been modified, then it's state will be released and it will become a ghost. as my objects contain references to OOBTrees, do i also have to call _p_deactivate() on these explicitly, or is it enough to call it on the containing object? thanks for your help! best regards, jürgen -- XLhost.de - eXperts in Linux hosting ® XLhost.de GmbH Jürgen Herrmann, Geschäftsführer Boelckestrasse 21, 93051 Regensburg, Germany Geschäftsführer: Volker Geith, Jürgen Herrmann Registriert unter: HRB9918 Umsatzsteuer-Identifikationsnummer: DE245931218 Fon: +49 (0)800 XLHOSTDE [0800 95467833] Fax: +49 (0)800 95467830 WEB: http://www.XLhost.de IRC: #xlh...@irc.quakenet.org ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope2/zodb cache size question
On Thu, Apr 22, 2010 at 1:19 PM, Jürgen Herrmann juergen.herrm...@xlhost.de wrote: On Thu, 22 Apr 2010 12:54:55 -0400, Jim Fulton j...@zope.com wrote: On Thu, Apr 22, 2010 at 9:00 AM, Jürgen Herrmann juergen.herrm...@xlhost.de wrote: hi there! today i ran a check script which iterates over approx 15 objects and does some sanity check calculations on these. during this loop i saw the zope process use up to about 4.5gb memory. the database has ~3.5million objects in it. i set the zodb cache size for the mount point in question to 1 objects. obviously this limit is not honoured during one transaction: Right. so two questions here: - would the byte-limited zodb cache setting help here? No. The limits aren't really limits. They are more like suggestions. :) In particular, they are only enforced at transaction (or subtransaction) boundaries, when a connection is opened or closed, or when applications call certain APIs explicitly. are these apis exposed somewhere? calling these periodically sounds cleaner than the aproach below. - if no - how can i iterate over a big set of objects without forcing them to stay in the cache for the whole transaction? after all i just need each object once during the iteration. As you're iterating through the objects, if you know you aren't going to need an object again or otherwise think it would be OK to free it from memory, you can call _p_deactivate on it. If the object hasn't been modified, then it's state will be released and it will become a ghost. as my objects contain references to OOBTrees, do i also have to call _p_deactivate() on these explicitly, or is it enough to call it on the containing object? You need to call it on any objects you want to go away. Another option is to periodically call cacheGC() on the connection object: some_object._p_jar.cacheGC() This will apply the limits, to the degree it can. As with _p_deactivate, only unmodified objects will be removed from memory. I'm assuming in this discussion that you're only analyzing, not updating objects. Jim -- Jim Fulton ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
