Re: [ZODB-Dev] cache not minimized at transaction boundaries?
Dieter Maurer wrote: I plan to implement that -- waiting for the new style contributor agreement from the Zope Foundation ;-) Yay! Go Dieter! cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] cache not minimized at transaction boundaries?
Hey Toby, Long time no "see" ;-) Toby Dickenson wrote: On Tuesday 31 January 2006 11:41, Chris Withers wrote: Tim Peters wrote: ...base ZODB cache targets on aggregate pickle size All good, but still never gonna happen, right? *wry grinz* Sounds like a reasonable evening project (Ive no chance for ZODB fun in the day job any more), but Im sure it wont ever be the *most* interesting option without some sponsorship. Chris, do you have a real business requirement, or is this just an idle wish? Sadly, for me, not a real business requirement... I know how the cache works and so can make allowances. For newbies, it does make death-by-swap an all too common phenomenon if they don't see the connection between object size and memory used by the cache... How much work in terms of days/money do you reckon it would take? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] cache not minimized at transaction boundaries?
Tim Peters wrote at 2006-1-31 01:32 -0500: >[Chris Withers] >> ... >> ...oh well, if only the ZODB cache was RAM-usage-based ratehr than object >> count based ;-) > >Ya, that's not feasible. More plausible would be to base ZODB cache targets >on aggregate pickle size; ZODB _could_ know that, and then it would also be >strongly related to how a ZEO cache's size is measured (the ZEO cache size >limit is given in bytes, and by far the bulk of those bytes are consumed by >pickles). I plan to implement that -- waiting for the new style contributor agreement from the Zope Foundation ;-) -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] cache not minimized at transaction boundaries?
On Tuesday 31 January 2006 11:41, Chris Withers wrote: > Tim Peters wrote: > >> ...base ZODB cache targets on aggregate pickle size > All good, but still never gonna happen, right? *wry grinz* Sounds like a reasonable evening project (Ive no chance for ZODB fun in the day job any more), but Im sure it wont ever be the *most* interesting option without some sponsorship. Chris, do you have a real business requirement, or is this just an idle wish? -- Toby Dickenson ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] cache not minimized at transaction boundaries?
[Chris Withers] >>> ... >>> ...oh well, if only the ZODB cache was RAM-usage-based ratehr than object >>> count based ;-) [Tim Peters] >> Ya, that's not feasible. More plausible would be to base ZODB cache targets >> on aggregate pickle size; ZODB _could_ know that, and then it would also be >> strongly related to how a ZEO cache's size is measured (the ZEO cache size >> limit is given in bytes, and by far the bulk of those bytes are consumed by >> pickles). [Chris Withers] > All good, but still never gonna happen, right? *wry grinz* Depends on whether someone cares enough to do the work and the salesmanship. Best guess is that ZC isn't going to pay an employee to do this. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] cache not minimized at transaction boundaries?
Tim Peters wrote: [Chris Withers] ... ...oh well, if only the ZODB cache was RAM-usage-based ratehr than object count based ;-) Ya, that's not feasible. More plausible would be to base ZODB cache targets on aggregate pickle size; ZODB _could_ know that, and then it would also be strongly related to how a ZEO cache's size is measured (the ZEO cache size limit is given in bytes, and by far the bulk of those bytes are consumed by pickles). All good, but still never gonna happen, right? *wry grinz* Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] cache not minimized at transaction boundaries?
[Chris Withers] > ... > ...oh well, if only the ZODB cache was RAM-usage-based ratehr than object > count based ;-) Ya, that's not feasible. More plausible would be to base ZODB cache targets on aggregate pickle size; ZODB _could_ know that, and then it would also be strongly related to how a ZEO cache's size is measured (the ZEO cache size limit is given in bytes, and by far the bulk of those bytes are consumed by pickles). ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] cache not minimized at transaction boundaries?
Hi Tim, Tim Peters wrote: Do: import ZODB print ZODB.__version__ to find out. Good to know, thanks... I have a Stepper (zopectl run on steroids) job that deals with lots of big objects. Can you quantify this? 60,000 File objects of the order of 2Mb each. It does not do cacheMinimize(). It tries to reduce the memory cache to the target number of objects specified for that cache, which is not at all the same as cache minimization (which latter shoots for a target size of 0). Whether that's "sane" or not depends on the product of: the cache's target number of objects times: "the average" byte size of an object Ah, that'll do it, I wondered why it was only this step that was hurting. My guess is that our cache size settings with lots of max-sized PData objects lead to the RAM blowup... ...oh well, if only the ZODB cache was RAM-usage-based ratehr than object count based ;-) thanks for the info! Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] cache not minimized at transaction boundaries?
[Chris Withers] > This is with whatever ZODB ships with Zope 2.8.5... Do: import ZODB print ZODB.__version__ to find out. > I have a Stepper (zopectl run on steroids) job that deals with lots of > big objects. Can you quantify this? > After processing each one, Stepper does a transaction.get().commit(). Note that "transaction.commit()" is a shortcut spelling. > I thought this was enough to keep the object cache at a sane size, It does not do cacheMinimize(). It tries to reduce the memory cache to the target number of objects specified for that cache, which is not at all the same as cache minimization (which latter shoots for a target size of 0). Whether that's "sane" or not depends on the product of: the cache's target number of objects times: "the average" byte size of an object ZODB has no say of its own about either of those. > however the job kept bombing out with MemoryErrors, and sure enough it > was using 2 or 3 gigs of memory when that happened. > > I fiddled about with the gc module and found that, sure enough, object > were being kept in memory. At a guess, I inserted something close to the > following: > > obj._p_jar.db().cacheMinimize() > > ...after each 5,000 objects were processed (there are 60,000 objects in > total) > > Lo and behold, memory usage became sane. > > Why is this step necessary? I thought transaction.get().commit() every so > often was enough to sort out the cache... See above. For most people it works OK. If `cn` is the Connection, then cn._cache.cache_size is the target number of non-ghost objects cn._cache.ringlen() is the current number of non-ghost objects At a transaction boundary, the cache gc method run tries to make ringlen() <= cache_size, and that's all. For example, using all defaults: >>> ZODB.__version__ # probably the version you're using '3.4.2' This loads a million-element OOBTree (the construction of which I won't show here): >>> len(t) 100 The number of non-ghost objects is then approximately 1e6/15 (the number of leaf-node OOBuckets in that tree; there are more than that because of non-leaf interior OOBTree nodes, but the leaf nodes account for the bulk of it): >>> cn._cache.cache_size, cn._cache.ringlen() (400, 67067) At a transaction boundary, a cache gc pass is run to try to reduce the number of non-ghost objects to cache_size: >>> transaction.commit() >>> cn._cache.cache_size, cn._cache.ringlen() (400, 400) So it booted 67067 - 400 = 7 non-ghost objects. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
