I'm the primary zc.catalog author.

He is running read-only, so savepoints shouldn't be very pertinent.

The original poster is correct that the cache is cleaned out at transaction boundaries (or savepoints, as Alan said).

I don't have experience with running the db in readonly mode in production. That said, if you have obvious boundaries for your searches, I suspect that beginning a transaction at the boundary would probably do the trick, and also work (with invalidations and the like) if you ever have a readonly connection to a ZEO database server that is getting updates from another source.

While it's interesting to contemplate the knob that Alan mentioned, I'm reasonably confident that this logic belongs in the app, not the index or catalog.

Gary

On Jul 31, 2008, at 11:55 AM, Alan Runyan wrote:

I believe it is common to have a theshold (say every 10000) objects to
do a savepoint
and then you can run cacheMinimize after the savepoint.

surprised the zc.catalog does not provide this sort of knob. maybe it
was a problem
in the original ZCatalog implementation?

cheers
alan

On Wed, Jul 30, 2008 at 1:21 PM,  <[EMAIL PROTECTED]> wrote:

I have a database consisting mainly of an IOBTree of ~700,000 items (each persistent mappings), and an zc.catalog indexing them by one of the mapping's fields (a date). I am opening the FileStorage in read_only mode. For each day in the index, I get the day's mappings and read the contents of another field. As I go through each day, the memory usage explodes (over 32 GB). Is there a way to configure the cache to automatically keep itself
under the value of the cache size parameter?

To avoid this problem in most cases, I wrap the IOBTree in another object which does nothing more than call db.cacheMinimize after every 10000 items are iterated over. But for random access, that's not an option.

From reading the archives, it sounds like cache cleaning does not happen while running a transaction. Is that my problem? I'm in read-only mode, so I can't perform a transaction anyway, but could the ZODB think I
want to?  Or is my problem something else?

Maybe if someone can point me to a description of how the caching
works.

--
Anthony Foglia
Princeton Consultants
(609) 987-8787 x233
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev





--
Alan Runyan
Enfold Systems, Inc.
http://www.enfoldsystems.com/
phone: +1.713.942.2377x111
fax: +1.832.201.8856
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Reply via email to