Re: [Zope-dev] RAMCacheManager and object size
Dieter Maurer wrote: Florent Guillaume wrote at 2005-7-8 20:36 +0200: The RAMCacheManager does a costly pseudo-pickling of the objects it stores to compute their size, but that information is only used in the statistics screen. I replaced it by the following code: [...] That's a fine compromise. Are you going to check it in? Shane ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RAMCacheManager and object size
Florent Guillaume wrote: The RAMCacheManager does a costly pseudo-pickling of the objects it stores to compute their size, but that information is only used in the statistics screen. The motivation was actually more subtle: I wanted to prevent applications from caching things that weren't self-contained. As I saw it, I would have otherwise spent countless hours debugging cache problems, and the caching framework would have ultimately been rejected as too fragile--similar to the fate of the Refresh product. Catalog results in particular are an obvious thing to cache, but they aren't safe for caching because they link back to the catalog. You'd have major thread problems and probably inconsistent results. The pickle machinery throws an exception when it hits many kinds of objects that aren't self-contained, so it made a good match. The only waste is the time spent building a string. It was only lucky that the string could also be used for estimating memory consumption. I think RAMCacheManager should, by default, continue to make a reasonable attempt to ensure objects are safe for caching, but it might be good to also have an advanced option that disables the filter. 2. or, how about using the size to have a cache threshold based on the size. That would help manage memory consumption better. Has anybody done something like that? I don't think that has been attempted, but I agree it would be interesting. Shane ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RAMCacheManager and object size
On Sun, Jul 10, 2005 at 12:18:03AM -0600, Shane Hathaway wrote: | Catalog results in particular are an obvious thing to cache, but they | aren't safe for caching because they link back to the catalog. You'd | have major thread problems and probably inconsistent results. Would using thread.local help here? -- Sidnei da Silva Enfold Systems, LLC. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RAMCacheManager and object size
Sidnei da Silva wrote: On Sun, Jul 10, 2005 at 12:18:03AM -0600, Shane Hathaway wrote: | Catalog results in particular are an obvious thing to cache, but they | aren't safe for caching because they link back to the catalog. You'd | have major thread problems and probably inconsistent results. Would using thread.local help here? I don't think so. You want either a shared cache (like RAMCacheManager) or a per-database-connection cache (which would let you cache catalog results.) Database connections are not bound to a specific thread. However, you might be on to something: a switch that turns RAMCacheManager into a per-database-connection cache would be useful for things that currently aren't cacheable. Shane ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RAMCacheManager and object size
On Sun, Jul 10, 2005 at 09:32:29AM -0600, Shane Hathaway wrote: | Would using thread.local help here? | | I don't think so. You want either a shared cache (like RAMCacheManager) | or a per-database-connection cache (which would let you cache catalog | results.) Database connections are not bound to a specific thread. | | However, you might be on to something: a switch that turns | RAMCacheManager into a per-database-connection cache would be useful for | things that currently aren't cacheable. Glad to have sparkled some idea ;) -- Sidnei da Silva Enfold Systems, LLC. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RAMCacheManager and object size
Florent Guillaume wrote at 2005-7-8 20:36 +0200: The RAMCacheManager does a costly pseudo-pickling of the objects it stores to compute their size, but that information is only used in the statistics screen. I replaced it by the following code: try: from cPickle import Pickler, HIGHEST_PROTOCOL except: from pickle import Pickler, HIGHEST_PROTOCOL ... class CacheEntry: ''' Represents a cached value. ''' def __init__(self, index, data, view_name): try: # This is a protective barrier that hopefully prevents # us from caching something that might result in memory # leaks. It's also convenient for determining the # approximate memory usage of the cache entry. # DM 2004-11-29: this code causes excessive time. # Note also that it does not prevent us from # caching objects with references to persistent objects # When we do, nasty persistency errors are likely # to occur (shouldn't load data while connection is closed). #self.size = len(dumps(index)) + len(dumps(data)) sizer = _ByteCounter() pickler = Pickler(sizer, HIGHEST_PROTOCOL) pickler.dump(index) pickler.dump(data) self.size = sizer.getCount() class _ByteCounter: '''auxiliary file like class which just counts the bytes written.''' _count = 0 def write(self, bytes): self._count += len(bytes) def getCount(self): return self._count -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
