Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-10 Thread Chris Withers
Jim Fulton wrote: My intuition is still that sharing objects between threads will introduce a host of subtle bugs. Yes, I'll +lots to this. I'm much more interested in seeing a memory-limited cache of some description an sad to see this thread derail into sharing data between threads, which

Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-10 Thread Pascal Peregrina
[EMAIL PROTECTED], zodb-dev@zope.org Objet : Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache Jim Fulton wrote: My intuition is still that sharing objects between threads will introduce a host of subtle bugs. Yes, I'll +lots to this. I'm much more interested in seeing

Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-09 Thread Dieter Maurer
Jim Fulton wrote at 2006-10-6 17:04 -0400: ... Thread-safety is not an issue for read-only data. Yes it is. An object that is read-only from a persistence point of view isn't read-only at the Python level. Okay: thread-safety is not an issue for (truely) read-only data. It is potentially an

Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-09 Thread Jim Fulton
Dieter Maurer wrote: Jim Fulton wrote at 2006-10-6 17:04 -0400: ... Or consider object activation and deactivation. If a ghost is shared among multiple threads, then __setstate__ could be called from separate threads. But, why should this be a problem? They would install the same state.

Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-09 Thread Tim Peters
... |Jim Fulton] Or consider object activation and deactivation. If a ghost is shared among multiple threads, then __setstate__ could be called from separate threads. [Dieter Maurer] But, why should this be a problem? They would install the same state. __setstate__ implementations are

Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-06 Thread Chris Withers
Dieter Maurer wrote: The pickle size is a *VERY* rough estimation (probably wrong by a factor of 5 to 15) But, as you point out, much better than a hard coded 1 ;-) We probably would get a much better estimation using PySizer but probably at a significantly higher cost. Right, I guess

Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-06 Thread Jim Fulton
Dieter Maurer wrote: Jim Fulton wrote at 2006-10-6 12:21 -0400: ... We would have a use case for this, too. We've started moving toward just using a single application thread per process (with many processes). There isn't really much advantage in running multiple threads if you have multiple

Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-06 Thread Dieter Maurer
Jim Fulton wrote at 2006-10-6 15:13 -0400: Dieter Maurer wrote: Jim Fulton wrote at 2006-10-6 12:21 -0400: ... We would have a use case for this, too. We've started moving toward just using a single application thread per process (with many processes). There isn't really much advantage in

Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-06 Thread Jim Fulton
Dieter Maurer wrote: ... The only benefit of multiple threads is that it is somewhat less likely that expensive requests will block inexpensive ones. Multiple threads can share resources such as main memory and a common cache (proposed by the original poster). If there is only one thread,

Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-06 Thread Dieter Maurer
Jim Fulton wrote at 2006-10-6 16:18 -0400: Dieter Maurer wrote: ... The only benefit of multiple threads is that it is somewhat less likely that expensive requests will block inexpensive ones. Multiple threads can share resources such as main memory and a common cache (proposed by the

Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-06 Thread Jim Fulton
Dieter Maurer wrote: Jim Fulton wrote at 2006-10-6 16:18 -0400: Dieter Maurer wrote: ... ... We abused Zope a bit and have build a desktop application with it. One of the main critiques of our customers is too high memory consumption. Many have computers with 256 MB memory and they

Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-05 Thread Jim Fulton
Dieter Maurer wrote: A new proposal: http://www.zope.org/Wikis/ZODB/MemorySizeLimitedCache It outlines how to implement a ZODB cache limited not be the number of containing objects but by their estimated memory size. Feedback welcome -- either here or in the Wiki. I'm going to reply here

Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-05 Thread Pascal Peregrina
 : Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache Dieter Maurer wrote: A new proposal: http://www.zope.org/Wikis/ZODB/MemorySizeLimitedCache It outlines how to implement a ZODB cache limited not be the number of containing objects but by their estimated memory size

Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-05 Thread Dieter Maurer
Jim Fulton wrote at 2006-10-5 08:46 -0400: ... I'll note that, as a guiding principle, any refactoring we do should allow pure-python implementation. This means that APIs need to be Python APIs, although we should consider efficient C implementations when designing these APIs. I