A very simple alternative to prefetching would be to load the whole DB
into memory indiscriminately, if it is configured to do so. This way,
you can store your catalog in a separate db and request all of it from
the ZEO server and cache it straight away.
I'm still partial to a memcached cache that can survive a restart. The
first prize would be if it's possible to share the cache between zeo
Upfront Systems http://www.upfrontsystems.co.za
On Fri, Mar 8, 2013 at 7:50 PM, Laurence Rowe <l...@lrowe.co.uk> wrote:
> On 8 March 2013 09:38, Claudiu Saftoiu <csaft...@gmail.com> wrote:
>> On Fri, Mar 8, 2013 at 12:31 PM, Leonardo Santagada <santag...@gmail.com>
>>> On Fri, Mar 8, 2013 at 2:17 PM, Claudiu Saftoiu <csaft...@gmail.com>
>>>> Once I know the difference I'll probably be able to answer this myself,
>>>> but I wonder why the ZEO server doesn't do the sort of caching that allow
>>>> the client to operate so quickly on the indices once they are loaded.
>>> IIRC zeo not only takes bytes from the storage and put them on a socket,
>>> it has a kind of heavy protocol for sending objects that has overhead on
>>> each object, so lots of small objects (that are 400mb in size) take a lot
>>> more time than sending a 400mb blob.
>> Ah that would make perfect sense. So ZEO and catalog indices really don't
>> mix well at all.
> The slowdown is largely because ZODB only loads objects one at a time.
> Loading a large catalogue requires paying that latency (network +
> software) each time, a 400mb of catalogue data may well equate to
> something like 10000 objects, and therefore 10000 loads in series.
> Once the data is loaded into the object cache you only need to fetch
> invalidated objects.
> It would be great if there was a way to advise ZODB in advance that
> certain objects would be required so it could fetch multiple object
> states in a single request to the storage server.
> For more information about ZODB, see http://zodb.org/
> ZODB-Dev mailing list - ZODB-Dev@zope.org
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org