A very simple alternative to prefetching would be to load the whole DB into memory indiscriminately, if it is configured to do so. This way, you can store your catalog in a separate db and request all of it from the ZEO server and cache it straight away.
I'm still partial to a memcached cache that can survive a restart. The first prize would be if it's possible to share the cache between zeo clients. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za On Fri, Mar 8, 2013 at 7:50 PM, Laurence Rowe <l...@lrowe.co.uk> wrote: > On 8 March 2013 09:38, Claudiu Saftoiu <csaft...@gmail.com> wrote: >> On Fri, Mar 8, 2013 at 12:31 PM, Leonardo Santagada <santag...@gmail.com> >> wrote: >>> >>> >>> On Fri, Mar 8, 2013 at 2:17 PM, Claudiu Saftoiu <csaft...@gmail.com> >>> wrote: >>>> >>>> Once I know the difference I'll probably be able to answer this myself, >>>> but I wonder why the ZEO server doesn't do the sort of caching that allow >>>> the client to operate so quickly on the indices once they are loaded. >>> >>> >>> IIRC zeo not only takes bytes from the storage and put them on a socket, >>> it has a kind of heavy protocol for sending objects that has overhead on >>> each object, so lots of small objects (that are 400mb in size) take a lot >>> more time than sending a 400mb blob. >> >> >> Ah that would make perfect sense. So ZEO and catalog indices really don't >> mix well at all. > > The slowdown is largely because ZODB only loads objects one at a time. > Loading a large catalogue requires paying that latency (network + > software) each time, a 400mb of catalogue data may well equate to > something like 10000 objects, and therefore 10000 loads in series. > Once the data is loaded into the object cache you only need to fetch > invalidated objects. > > It would be great if there was a way to advise ZODB in advance that > certain objects would be required so it could fetch multiple object > states in a single request to the storage server. > > Laurence > _______________________________________________ > For more information about ZODB, see http://zodb.org/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > https://mail.zope.org/mailman/listinfo/zodb-dev _______________________________________________ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
