On Fri, May 6, 2011 at 10:14 PM, Shane Hathaway <sh...@hathawaymix.org> wrote:
> From my experience, most people who want ZODB to be faster want Zope
> catalogs in particular to be faster. I don't think prefetching can make
> catalogs much faster, though.
I've spent a lot of time lately on making ZCatalog faster. The main
trick there is to store data in smarter ways, load fewer objects in
the first place and trying to minimize data sets as early as possible,
so the cost of intersection() and union() gets lower. There's a lot
more you can do about optimizing ZCatalog, but prefetching would
indeed not help much.
The only cases where you could do prefetching are the ones you don't
want to do anyways, like loading an entire BTree or TreeSet, because
you need to do a len(tree) or actually iterate over the entire thing.
All that said, if you hit large datasets, it gets problematic to do
catalog operations on each Zope client. At some point a centralized
query approach on the server side or via a web API wins in terms of
overall resource efficiency.
For more information about ZODB, see the ZODB Wiki:
ZODB-Dev mailing list - ZODB-Dev@zope.org