Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-27 Thread Matt Hamilton
Jim Fulton jim at zope.com writes: On Wed, Jan 26, 2011 at 3:15 PM, Matt Hamilton matth at netsight.co.uk wrote: So, with up to 300,000 items in some of these IISets, it means to iterate over the entire set (during a Catalog query) means loading 5,000 objects over ZEO from the ZODB,

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-27 Thread Hanno Schlichting
Hi. On Thu, Jan 27, 2011 at 9:00 AM, Matt Hamilton ma...@netsight.co.uk wrote: Alas we are. Or rather, alas, ZCatalog does ;) It would be great if it didn't but it's just the way it is. If I have 300,000 items in my site, and everyone of them visible to someone with the 'Reader' role, then

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-27 Thread Matt Hamilton
Hanno Schlichting hanno at hannosch.eu writes: You are using queryplan in the site, right? The most typical catalog query for Plone consists of something like ('allowedRolesAndUsers', 'effectiveRange', 'path', 'sort_on'). Without queryplan you indeed load the entire tree (or trees inside

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-27 Thread Hanno Schlichting
On Thu, Jan 27, 2011 at 11:09 AM, Matt Hamilton ma...@netsight.co.uk wrote: Hanno Schlichting hanno at hannosch.eu writes: There still seem to be instances in which the entire set is loaded.  This could be an artifact of the fact I am clearing the ZODB cache before each ]test, which I think

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-26 Thread Stefan H. Holek
On 26.01.2011, at 21:15, Matt Hamilton wrote: I'm not sure how the current MAX_BUCKET_SIZE values were determined, but looks like they have been the same since the dawn of time, and I'm guessing might be due a tune? Go Matt go! :-) -- Stefan H. Holek ste...@epy.co.at

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-26 Thread Jim Fulton
On Wed, Jan 26, 2011 at 3:15 PM, Matt Hamilton ma...@netsight.co.uk wrote: All,  I have been doing some performance investigation into a large Plone site we have running. The site in question has approx 300,000 items of content. Each piece of content is indexed by ZCatalog. The main thing I

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-26 Thread Roché Compaan
On Wed, 2011-01-26 at 20:15 +, Matt Hamilton wrote: So... has anyone tried increasing the size of MAX_BUCKET_SIZE in real life? I tried back in 2007 but didn't see a big difference. https://mail.zope.org/pipermail/zodb-dev/2007-November/011279.html My benchmark was focussed on writes not