Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-27 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/27/2011 05:09 AM, Matt Hamilton wrote: >> Conflicts inside the same bucket can be resolved and you won't get to >> see any log message for them. If you get a ConflictError in the logs, >> it's one where the request is being retried. > > Great.

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-27 Thread Hanno Schlichting
On Thu, Jan 27, 2011 at 11:09 AM, Matt Hamilton wrote: > Hanno Schlichting hannosch.eu> writes: > > There still seem to be instances in which the entire set is loaded.  This > could be an artifact of the fact I am clearing the ZODB cache before each > ]test, which I think seems to be clearing the

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-27 Thread Matt Hamilton
Hanno Schlichting hannosch.eu> writes: > You are using queryplan in the site, right? The most typical catalog > query for Plone consists of something like ('allowedRolesAndUsers', > 'effectiveRange', 'path', 'sort_on'). Without queryplan you indeed > load the entire tree (or trees inside allowedR

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-27 Thread Hanno Schlichting
Hi. On Thu, Jan 27, 2011 at 9:00 AM, Matt Hamilton wrote: > Alas we are. Or rather, alas, ZCatalog does ;) It would be great if it > didn't but it's just the way it is. If I have 300,000 items in my > site, and everyone of them visible to someone with the 'Reader' > role, then the allowedRolesAnd

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-27 Thread Matt Hamilton
Jim Fulton zope.com> writes: > > On Wed, Jan 26, 2011 at 3:15 PM, Matt Hamilton > netsight.co.uk> wrote: > > So, with up to 300,000 items in some of these IISets, it means to > > iterate over the entire set (during a Catalog query) means loading > > 5,000 objects over ZEO from the ZODB, which

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-26 Thread Roché Compaan
On Wed, 2011-01-26 at 20:15 +, Matt Hamilton wrote: > So... has anyone tried increasing the size of MAX_BUCKET_SIZE in real > life? I tried back in 2007 but didn't see a big difference. https://mail.zope.org/pipermail/zodb-dev/2007-November/011279.html My benchmark was focussed on writes not

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-26 Thread Jim Fulton
On Wed, Jan 26, 2011 at 3:15 PM, Matt Hamilton wrote: > All, >  I have been doing some performance investigation into a large Plone > site we have running. The site in question has approx 300,000 items of > content. Each piece of content is indexed by ZCatalog. > > The main thing I was tracking do

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-26 Thread Stefan H. Holek
On 26.01.2011, at 21:15, Matt Hamilton wrote: > I'm not sure how the current MAX_BUCKET_SIZE values were determined, > but looks like they have been the same since the dawn of time, and I'm > guessing might be due a tune? Go Matt go! :-) -- Stefan H. Holek ste...@epy.co.at ___

[ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-26 Thread Matt Hamilton
All, I have been doing some performance investigation into a large Plone site we have running. The site in question has approx 300,000 items of content. Each piece of content is indexed by ZCatalog. The main thing I was tracking down was the very large number of objects being loaded by the ZODB,