Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/27/2011 05:09 AM, Matt Hamilton wrote: >> Conflicts inside the same bucket can be resolved and you won't get to >> see any log message for them. If you get a ConflictError in the logs, >> it's one where the request is being retried. > > Great. That was that I always thought, but just wanted to check. So in > that case, what does it mean if I see a conflict error for an IISet? Can > they not resolve conflicts internally? IIRC: unlike normal adds / removes, bucket splits can cause un-resolvable conflicts (because more than one persistent object's state is involved). In that case, arger buckets should make splits less frequent, and therefore less likely to conflict. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1BnigACgkQ+gerLs4ltQ5nlQCfVxx3V+vtUGWDsyv4iyd/2CRg v0IAn3sfsBbD2iCJ2lwJzU5i5/IK8e+w =xMIT -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc
On Thu, Jan 27, 2011 at 11:09 AM, Matt Hamilton wrote:
> Hanno Schlichting hannosch.eu> writes:
>
> There still seem to be instances in which the entire set is loaded. This
> could be an artifact of the fact I am clearing the ZODB cache before each
> ]test, which I think seems to be clearing the query plan.
Yes. The queryplan is stored in a volatile attribute, so clearing the
zodb cache will throw away the plan. The queryplan version integrated
into Zope 2.13 stores the plan in a module global with thread locks
around it.
> Speaking of
> which I saw in the query plan code, some hook to load a pre-defined query
> plan... but I can't see exactly how you supply this plan or in what format
> it is. Do you use this feature?
You get a plan representation by calling:
http://localhost:8080/Plone/@@catalogqueryplan-prioritymap
Then add an environment variable pointing to a variable inside a module:
[instance]
recipe = plone.recipe.zope2instance
environment-vars =
CATALOGQUERYPLAN my.customer.module.queryplan
Create that module and put the dump in it. it should start with something like:
# query plan dumped at 'Mon May 24 01:33:28 2010'
queryplan = {
'/Plone/portal_catalog': {
...
}
You can keep updating this plan with some new data from the dump once
in a while.
Ideally this plan should be persisted in the database at certain
intervals, but we haven't implemented that yet. You don't want to
persist the plan in every request doing a catalog query.
Hanno
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc
Hanno Schlichting hannosch.eu> writes:
> You are using queryplan in the site, right? The most typical catalog
> query for Plone consists of something like ('allowedRolesAndUsers',
> 'effectiveRange', 'path', 'sort_on'). Without queryplan you indeed
> load the entire tree (or trees inside allowedRolesAndUsers) for each
> of these indexes.
Yes we are using queryplan. Without it the site becomes pretty much
unusable.
> With queryplan it knows from prior execution, that the set returned by
> the path index is the smallest. So it first calculates this. Then it
> uses this small set (usually 10-100 items per folder) to look inside
> the other indexes. It then only needs to do an intersection of the
> small path set with each of the trees. If the path set has less then
> 1000 items, it won't even use the normal intersection function from
> the BTrees module, but use the optimized Cython based version from
> queryplan, which essentially does a for-in loop over the path set.
> Depending on the size ratio between the sets this is up to 20 times
> faster with in-memory data, and even more so if it avoids database
> loads. In the worst case you would load buckets equal to length of the
> path set, usually you should load a lot less.
There still seem to be instances in which the entire set is loaded. This
could be an artifact of the fact I am clearing the ZODB cache before each
]test, which I think seems to be clearing the query plan. Speaking of
which I saw in the query plan code, some hook to load a pre-defined query
plan... but I can't see exactly how you supply this plan or in what format
it is. Do you use this feature?
> We have large Plone sites in the same range of multiple 100.000 items
> and with queryplan and blobs we can run them with ZODB cache sizes of
> less than 100.000 items and memory usage of 500mb per single-threaded
> process.
>
> Of course it would still be really good to optimize the underlying
> data structures, but queryplan should help make this less urgent.
Well, I think we are already at that point ;) There are also I think other
times in which the full set is loaded.
> > Ahh interesting, that is good to know. I've not actually checked the
> > conflict resolution code, but do bucket change conflicts actually get
> > resolved in some sane way, or does the transaction have to be
> > retried?
>
> Conflicts inside the same bucket can be resolved and you won't get to
> see any log message for them. If you get a ConflictError in the logs,
> it's one where the request is being retried.
Great. That was that I always thought, but just wanted to check. So in
that case, what does it mean if I see a conflict error for an IISet? Can
they not resolve conflicts internally?
> >> And imagine if you use zc.zlibstorage to compress records! :)
> >
> > This is Plone 3, which is Zope 2.10.11, does zc.zlibstorage work on
> > that, or does it need newer ZODB?
>
> zc.zlibstorage needs a newer ZODB version. 3.10 and up to be exact.
>
> > Also, unless I can sort out that
> > large number of small pickles being loaded, I'd imagine this would
> > actually slow things down.
>
> The Data.fs would be smaller, making it more likely to fit into the OS
> disk cache. The overhead of uncompressing the data is small compared
> to the cost of a disk read instead of a memory read. But it's hard to
> say what exactly happens with the cache ratio in practice.
Yeah, if we could use it I certainly would :) I guess what I mean above is
that larger pickles would compress better, so lots of small pickles the
compression would be less effective.
-Matt
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc
Hi.
On Thu, Jan 27, 2011 at 9:00 AM, Matt Hamilton wrote:
> Alas we are. Or rather, alas, ZCatalog does ;) It would be great if it
> didn't but it's just the way it is. If I have 300,000 items in my
> site, and everyone of them visible to someone with the 'Reader'
> role, then the allowedRolesAndUsers index will have an IITreeSet
> with 300,000 elements in it. Yes, we could try and optimize out that
> specific case, but there are others like that too. If all of my
> items have no effective or expires date, then the same happens with
> the effective range index (DateRangeIndex 'always' set).
You are using queryplan in the site, right? The most typical catalog
query for Plone consists of something like ('allowedRolesAndUsers',
'effectiveRange', 'path', 'sort_on'). Without queryplan you indeed
load the entire tree (or trees inside allowedRolesAndUsers) for each
of these indexes.
With queryplan it knows from prior execution, that the set returned by
the path index is the smallest. So it first calculates this. Then it
uses this small set (usually 10-100 items per folder) to look inside
the other indexes. It then only needs to do an intersection of the
small path set with each of the trees. If the path set has less then
1000 items, it won't even use the normal intersection function from
the BTrees module, but use the optimized Cython based version from
queryplan, which essentially does a for-in loop over the path set.
Depending on the size ratio between the sets this is up to 20 times
faster with in-memory data, and even more so if it avoids database
loads. In the worst case you would load buckets equal to length of the
path set, usually you should load a lot less.
We have large Plone sites in the same range of multiple 100.000 items
and with queryplan and blobs we can run them with ZODB cache sizes of
less than 100.000 items and memory usage of 500mb per single-threaded
process.
Of course it would still be really good to optimize the underlying
data structures, but queryplan should help make this less urgent.
> Ahh interesting, that is good to know. I've not actually checked the
> conflict resolution code, but do bucket change conflicts actually get
> resolved in some sane way, or does the transaction have to be
> retried?
Conflicts inside the same bucket can be resolved and you won't get to
see any log message for them. If you get a ConflictError in the logs,
it's one where the request is being retried.
>> And imagine if you use zc.zlibstorage to compress records! :)
>
> This is Plone 3, which is Zope 2.10.11, does zc.zlibstorage work on
> that, or does it need newer ZODB?
zc.zlibstorage needs a newer ZODB version. 3.10 and up to be exact.
> Also, unless I can sort out that
> large number of small pickles being loaded, I'd imagine this would
> actually slow things down.
The Data.fs would be smaller, making it more likely to fit into the OS
disk cache. The overhead of uncompressing the data is small compared
to the cost of a disk read instead of a memory read. But it's hard to
say what exactly happens with the cache ratio in practice.
Hanno
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc
Jim Fulton zope.com> writes: > > On Wed, Jan 26, 2011 at 3:15 PM, Matt Hamilton > netsight.co.uk> wrote: > > So, with up to 300,000 items in some of these IISets, it means to > > iterate over the entire set (during a Catalog query) means loading > > 5,000 objects over ZEO from the ZODB, which adds up to quite a bit > > of > > latency. With quite a number of these data structures about, means > > we > > can end up with in the order of 50,000 object in the ZODB cache > > *just* > > for these IISets! > > Hopefully, you're not iterating over the entire tree, but still. :) Alas we are. Or rather, alas, ZCatalog does ;) It would be great if it didn't but it's just the way it is. If I have 300,000 items in my site, and everyone of them visible to someone with the 'Reader' role, then the allowedRolesAndUsers index will have an IITreeSet with 300,000 elements in it. Yes, we could try and optimize out that specific case, but there are others like that too. If all of my items have no effective or expires date, then the same happens with the effective range index (DateRangeIndex 'always' set). > > So... has anyone tried increasing the size of MAX_BUCKET_SIZE in real > > life? > > We have, mainly to reduce the number of conflicts. > > > I understand that this will increase the potential for conflicts > > if the bucket/set size is larger (however in reality this probably > > can't get worse than it is, as currently as the value inserted is 99% > > of the time greater than the current max value stored -- it is a > > timestamp -- you always hit the last bucket/set in the tree). > > Actually, it reduces the number of unresolveable conflicts. > Most conflicting bucket changes can be resolved, but bucket > splits can't be and bigger buckets means fewer splits. > > The main tradeoff is record size. Ahh interesting, that is good to know. I've not actually checked the conflict resolution code, but do bucket change conflicts actually get resolved in some sane way, or does the transaction have to be retried? Actually... that is a good point, and something I never thought of... when you get a Conflict Error in the logs (that was resolved) does that mean that _p_resolveConflict was called and successful, or does it mean that the transactions were retried and that resolved the conflict? > > I was going to experiment with increasing the MAX_BUCKET_SIZE on an IISet > > from 120 to 1200. Doing a quick test, a pickle of an IISet of 60 items > > is around 336 bytes, an of 600 items is 1580 bytes... so still very > > much in the realms of a single disk read / network packet. > > And imagine if you use zc.zlibstorage to compress records! :) This is Plone 3, which is Zope 2.10.11, does zc.zlibstorage work on that, or does it need newer ZODB? Also, unless I can sort out that large number of small pickles being loaded, I'd imagine this would actually slow things down. > > I'm not sure how the current MAX_BUCKET_SIZE values were determined, > > but looks like they have been the same since the dawn of time, and I'm > > guessing might be due a tune? > > Probably. > > > It looks like I can change that constant and recompile the BTree > > package, and it will work fine with existing IISets and just take > > effect on new sets created (ie clear and rebuild the catalog index). > > > > Anyone played with this before or see any major flaws to my cunning plan? > > We have. My long term goal is to arrange things so that you can > specify/change limits by sub-classing the BTree classes. > Unfortunately, that's been a long-term priority for too long. > This could be a great narrow project for someone who's willing > to grok the Python C APIs. I remember you introduced me to the C API for things like this wy back in Reading at the first non US Zope 3 sprint... I was trying to create compressed list data structures for catalogs I never could quite get rid of the memory leaks I was getting! ;) Maybe I'll be brave and take another look. > Changing the default sizes for the II ad LL BTrees is pretty straightforward. > We were more interested in LO (and similar) BTrees. For those, > it's much harder to guess sizes because you don't know generally > how big the objects will be, which is why I'd like to make it tunable at the > application level. Yeah, I guess that is the issue. I wonder if it would be easy for the code to work out the total size of the bucket in bytes and then split based upon that. Or something like 120 items, or 500kB, whichever comes first. Just looking at the cache on the site at the moment, and we have a total of 978,355 objects in cache, of which: 312,523 IOBucket 274,025 IISet 116,136 OOBucket 114,626 IIBucket So 83% of my cache is just those four object types. -Matt ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.
Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc
On Wed, 2011-01-26 at 20:15 +, Matt Hamilton wrote: > So... has anyone tried increasing the size of MAX_BUCKET_SIZE in real > life? I tried back in 2007 but didn't see a big difference. https://mail.zope.org/pipermail/zodb-dev/2007-November/011279.html My benchmark was focussed on writes not on reads though so I think it is worthwhile to check out. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc
On Wed, Jan 26, 2011 at 3:15 PM, Matt Hamilton wrote: > All, > I have been doing some performance investigation into a large Plone > site we have running. The site in question has approx 300,000 items of > content. Each piece of content is indexed by ZCatalog. > > The main thing I was tracking down was the very large number of > objects being loaded by the ZODB, mostly IISet instances. > > The large numebr of instances seems to be caused by a particular usage > pattern, in various indexes in the Catalog there are a number of > IITreeSet instances that are used to map, for instance, time -> > UID. As content items are added, you end up adding monotonically > increasing values to a set. The result of this is that you end up > 'leaving behind' loads of buckets (or IISets in the case of an > IITreeSet) that are half full. > > Looking at the BTrees code, I see there is a MAX_BUCKET_SIZE constant > that is set for the various BTree/Set types, and in the case of an > IISet it is set to 120. This means, when inserting into a IITreeSet, > when the IISet gets beyond 120 items it is split and a new IISet > created. Hence as above I see a lage number of 60 item IISets due to > the pattern in which these data structures are filled. > > So, with up to 300,000 items in some of these IISets, it means to > iterate over the entire set (during a Catalog query) means loading > 5,000 objects over ZEO from the ZODB, which adds up to quite a bit of > latency. With quite a number of these data structures about, means we > can end up with in the order of 50,000 object in the ZODB cache *just* > for these IISets! Hopefully, you're not iterating over the entire tree, but still. :) > So... has anyone tried increasing the size of MAX_BUCKET_SIZE in real > life? We have, mainly to reduce the number of conflicts. > I understand that this will increase the potential for conflicts > if the bucket/set size is larger (however in reality this probably > can't get worse than it is, as currently as the value inserted is 99% > of the time greater than the current max value stored -- it is a > timestamp -- you always hit the last bucket/set in the tree). Actually, it reduces the number of unresolveable conflicts. Most conflicting bucket changes can be resolved, but bucket splits can't be and bigger buckets means fewer splits. The main tradeoff is record size. > I was going to experiment with increasing the MAX_BUCKET_SIZE on an IISet > from 120 to 1200. Doing a quick test, a pickle of an IISet of 60 items > is around 336 bytes, an of 600 items is 1580 bytes... so still very > much in the realms of a single disk read / network packet. And imagine if you use zc.zlibstorage to compress records! :) > I'm not sure how the current MAX_BUCKET_SIZE values were determined, > but looks like they have been the same since the dawn of time, and I'm > guessing might be due a tune? Probably. > It looks like I can change that constant and recompile the BTree > package, and it will work fine with existing IISets and just take > effect on new sets created (ie clear and rebuild the catalog index). > > Anyone played with this before or see any major flaws to my cunning plan? We have. My long term goal is to arrange things so that you can specify/change limits by sub-classing the BTree classes. Unfortunately, that's been a long-term priority for too long. This could be a great narrow project for someone who's willing to grok the Python C APIs. Changing the default sizes for the II ad LL BTrees is pretty straightforward. We were more interested in LO (and similar) BTrees. For those, it's much harder to guess sizes because you don't know generally how big the objects will be, which is why I'd like to make it tunable at the application level. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc
On 26.01.2011, at 21:15, Matt Hamilton wrote: > I'm not sure how the current MAX_BUCKET_SIZE values were determined, > but looks like they have been the same since the dawn of time, and I'm > guessing might be due a tune? Go Matt go! :-) -- Stefan H. Holek ste...@epy.co.at ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc
All, I have been doing some performance investigation into a large Plone site we have running. The site in question has approx 300,000 items of content. Each piece of content is indexed by ZCatalog. The main thing I was tracking down was the very large number of objects being loaded by the ZODB, mostly IISet instances. The large numebr of instances seems to be caused by a particular usage pattern, in various indexes in the Catalog there are a number of IITreeSet instances that are used to map, for instance, time -> UID. As content items are added, you end up adding monotonically increasing values to a set. The result of this is that you end up 'leaving behind' loads of buckets (or IISets in the case of an IITreeSet) that are half full. Looking at the BTrees code, I see there is a MAX_BUCKET_SIZE constant that is set for the various BTree/Set types, and in the case of an IISet it is set to 120. This means, when inserting into a IITreeSet, when the IISet gets beyond 120 items it is split and a new IISet created. Hence as above I see a lage number of 60 item IISets due to the pattern in which these data structures are filled. So, with up to 300,000 items in some of these IISets, it means to iterate over the entire set (during a Catalog query) means loading 5,000 objects over ZEO from the ZODB, which adds up to quite a bit of latency. With quite a number of these data structures about, means we can end up with in the order of 50,000 object in the ZODB cache *just* for these IISets! So... has anyone tried increasing the size of MAX_BUCKET_SIZE in real life? I understand that this will increase the potential for conflicts if the bucket/set size is larger (however in reality this probably can't get worse than it is, as currently as the value inserted is 99% of the time greater than the current max value stored -- it is a timestamp -- you always hit the last bucket/set in the tree). I was going to experiment with increasing the MAX_BUCKET_SIZE on an IISet from 120 to 1200. Doing a quick test, a pickle of an IISet of 60 items is around 336 bytes, an of 600 items is 1580 bytes... so still very much in the realms of a single disk read / network packet. I'm not sure how the current MAX_BUCKET_SIZE values were determined, but looks like they have been the same since the dawn of time, and I'm guessing might be due a tune? It looks like I can change that constant and recompile the BTree package, and it will work fine with existing IISets and just take effect on new sets created (ie clear and rebuild the catalog index). Anyone played with this before or see any major flaws to my cunning plan? -Matt ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
