Re: [ZODB-Dev] Optimize BTree node sizes?
Hanno Schlichting wrote: > Jim Fulton wrote: >> On Nov 4, 2008, at 12:12 PM, Benji York wrote: >> >>> On Tue, Nov 4, 2008 at 12:01 PM, Jim Fulton <[EMAIL PROTECTED]> wrote: A few months back, there was a lot of discussion here about BTree performance. I got a sense that maximum BTree-node and bucket sizes should be increased. Does anyone have recommendations for new sizes? >>> It'd be cool if the bucket size could be dynamic (say governed by an >>> attribute on the BTree), but I suspect that is dramatically out of >>> scope >>> for what you were planning on doing. >> I have a list of projects I might try to do for 3.9. One would be to >> make BTrees subclassable >> and to modify BTrees to get these limits from class attributes. But >> whether I get to this or not, >> if there are more sensible defaults, it would make sense to use them. > > Personally I think the numbers for the integer BTree's could be > increased indeed, but for the object BTree's, a way to adjust the bucket > sizes would be more important. > > The problem for BTree's storing objects, is of course that you have no > idea how large those objects are going to be. Sometimes they might be > simple types with just a couple of bytes each, sometimes catalog brains > with one kilobyte, but sometimes they might be actual content-like > object of various megabyte in size. > > In general there is no way of knowing what you are going to use the > BTree's for. Making it possible for the application code to adjust the > values for the particular kind of data, is probably better than guessing > a better default value. +1 The classic case here is comparing a catalogue metadata BTree with one containing persistent objects, where only the oid needs storing in the pickle. Laurence ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Optimize BTree node sizes?
Jim Fulton wrote: > On Nov 4, 2008, at 12:12 PM, Benji York wrote: > >> On Tue, Nov 4, 2008 at 12:01 PM, Jim Fulton <[EMAIL PROTECTED]> wrote: >>> A few months back, there was a lot of discussion here about BTree >>> performance. I got a sense that maximum BTree-node and bucket sizes >>> should be increased. Does anyone have recommendations for new sizes? >> It'd be cool if the bucket size could be dynamic (say governed by an >> attribute on the BTree), but I suspect that is dramatically out of >> scope >> for what you were planning on doing. > > I have a list of projects I might try to do for 3.9. One would be to > make BTrees subclassable > and to modify BTrees to get these limits from class attributes. But > whether I get to this or not, > if there are more sensible defaults, it would make sense to use them. Personally I think the numbers for the integer BTree's could be increased indeed, but for the object BTree's, a way to adjust the bucket sizes would be more important. The problem for BTree's storing objects, is of course that you have no idea how large those objects are going to be. Sometimes they might be simple types with just a couple of bytes each, sometimes catalog brains with one kilobyte, but sometimes they might be actual content-like object of various megabyte in size. In general there is no way of knowing what you are going to use the BTree's for. Making it possible for the application code to adjust the values for the particular kind of data, is probably better than guessing a better default value. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Optimize BTree node sizes?
On Tue, Nov 4, 2008 at 12:01 PM, Jim Fulton <[EMAIL PROTECTED]> wrote: > > A few months back, there was a lot of discussion here about BTree > performance. I got a sense that maximum BTree-node and bucket sizes > should be increased. Does anyone have recommendations for new sizes? It'd be cool if the bucket size could be dynamic (say governed by an attribute on the BTree), but I suspect that is dramatically out of scope for what you were planning on doing. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Optimize BTree node sizes?
On Nov 4, 2008, at 12:12 PM, Benji York wrote: > On Tue, Nov 4, 2008 at 12:01 PM, Jim Fulton <[EMAIL PROTECTED]> wrote: >> >> A few months back, there was a lot of discussion here about BTree >> performance. I got a sense that maximum BTree-node and bucket sizes >> should be increased. Does anyone have recommendations for new sizes? > > It'd be cool if the bucket size could be dynamic (say governed by an > attribute on the BTree), but I suspect that is dramatically out of > scope > for what you were planning on doing. I have a list of projects I might try to do for 3.9. One would be to make BTrees subclassable and to modify BTrees to get these limits from class attributes. But whether I get to this or not, if there are more sensible defaults, it would make sense to use them. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
