Re: [Zope-dev] objectValues performance
Casey Duncan wrote: > > Brett Carter wrote: > > > Ok, I'll bite. Why doesn't the standard folder scale? Seems like a > > design flaw to me - why doesn't the default folder use catalogs or BTrees? > > -Brett i believe folders store objects as attrs which implies using a hash in __dict__ for lookup. some quick conclusions. the btree folder is slower on average for creation by anywhere from 15-25% andt btree's __getattr is on average almost twice as slow as the folder's getattr. btrees folders are great, but they currently have a performance penalty at the benefit of larger folders that behave better during concurrent changes, mainly because they implement a __getattr__ hook in python instead of c. i did some tests a few months ago(posted to zope-dev) and i believe the performance penalty varied depending on the operation, approx 15-25% on ob. creation and 40-50% slower on access. unless you know you'll have a large number of objects or lots of concurrent changes you'd be better off with a standard folder. as for what constitues a large number, i'm not sure, it depends on object size and number of objects, during my tests i didn't see any changes with scaling the numbers up to 5000, but granted my objects were lightweight and poor examples on anything resembling realworld. regarding objectids in this context its the difference between dict.keys() and dict.items(). my catalog suggestion was under the assumption that in getting a filtered set from the catalog would allow you to do dict[id] on individual items for the set, using the metadata to perform any runtime application filtering computations. cheers kapil > AFAIK a standard folder uses a linear search when you request an object from it > (ala Python dictionaries, someone please correct me if I'm wrong). This works > great except that the search time grows linearly (by n) as you add objects. The > BTreeFolder as the name implies creates a binary tree of the objects where the > search time grows by only log n. For small folders the search time difference > is minimal to non-existant, but as n increases the BTreeFolder search time > increases minimally. B-trees are fairly complex entities to manage and for the > vast majority of folders are total overkill. That is why standard folders work > the way they do, the implementation is simple and efficient for 99.9% of > applications. Your case is fairly atypical of most Zope folders. > > Perhaps a future implementation of Zope folders could automatically use a > b-tree after a certain threshold is reached, for now you must explicitly select > them. > > Andy's idea of using objectIds instead of objectValues is also a good one which > will save significant amounts of memory. You can always access each object > individually via id if you need to. Using a ZCatalog could also help in this > because you can query the objects without loading them into memory and the > returned result does not load the objects themselves, only the meta-data and > only once a result item is explicitly accessed (By using so-called lazy > sequences). However the catalog will not speed up your actual object access > time unless you divide them up amongst several folders or use a BTreeFolder. > The latter being a simpler solution from a design standpoint. > > Good luck! > > Casey Duncan > > ___ > Zope-Dev maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] objectValues performance
> "Michel" == Michel Pelletier <[EMAIL PROTECTED]> writes: Michel> Because massive scale is not a requirment of folders, they Michel> are meant to organize content for humans, not to be Michel> large-collection containers. A folder with 5000 elements Michel> is not very useful to a human. Michel> On a similar note, create 5000 files in a linux directory on a ext2 Michel> (standard) filesystem and then type 'ls'. You'll notice they don't Michel> scale very well either, which is why there are filesystems like Michel> ReiserFS. Michel> -Michel Point taken, but in Zope where data and logic reside in the same place (ZODB) shouldn't we have some sort of effcient storage for large amounts of data? What happens when a site gets three or four thousand users? That won't fit well in an UserFolder. -Brett [...] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] objectValues performance
Brett Carter wrote: > Ok, I'll bite. Why doesn't the standard folder scale? Seems like a > design flaw to me - why doesn't the default folder use catalogs or BTrees? > -Brett AFAIK a standard folder uses a linear search when you request an object from it (ala Python dictionaries, someone please correct me if I'm wrong). This works great except that the search time grows linearly (by n) as you add objects. The BTreeFolder as the name implies creates a binary tree of the objects where the search time grows by only log n. For small folders the search time difference is minimal to non-existant, but as n increases the BTreeFolder search time increases minimally. B-trees are fairly complex entities to manage and for the vast majority of folders are total overkill. That is why standard folders work the way they do, the implementation is simple and efficient for 99.9% of applications. Your case is fairly atypical of most Zope folders. Perhaps a future implementation of Zope folders could automatically use a b-tree after a certain threshold is reached, for now you must explicitly select them. Andy's idea of using objectIds instead of objectValues is also a good one which will save significant amounts of memory. You can always access each object individually via id if you need to. Using a ZCatalog could also help in this because you can query the objects without loading them into memory and the returned result does not load the objects themselves, only the meta-data and only once a result item is explicitly accessed (By using so-called lazy sequences). However the catalog will not speed up your actual object access time unless you divide them up amongst several folders or use a BTreeFolder. The latter being a simpler solution from a design standpoint. Good luck! Casey Duncan ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] objectValues performance
Brett Carter wrote: > > Ok, I'll bite. Why doesn't the standard folder scale? Seems like a > design flaw to me - why doesn't the default folder use catalogs or BTrees? > -Brett Because massive scale is not a requirment of folders, they are meant to organize content for humans, not to be large-collection containers. A folder with 5000 elements is not very useful to a human. On a similar note, create 5000 files in a linux directory on a ext2 (standard) filesystem and then type 'ls'. You'll notice they don't scale very well either, which is why there are filesystems like ReiserFS. -Michel > > "Casey" == Casey Duncan <[EMAIL PROTECTED]> writes: > > Casey> Brett Carter wrote: > >>> I have a folder with greater than 5000 ZClass instances in it. It > >>> takes > 5mins to do an objectValues for every object in the folder - > >>> is there a higher perfomance call I could make? > >>> -Brett > > Casey> Standard folder performance degrades pretty quickly once you get > Casey> a lot of objects in it. There are two solutions to this: > > Casey> Subdivide your objects into multiple folders. > Casey> Use a BTreeFolder which should be much faster. > > Casey> You can download the BTreeFolder product > Casey> here: http://www.zope.org/Members/hathawsh/BTreeFolder/ > > Casey> hth, > Casey> Casey Duncan > > ___ > Zope-Dev maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] objectValues performance
Ok, I'll bite. Why doesn't the standard folder scale? Seems like a design flaw to me - why doesn't the default folder use catalogs or BTrees? -Brett > "Casey" == Casey Duncan <[EMAIL PROTECTED]> writes: Casey> Brett Carter wrote: >>> I have a folder with greater than 5000 ZClass instances in it. It >>> takes > 5mins to do an objectValues for every object in the folder - >>> is there a higher perfomance call I could make? >>> -Brett Casey> Standard folder performance degrades pretty quickly once you get Casey> a lot of objects in it. There are two solutions to this: Casey> Subdivide your objects into multiple folders. Casey> Use a BTreeFolder which should be much faster. Casey> You can download the BTreeFolder product Casey> here: http://www.zope.org/Members/hathawsh/BTreeFolder/ Casey> hth, Casey> Casey Duncan ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] objectValues performance
Id recommend all the above but just for reference "objectIds" is faster than "objectValues". -- Andy McKay, Developer. ActiveState. - Original Message - From: "Casey Duncan" <[EMAIL PROTECTED]> To: "Brett Carter" <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> Sent: Tuesday, November 28, 2000 7:58 AM Subject: Re: [Zope-dev] objectValues performance > Brett Carter wrote: > > > I have a folder with greater than 5000 ZClass instances in it. It > > takes > 5mins to do an objectValues for every object in the folder - > > is there a higher perfomance call I could make? > > -Brett > > Standard folder performance degrades pretty quickly once you get > a lot of objects in it. There are two solutions to this: > > Subdivide your objects into multiple folders. > Use a BTreeFolder which should be much faster. > > You can download the BTreeFolder product > here: http://www.zope.org/Members/hathawsh/BTreeFolder/ > > hth, > Casey Duncan > > > > > > > ___ > Zope-Dev maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] objectValues performance
Brett Carter wrote: > I have a folder with greater than 5000 ZClass instances in it. It > takes > 5mins to do an objectValues for every object in the folder - > is there a higher perfomance call I could make? > -Brett Standard folder performance degrades pretty quickly once you get a lot of objects in it. There are two solutions to this: Subdivide your objects into multiple folders. Use a BTreeFolder which should be much faster. You can download the BTreeFolder product here: http://www.zope.org/Members/hathawsh/BTreeFolder/ hth, Casey Duncan ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] objectValues performance
Brett Carter wrote: > > I have a folder with greater than 5000 ZClass instances in it. It > takes > 5mins to do an objectValues for every object in the folder - > is there a higher perfomance call I could make? > -Brett use a catalog. (which only help if you want a filtered set or a schema attrs of the object) beefing up your memory (assuming you want zope to use alot), could also help, assuming these objects are accessed frequently enough to stay in the object cache. kapil ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] objectValues performance
I have a folder with greater than 5000 ZClass instances in it. It takes > 5mins to do an objectValues for every object in the folder - is there a higher perfomance call I could make? -Brett ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
