Re: [Zope-dev] objectValues performance
Brett Carter wrote: I have a folder with greater than 5000 ZClass instances in it. It takes 5mins to do an objectValues for every object in the folder - is there a higher perfomance call I could make? -Brett Standard folder performance degrades pretty quickly once you get a lot of objects in it. There are two solutions to this: Subdivide your objects into multiple folders. Use a BTreeFolder which should be much faster. You can download the BTreeFolder product here: http://www.zope.org/Members/hathawsh/BTreeFolder/ hth, Casey Duncan ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] objectValues performance
Id recommend all the above but just for reference "objectIds" is faster than "objectValues". -- Andy McKay, Developer. ActiveState. - Original Message - From: "Casey Duncan" [EMAIL PROTECTED] To: "Brett Carter" [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Tuesday, November 28, 2000 7:58 AM Subject: Re: [Zope-dev] objectValues performance Brett Carter wrote: I have a folder with greater than 5000 ZClass instances in it. It takes 5mins to do an objectValues for every object in the folder - is there a higher perfomance call I could make? -Brett Standard folder performance degrades pretty quickly once you get a lot of objects in it. There are two solutions to this: Subdivide your objects into multiple folders. Use a BTreeFolder which should be much faster. You can download the BTreeFolder product here: http://www.zope.org/Members/hathawsh/BTreeFolder/ hth, Casey Duncan ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] objectValues performance
Ok, I'll bite. Why doesn't the standard folder scale? Seems like a design flaw to me - why doesn't the default folder use catalogs or BTrees? -Brett "Casey" == Casey Duncan [EMAIL PROTECTED] writes: Casey Brett Carter wrote: I have a folder with greater than 5000 ZClass instances in it. It takes 5mins to do an objectValues for every object in the folder - is there a higher perfomance call I could make? -Brett Casey Standard folder performance degrades pretty quickly once you get Casey a lot of objects in it. There are two solutions to this: Casey Subdivide your objects into multiple folders. Casey Use a BTreeFolder which should be much faster. Casey You can download the BTreeFolder product Casey here: http://www.zope.org/Members/hathawsh/BTreeFolder/ Casey hth, Casey Casey Duncan ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] objectValues performance
Brett Carter wrote: Ok, I'll bite. Why doesn't the standard folder scale? Seems like a design flaw to me - why doesn't the default folder use catalogs or BTrees? -Brett Because massive scale is not a requirment of folders, they are meant to organize content for humans, not to be large-collection containers. A folder with 5000 elements is not very useful to a human. On a similar note, create 5000 files in a linux directory on a ext2 (standard) filesystem and then type 'ls'. You'll notice they don't scale very well either, which is why there are filesystems like ReiserFS. -Michel "Casey" == Casey Duncan [EMAIL PROTECTED] writes: Casey Brett Carter wrote: I have a folder with greater than 5000 ZClass instances in it. It takes 5mins to do an objectValues for every object in the folder - is there a higher perfomance call I could make? -Brett Casey Standard folder performance degrades pretty quickly once you get Casey a lot of objects in it. There are two solutions to this: Casey Subdivide your objects into multiple folders. Casey Use a BTreeFolder which should be much faster. Casey You can download the BTreeFolder product Casey here: http://www.zope.org/Members/hathawsh/BTreeFolder/ Casey hth, Casey Casey Duncan ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] objectValues performance
Brett Carter wrote: Ok, I'll bite. Why doesn't the standard folder scale? Seems like a design flaw to me - why doesn't the default folder use catalogs or BTrees? -Brett AFAIK a standard folder uses a linear search when you request an object from it (ala Python dictionaries, someone please correct me if I'm wrong). This works great except that the search time grows linearly (by n) as you add objects. The BTreeFolder as the name implies creates a binary tree of the objects where the search time grows by only log n. For small folders the search time difference is minimal to non-existant, but as n increases the BTreeFolder search time increases minimally. B-trees are fairly complex entities to manage and for the vast majority of folders are total overkill. That is why standard folders work the way they do, the implementation is simple and efficient for 99.9% of applications. Your case is fairly atypical of most Zope folders. Perhaps a future implementation of Zope folders could automatically use a b-tree after a certain threshold is reached, for now you must explicitly select them. Andy's idea of using objectIds instead of objectValues is also a good one which will save significant amounts of memory. You can always access each object individually via id if you need to. Using a ZCatalog could also help in this because you can query the objects without loading them into memory and the returned result does not load the objects themselves, only the meta-data and only once a result item is explicitly accessed (By using so-called lazy sequences). However the catalog will not speed up your actual object access time unless you divide them up amongst several folders or use a BTreeFolder. The latter being a simpler solution from a design standpoint. Good luck! Casey Duncan ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] objectValues performance
"Michel" == Michel Pelletier [EMAIL PROTECTED] writes: Michel Because massive scale is not a requirment of folders, they Michel are meant to organize content for humans, not to be Michel large-collection containers. A folder with 5000 elements Michel is not very useful to a human. Michel On a similar note, create 5000 files in a linux directory on a ext2 Michel (standard) filesystem and then type 'ls'. You'll notice they don't Michel scale very well either, which is why there are filesystems like Michel ReiserFS. Michel -Michel Point taken, but in Zope where data and logic reside in the same place (ZODB) shouldn't we have some sort of effcient storage for large amounts of data? What happens when a site gets three or four thousand users? That won't fit well in an UserFolder. -Brett [...] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] objectValues performance
Brett Carter wrote: I have a folder with greater than 5000 ZClass instances in it. It takes 5mins to do an objectValues for every object in the folder - is there a higher perfomance call I could make? -Brett use a catalog. (which only help if you want a filtered set or a schema attrs of the object) beefing up your memory (assuming you want zope to use alot), could also help, assuming these objects are accessed frequently enough to stay in the object cache. kapil ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
