[Zope-dev] Re: any limits on object number?
Hello all, thanks to all the responses. I will test with BTreeFolder2, and subfoldering with first character of id as another subfolder2, so that working with ZMI will also be possible without further scripting or customized views. i will report the results back to the list. just an additional info: I run into this problem recently... I've got something around 40k objects to insert into ZODB. I started doing some tests with normal Folders, but I discarded it completely in favor of BTreeFolders. I got information that BTreeFolder was already tested with more than 400k objects... and that made me happy! :-) But (there's always a 'but'!)... Even using BTreeFolder to store all objects I was getting 300s delay to show a single object (ok, it's an archetypes-based one, containing 50 fields, splitted into 7 schematas, with lots of fancy stuff...). So I made a directory hash structure based on UID from each object (an AT UID is md5, so we have a hex base). Using this hash structure, with 16 divisions, I got a better performance (but still far from acceptable): 'only' ~100s to show a single object... :-( So I decided to create another level for the hash structure: now each folder has another 16 folder inside it. This time I was getting a delay of ~20s... As you're already thinking... it's time for another hash level. Now, with 16^3 additional BTreeFolders to split all my objects I got an acceptable performance: ~3s... The path is uggly as hell, f/f0/f0a/f0a5aac38aeff101b3168f2592dd879b, but at least the system is usable... Resuming what I've learned: don't abuse of BTreeFolder, hash your content and live happy forever... ;-) PS: The server is a modest PIII 1.2GHz, 1GB RAM with a 160MB/s SCSI controller running only one instance of Zope 2.7.1 without ZEO. HTH, -- Dorneles Treméa Caxias do Sul - RS - Brasil +55 54 9114 9312 - UIN: 2413568 X3ng Web Technology http://www.x3ng.com.br -BEGIN GEEK CODE BLOCK- Version: 3.12 GCS/IT d- s:-: a25 C+++ UBL$ P--- L++ E-- W+++ N++ o? K? w+ O M+ V-- PS+ PE- Y-- PGP++ t+ 5 X++ R+ tv+ b(++) DI+ D++ G e++ h r+++ y+++** --END GEEK CODE BLOCK-- ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: any limits on object number?
On Mon, 2004-07-19 at 11:38, Dorneles Trema wrote: Resuming what I've learned: don't abuse of BTreeFolder, hash your content and live happy forever... ;-) PS: The server is a modest PIII 1.2GHz, 1GB RAM with a 160MB/s SCSI controller running only one instance of Zope 2.7.1 without ZEO. Thanks a lot. Your report that performance enhances due to hashing is encouraging the decision we took. In our case we already began using BTreeFolder2, havent tried creating the large object base yet (will do this week), but we doing something like what you did. we are creating alphabetized subfolders, by taking the first char of each id. Are there any good popular hashing algorithms? If we have such a algo then further subfoldering could be made automatic after a certain number of objects increase in each folder. This way large scalable databases can be created in Zope. Nagarjuna ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: any limits on object number?
Even using BTreeFolder to store all objects I was getting 300s delay to show a single object (ok, it's an archetypes-based one, containing 50 fields, splitted into 7 schematas, with lots of fancy stuff...). So I made a directory hash structure based on UID from each object (an AT UID is md5, so we have a hex base). My own experience with BTreeFolder2 does not support your diagnosis. I do not believe BTreeFolder2 is the problem here. On a CMF-based CMS that I helped develop the largest BTreeFolder2-based containers held ca. 60,000 items at the top level, meaning we did not use a directory structure to restrict the number of items per folder. There was zero delay retrieving singe items and even stepping into the ZMI where it shows 1000 ids at a time was sub-2 second response time. jens ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: any limits on object number?
thanks to all the responses. I will test with BTreeFolder2, and subfoldering with first character of id as another subfolder2, so that working with ZMI will also be possible without further scripting or customized views. i will report the results back to the list. just an additional info: I run into this problem recently... I've got something around 40k objects to insert into ZODB. I started doing some tests with normal Folders, but I discarded it completely in favor of BTreeFolders. I got information that BTreeFolder was already tested with more than 400k objects... and that made me happy! :-) But (there's always a 'but'!)... Even using BTreeFolder to store all objects I was getting 300s delay to show a single object (ok, it's an archetypes-based one, containing 50 fields, splitted into 7 schematas, with lots of fancy stuff...). So I made a directory hash structure based on UID from each object (an AT UID is md5, so we have a hex base). Using this hash structure, with 16 divisions, I got a better performance (but still far from acceptable): 'only' ~100s to show a single object... :-( So I decided to create another level for the hash structure: now each folder has another 16 folder inside it. This time I was getting a delay of ~20s... As you're already thinking... it's time for another hash level. Now, with 16^3 additional BTreeFolders to split all my objects I got an acceptable performance: ~3s... The path is uggly as hell, f/f0/f0a/f0a5aac38aeff101b3168f2592dd879b, but at least the system is usable... Resuming what I've learned: don't abuse of BTreeFolder, hash your content and live happy forever... ;-) I very much doubt you solved the problem you think you solved. Access time to a single object in the hundred of seconds is not a BTreeFolder problem. BTreeFolder is designed to not be a bottleneck for concurrent access and large number of objects. You should have benched (using ZopeProfiler for instance) to find out where time really is spent. Maybe some of your/AT's code does a stupid loop on folder.objectIds() or something. Florent -- Florent Guillaume, Nuxeo (Paris, France) +33 1 40 33 79 87 http://nuxeo.com mailto:[EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: any limits on object number?
On Wed, 2004-07-14 at 18:55, Casey Duncan wrote: On Wed, 14 Jul 2004 13:46:57 +0530 Nagarjuna G. [EMAIL PROTECTED] wrote: We are builiding a large portal using Zope. We need to create a large number of objects. The data component of the objects is small, but each object carries lots of metadata. My question is: Is there any limit on the number of objects in a given folder? I am not planning to use any external RDBMS. Are there any known performace isssues when the numbeer of objects increase, particularly when we store them in the same folder? Normal zope folders should probably not be used to hold more than a few dozen items. They store a list of their children in a single ZODB record, and as their numbers increase so do the size of the transactions that change the folder. Also normal Zope folders do not handle concurrent updates and will thus not perform well when multiple users are adding items to the folder. Shane Hathaway's BTreeFolder2 product solves these problems. It is the thing to use when you want to store large numbers of objects in a single folder. It also handles concurrency much better. hth, thanks to all the responses. I will test with BTreeFolder2, and subfoldering with first character of id as another subfolder2, so that working with ZMI will also be possible without further scripting or customized views. i will report the results back to the list. Nagarjuna ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: any limits on object number?
On Wed, 14 Jul 2004 13:46:57 +0530 Nagarjuna G. [EMAIL PROTECTED] wrote: We are builiding a large portal using Zope. We need to create a large number of objects. The data component of the objects is small, but each object carries lots of metadata. My question is: Is there any limit on the number of objects in a given folder? I am not planning to use any external RDBMS. Are there any known performace isssues when the numbeer of objects increase, particularly when we store them in the same folder? Normal zope folders should probably not be used to hold more than a few dozen items. They store a list of their children in a single ZODB record, and as their numbers increase so do the size of the transactions that change the folder. Also normal Zope folders do not handle concurrent updates and will thus not perform well when multiple users are adding items to the folder. Shane Hathaway's BTreeFolder2 product solves these problems. It is the thing to use when you want to store large numbers of objects in a single folder. It also handles concurrency much better. hth, -Casey ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
