Re: [Zope-dev] any limits on object number?
On Wed, 2004-07-14 at 14:20, Andreas Jung wrote: --On Mittwoch, 14. Juli 2004 13:46 Uhr +0530 Nagarjuna G. [EMAIL PROTECTED] wrote: We are builiding a large portal using Zope. We need to create a large number of objects. How large? in one case, we want the metadata of all the wikipedia contents, that will be around 300,000 objects. The data component of the objects is small, but each object carries lots of metadata. My question is: Is there any limit on the number of objects in a given folder? No, but you should use BTreeFolder2 in any case and don't use the standard Zope folder! thanks, this is certainly going to help, I will try inheriting this class. I am not planning to use any external RDBMS. Are there any known performace isssues when the numbeer of objects increase, particularly when we store them in the same folder? Maybe tell us some details. The project is GNOWSYS, an application that builds semantic web portals on top of Zope. When we tried to upload objects from wikipedia's sqldump of after about 40,000 objects in a folder the script went for a toss. roughly each time we ran the script the same problem. even to visit the folder of even 1000 objects through ZMI is a pain. I think that is where BTreeFolder2 will be very useful. I am also thinking to use a squid cache style of storing objects in zodb, where the objects go into many subfolders (i dont know what is this method called technically) with several sub folders with names alphabetized/numbered. this way i can store large number within zodb though not in the same folder/BTreefolder2. Should I use both methods? Not related to Zope, but what about filesystems? there must be a limit on the number of files within one folder depending on which filesystem one uses. Thanks Nagarjuna ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] any limits on object number?
When we tried to upload objects from wikipedia's sqldump of after about 40,000 objects in a folder the script went for a toss. roughly each time we ran the script the same problem. even to visit the folder of even 1000 objects through ZMI is a pain. I think that is where BTreeFolder2 will be very useful. Your observations make sense in the context of the standard Zope folder. BTreeFolder2 will help. I am assuming whatever script you use to populate the folder(s) commits the transaction ever N objects, right? I am also thinking to use a squid cache style of storing objects in zodb, where the objects go into many subfolders (i dont know what is this method called technically) with several sub folders with names alphabetized/numbered. this way i can store large number within zodb though not in the same folder/BTreefolder2. Should I use both methods? I would definitely use the subfolder method. Not related to Zope, but what about filesystems? there must be a limit on the number of files within one folder depending on which filesystem one uses. The ZODB, in case you use the standard FileStorage, is one monolithic file. So there is no large number of files on the server file system regardless of how many objects you store. For files on the file systems your choice of file system depends not just on the number of objects but also its size. To re-use your earlier example of Squid cache files, which are numerous and mostly small, ReiserFS is normally recommended. It deals much better with large quantities of files in one folder. jens ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] any limits on object number?
[resend] We are builiding a large portal using Zope. We need to create a large number of objects. How large? in one case, we want the metadata of all the wikipedia contents, that will be around 300,000 objects. The data component of the objects is small, but each object carries lots of metadata. My question is: Is there any limit on the number of objects in a given folder? No, but you should use BTreeFolder2 in any case and don't use the standard Zope folder! 'Folders' provide an easy gui for managing sub-items one at a time, and a single interface for accessing sub-items that enables things like the tree view (in the left pane of the management interface) to work with different folder implementations. Im not sure any of that is a benefit to you, and Folders arent the only kind of ZODB container. I suspect you would be better off directly using a BTree of some kind. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
