When we tried to upload objects from wikipedia's sqldump of after about
40,000 objects in a folder the script went for a toss. roughly each time
we ran the script the same problem. even to visit the folder of even
1000 objects through ZMI is a pain. I think that is where BTreeFolder2
will be very useful.
Your observations make sense in the context of the standard Zope folder. BTreeFolder2 will help. I am assuming whatever script you use to populate the folder(s) commits the transaction ever N objects, right?
I am also thinking to use a squid cache style of storing objects in zodb, where the objects go into many subfolders (i dont know what is this method called technically) with several sub folders with names alphabetized/numbered. this way i can store large number within zodb though not in the same folder/BTreefolder2.
Should I use both methods?
I would definitely use the subfolder method.
Not related to Zope, but what about filesystems? there must be a limit on the number of files within one folder depending on which filesystem one uses.
The ZODB, in case you use the standard FileStorage, is one monolithic file. So there is no large number of files on the server file system regardless of how many objects you store.
For files on the file systems your choice of file system depends not just on the number of objects but also its size. To re-use your earlier example of Squid cache files, which are numerous and mostly small, ReiserFS is normally recommended. It deals much better with large quantities of files in one folder.
Zope-Dev maillist - [EMAIL PROTECTED]
** No cross posts or HTML encoding! **
(Related lists - http://mail.zope.org/mailman/listinfo/zope-announce