-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Christian Theune wrote: > Hi, > > one of our installations hit a scalability limits with the current blob > directory structure. > > The current structure looks like: > > \blobs\ > <oid>\ > <tid>.blob > <tid>.blob > ... > <oid>\ > ... > ...\ > > We hit a limit with a database that contains more than 32k blob objects > because ext3 doesn't allow more than 32k entries in a directory. > > We propose to introduce a new mode for the blob storage which breaks the > directory structure into one level per byte of the oid. This would lead to > directories 0x00-0xFF nested in 8 levels. > > The last directory denotes the blob itself and looks like the current > directory: a list of blob files named by the tids they were committed for. > > We propose to keep both implementations around and allow to select which one > to use. We would extend the FileSystemHelper to abstract the two strategies. > > We would also provide a migration tool that can convert the old format to the > new format. > > Comments?
The squid cache directories are built like this, but they use the least significant byte first, I think to provide for better hashing (since OIDs are linearly increasing). You might also look at how the DirectoryStorage "formats" work: DS provides different strategies have different "bushiness" of the tree based on the underlying filesystem's characteristics: http://dirstorage.sourceforge.net/formats.html The strategy you propose sounds a lot like their "bushy" one, which fits your use case (ext3, needs to limit number of directory entries sharply). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIWk7J+gerLs4ltQ4RAm+OAKCHTcv1OtCLZjAHYVBvoWdLwrhSOQCfSvTY Hrw2EH4rSLYMsdbC7rgWU9k= =F+lO -----END PGP SIGNATURE----- _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
