On Dec 3, 2008, at 1:50 AM, Christian Theune wrote:
> On Tue, 2008-12-02 at 12:03 -0500, Jim Fulton wrote:
>> ZEO has two modes for dealing with client blob data, shared, and non-
>> shared. In shared mode, a distributed file system is used to share a
>> blob directory with a ZEO server. This requires management of a
>> distributed file system, in addition to the ZEO protocol. Any
>> is provided by the distributed file system.
>> In non-shared mode, blob data are downloaded to the ZEO client using
>> the ZEO protocol. No distributed file-system is needed and blob
>> are cached locally. Unfortunately, the current implementation
>> no facilities for managing the client cache. There are no provisions
>> in the ZEO client software for removing unused blob files and the
>> implementation makes almost no provision for blob file removal.
>> I'm working on refactoring ClientStorage's handling of non-shared
>> data. I'm implementing a mechanism for periodically cleaning out
>> files that haven't been accessed in a while. As part of this, I'm
>> going to radically change the layout of the ClientStorage's non-
>> blob directory.
>> Currently, the bushy layout, with deeply nested directories is used.
>> While I think this layout makes some sense on the server, I don't
>> think it makes much sense on the client. Cleaning up unused blob
>> files is complicated by the need to clean up directories too. I'm
>> going to go for a fairly flat layout. There will be a small number
>> (997) of directories and blob files will reside directly in these
>> directories. (The directory will be chosen by taking the remainder
>> dividing an oid by 997.)
> Any specific reason for this specific number?
It is prime, and ~1000 directories seems pretty manageable. More
importantly, I'm using a file lock per directory and I only allow one
process/thread at a time to operate on a file in the directory. I
want a somewhat large number to try to avoid contention.
>> It appears that modern operating systems can
>> handle large directories just fine. I've created directories with 1
>> million files on Linux/Ext, Mac OS X/HFS+, and Windows XP/NTFS and
>> no degredation in performance as the number of files in a directory
> FTR: The reason for introducing the bushy layout is due to
> on the number of directory entries a directory can contain which
> seem to
> be a different restriction than the number of file entries a directory
> can contain. At least on ext3 I can't create more than 65k directories
> in a directory while I still can create a lot more files in the same
> directory. Wikipedia has a generally good overview and comparison
> between file systems but doesn't cover the maximum number of directory
> entries per directory.
The ext limitation on the number of subdirectories arises from a limit
on the number of links to an inode. Each subdirectory has a ".."
entry which ads a link to the containing directory. I don't know if
there is a limit on the number of directory entries. If there is, it
is quite large. (I did a test of adding 10 million files to a ext
directory, although I got tired of waiting for it after it had gotten
up to a bit over 6 million.)
>> I plan to have ClientStorage use the file layout mentioned above.
>> ClientStorage constructor will fail if an older layout is found. An
>> alternative is to just log a warning and ignore the existing
>> directories, as the new directories will have non-overlapping names.
>> I mention this both as a heads up and to see if anyone can point
>> out a
>> problem with my approach. I have a feeling that no one is using non-
>> shared client blob directories for anything important yet, so I
>> the change won't have much effect.
> I am. I'd prefer if you'd fail on the directory structure instead of
> mixing it with the new approach.
For more information about ZODB, see the ZODB Wiki:
ZODB-Dev mailing list - ZODB-Dev@zope.org