On 22/04/07, Marc A. Volovic <[EMAIL PROTECTED]> wrote:
Quoth Amos Shapira: > Hi, > > Our servers have to deal with huge amounts of small files (tens, > sometimes hundreds of thousands of files IN ONE DIRECTORY). > > Currently they use ext3 but I wonder wether this is the prefered FS. Ext3 is - last I chaecked (about two years ago) possibly the worst filesystem for dealing with LOTS of files in a single directory. Reiser 3 was very good (did not try reiser 4). However, I am very wary of reiser now - what with poor (or, maybe, not so poor) Hans being in jail, reiserfs may be going the way of the dodo.
If Reiser3 is already in mainline and stable - wouldn't it be supported even if Hans/Nemesis vanishes? Reiser4 is not relevant because I want to stick to mainline kernels, much preferably Debian supplied kernels. I'd run bonnie (just the creation/deletion tests) for JFS, XFS and Ext4
(which is starting to make an appearance here and there). IIRC - XFS is ALSO not very good with lots of small files.
Will try to do that, though again - if ext4 isn't in the mainline yet then it's not relevant for me.
I'm also thinking about better ways to handle the files (e.g. putting every > few thousands of them in a .zip file to transfer, spreading them across a > two-level directory tree etc) but I'd rathertry to keep the changes to the > existing software and scripts the the minimum which is required to speed > things up. B-sort em? Switch the back-end to database (assuming the blobs are small)?
I'm thinking of databases sometimes (the files are around 4k on average) but it feels like Hans Reiser was sort of right about that - a filesystem can be used as a database for this sort of data. Cheers, --Amos
