I'm considering setting up my server as a mirror site for the
lookup database.  Unfortunately, I've seem to have run into a stumbling
block.  The server app requires over 250,000 files in a single directory.
Each file is about 2k in size.

It was a surprise to me (but probably not to those on this list) that the
file system does not handle that many files in an expeditious manner (I'm
being kind here).

Is there anything I can do so that the file system works faster with such a
large number of files?  I'm looking for an increase in the area of 5 to 1.
 For example, the command "rm -rf misc" where "misc" is the directory
containing the 250,000 files takes a couple of hours to run.  If "misc" is
my current working directory, and I type "rm *" I get the message that
there are too many arguments being passed into rm.
Try the dirhash option in your kernel (as suggested elsewhere).

As for commands like rm, you have a few options (as I was recently taught).
find(1) appears to handle large numbers of files, thus you can use it as
a pipe for just about anything, i.e.:
find /path/to/misc -name '*' -print0 | xargs -0 rm
(although you don't really need the pipe for rm, but it's an example)

I don't know how much of a problem this is for you, but it will work for
mv, cp, grep ... just about anything. The quotes around * are important.

Also, there is a sysctl (kern.argmax) that you can tweak to increase the
length of command line arguments that can be processed, but I don't know
how big you'd have to make it to handle 250,000 files!

Good luck,

MSN 8 with e-mail virus protection service: 2 months FREE*

To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-questions" in the body of the message

Reply via email to