Re: Lots of files in a directory

2003-01-04 Thread MikeM
On 1/3/2003 at 9:25 AM MikeM wrote:

| [snip]
|
|To you and others who have helped me, here's what I'm doing.
|
|- tomorrow I am upgrading from 4.5 to 4.7
|- once that proves stable, I'll increase the DIRHASH value appropriately.
|- once that proves stable, I'll tunefs -n enable /usr to turn on soft
|dependencies
|
|I'll report back with my experiences with the new configuration so that
the
|archive remains complete.
 =


The upgrade to 4.7 went very smoothly.  Many thanks to the documentation
team for the excellent documentation on upgrading
(http://www.freebsd.org/releases/4.7R/relnotes-i386.html#AEN577).  This was
my first major upgrade on a remote server (I have been keeping up-to-date
with the security patches).  Needless to say, I was a bit apprehensive when
I typed that reboot command.  I never thought I'd be so very happy to see
the words 

Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
   The Regents of the University of California.  All rights reserved.
 FreeBSD 4.7-RELEASE-p2 ...

appear on my ssh screen.  :-)


Next I did the tunefs step.   I had to ask the techs at the datacenter to
boot into single user mode and perform the tunefs on the local console.
That also went smoothly.

When I tried to  tar -yxf the freedb archive, the server became
unresponsive.  I am not sure exactly what the root cause of the problem
was, as my ssh session went dead and I could not re-establish a connection.
 Long story short, I found out the tech support in the data center
(www.tera-byte.com) is excellent.   The tech re-established the local
console and ran the fsck required to bring the server back to life.  He did
mention the following in one of his emails, [the server] appears to be
having some problems with the fsck due to some dirty inodes... .

The server is back online now, and running well.  Trying to duplicate the
problem, I ran a similar sequence sequence of events on a server here at my
house.   It didn't crash, but I did get screenfuls of the following error
message:

   /usr: create/symlink failed, no inodes free
   tar: misc/ed11d70f: Cannot open: No space left on device

That message was repeated for each of the hundreds of thousands of files.
I suspect 'something' on my remote server was not happy about dealing with
such a flood of error messages.

At this point, I'm not willing to newfs the file system to get the number
of inodes that I need.  So I think, so far as I am concerned, this
experiment is over.  :-)

Once again, thanks to everyone who gave me help on this.













To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-questions in the body of the message



Re: Lots of files in a directory

2003-01-04 Thread Giorgos Keramidas
On 2003-01-04 09:04, MikeM [EMAIL PROTECTED] wrote:
 The upgrade to 4.7 went very smoothly.
 [...]
 When I tried to  tar -yxf the freedb archive, the server became
 unresponsive.
 [...]
 The server is back online now, and running well.  Trying to duplicate the
 problem, I ran a similar sequence sequence of events on a server here at my
 house.   It didn't crash, but I did get screenfuls of the following error
 message:

/usr: create/symlink failed, no inodes free
tar: misc/ed11d70f: Cannot open: No space left on device

What does `df -i' report for your /usr filesystem?  Every file needs
an i-node of itself on a ufs filesystem, and having many thousands of
files takes many thousands of i-nodes...  At home, I can see something
like the following:

giorgos@gothmog[20:36]/home/giorgos$ df -i /usr
Filesystem  1K-blocksUsedAvail Capacity iused   ifree %iused  Mounted on
/dev/ad0s3g   4636750 1793606  247220442%  205145  377253   35%   /usr

The iused, ifree and %iused columns are those you're interested in.

- Giorgos

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-questions in the body of the message



Re: Lots of files in a directory

2003-01-03 Thread Giorgos Keramidas
On 2003-01-02 09:01, MikeM [EMAIL PROTECTED] wrote:
 I'm considering setting up my server as a mirror site for the freedb.org
 lookup database.  Unfortunately, I've seem to have run into a stumbling
 block.  The server app requires over 250,000 files in a single directory.
 Each file is about 2k in size.

On 2003-01-02 12:13, MikeM [EMAIL PROTECTED] wrote:
 On 1/2/03 at 8:58 AM Mikko Ty?l?j?rvi wrote:
  It can be argued that the application is poorly designed...

 There's no arguing.  It *is* poorly designed.   I would have used a
 hash to spread the files across multiple directories.
 Unfortunately, I didn't design it.

You are probably hitting the limit of fragments that are allowed on a
filesystem and time optimization becomes space optimization.  Are
you getting entries in your system logs like the following?

optimization changed from TIME to SPACE

- Giorgos

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-questions in the body of the message



Re: Lots of files in a directory

2003-01-03 Thread MikeM
On 1/3/03 at 6:45 AM Giorgos Keramidas wrote:

|On 2003-01-02 09:01, MikeM [EMAIL PROTECTED] wrote:
| I'm considering setting up my server as a mirror site for the freedb.org
| lookup database.  Unfortunately, I've seem to have run into a stumbling
| block.  The server app requires over 250,000 files in a single
directory.
| Each file is about 2k in size.
|
|On 2003-01-02 12:13, MikeM [EMAIL PROTECTED] wrote:
| On 1/2/03 at 8:58 AM Mikko Ty?l?j?rvi wrote:
|  It can be argued that the application is poorly designed...
|
| There's no arguing.  It *is* poorly designed.   I would have used a
| hash to spread the files across multiple directories.
| Unfortunately, I didn't design it.
|
|You are probably hitting the limit of fragments that are allowed on a
|filesystem and time optimization becomes space optimization.  Are
|you getting entries in your system logs like the following?
|
|   optimization changed from TIME to SPACE
 =

I just grep'd /var/log/* and the word optimization did not appear.


To you and others who have helped me, here's what I'm doing.

- tomorrow I am upgrading from 4.5 to 4.7
- once that proves stable, I'll increase the DIRHASH value appropriately.
- once that proves stable, I'll tunefs -n enable /usr to turn on soft
dependencies

I'll report back with my experiences with the new configuration so that the
archive remains complete.


Thanks to everyone who helped.


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-questions in the body of the message



Re: Lots of files in a directory

2003-01-02 Thread Roman Neuhauser
# [EMAIL PROTECTED] / 2003-01-02 09:01:27 -0500:
 I'm considering setting up my server as a mirror site for the freedb.org
 lookup database.  Unfortunately, I've seem to have run into a stumbling
 block.  The server app requires over 250,000 files in a single directory.
 Each file is about 2k in size.
 
 It was a surprise to me (but probably not to those on this list) that the
 file system does not handle that many files in an expeditious manner (I'm
 being kind here). 
 
 Is there anything I can do so that the file system works faster with such a
 large number of files?  I'm looking for an increase in the area of 5 to 1.
  For example, the command rm -rf misc where misc is the directory
 containing the 250,000 files takes a couple of hours to run.  If misc is
 my current working directory, and I type rm * I get the message that
 there are too many arguments being passed into rm.
 
 Some details:
 FreeBSD 4.5, dmesg below.
 Filesystem containing the files: /dev/da0s1f on /usr (ufs, local, with
 quotas)

would this help? (I don't know if this was available in 4.5, look in
your /sys/i386/conf/LINT)

roman@freepuppy ~ 1003:0  grep -B2 DIRHASH /sys/i386/conf/LINT
# Directory hashing improves the speed of operations on very large
# directories at the expense of some memory.
options UFS_DIRHASH

-- 
If you cc me or remove the list(s) completely I'll most likely ignore
your message.see http://www.eyrie.org./~eagle/faqs/questions.html

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-questions in the body of the message



Re: Lots of files in a directory

2003-01-02 Thread Bill Moran
From: MikeM [EMAIL PROTECTED]

I'm considering setting up my server as a mirror site for the freedb.org
lookup database.  Unfortunately, I've seem to have run into a stumbling
block.  The server app requires over 250,000 files in a single directory.
Each file is about 2k in size.

It was a surprise to me (but probably not to those on this list) that the
file system does not handle that many files in an expeditious manner (I'm
being kind here).

Is there anything I can do so that the file system works faster with such a
large number of files?  I'm looking for an increase in the area of 5 to 1.
 For example, the command rm -rf misc where misc is the directory
containing the 250,000 files takes a couple of hours to run.  If misc is
my current working directory, and I type rm * I get the message that
there are too many arguments being passed into rm.


Try the dirhash option in your kernel (as suggested elsewhere).

As for commands like rm, you have a few options (as I was recently taught).
find(1) appears to handle large numbers of files, thus you can use it as
a pipe for just about anything, i.e.:
find /path/to/misc -name '*' -print0 | xargs -0 rm
(although you don't really need the pipe for rm, but it's an example)

I don't know how much of a problem this is for you, but it will work for
mv, cp, grep ... just about anything.  The quotes around * are important.

Also, there is a sysctl (kern.argmax) that you can tweak to increase the
length of command line arguments that can be processed, but I don't know
how big you'd have to make it to handle 250,000 files!

Good luck,
Bill

_
MSN 8 with e-mail virus protection service: 2 months FREE* 
http://join.msn.com/?page=features/virus


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-questions in the body of the message


Re: Lots of files in a directory

2003-01-02 Thread MikeM
On 1/2/03 at 3:13 PM Roman Neuhauser wrote:

|# [EMAIL PROTECTED] / 2003-01-02 09:01:27 -0500:
| I'm considering setting up my server as a mirror site for the freedb.org
| lookup database.  Unfortunately, I've seem to have run into a stumbling
| block.  The server app requires over 250,000 files in a single
directory.
| Each file is about 2k in size.
| 
| It was a surprise to me (but probably not to those on this list) that
the
| file system does not handle that many files in an expeditious manner
(I'm
| being kind here). 
| 
|
|would this help? (I don't know if this was available in 4.5, look in
|your /sys/i386/conf/LINT)
|
|roman@freepuppy ~ 1003:0  grep -B2 DIRHASH /sys/i386/conf/LINT
|# Directory hashing improves the speed of operations on very large
|# directories at the expense of some memory.
|options UFS_DIRHASH
 =


Thanks for the quick reply.

I already have UFS_DIRHASH specified in my kernel config file.  :-(




To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-questions in the body of the message



Re: Lots of files in a directory

2003-01-02 Thread Mikko Työläjärvi
On Thu, 2 Jan 2003, MikeM wrote:

 On 1/2/03 at 3:13 PM Roman Neuhauser wrote:

 |# [EMAIL PROTECTED] / 2003-01-02 09:01:27 -0500:
 | I'm considering setting up my server as a mirror site for the freedb.org
 | lookup database.  Unfortunately, I've seem to have run into a stumbling
 | block.  The server app requires over 250,000 files in a single
 directory.
 | Each file is about 2k in size.

It can be argued that the application is poorly designed...

[...]

 I already have UFS_DIRHASH specified in my kernel config file.  :-(

Maybe you can tweak some sysctls:

  atlas% sysctl -a | grep dirhash
  vfs.ufs.dirhash_minsize: 2560
  vfs.ufs.dirhash_maxmem: 2097152
  vfs.ufs.dirhash_mem: 795239
  vfs.ufs.dirhash_docheck: 0

Try playing with with vfs.ufs.dirhash_maxmem.  I suppose it has to
be at least as large as the size of the Directory From Hell.

Also remember that any application that lists all of those files
sequentially (as opposed to accessing a single one by name) will be
very slow, no matter what.  And don't forget that ls and shell
expansion of * will not only read through all entries, but also
sort them...

It would probably be better to patch the dang server to use a
multi-level directory structure instead.

$.02,
/Mikko


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-questions in the body of the message



Re: Lots of files in a directory

2003-01-02 Thread MikeM
On 1/2/03 at 8:58 AM Mikko Työläjärvi wrote:

|On Thu, 2 Jan 2003, MikeM wrote:
|
| On 1/2/03 at 3:13 PM Roman Neuhauser wrote:
|
| |# [EMAIL PROTECTED] / 2003-01-02 09:01:27 -0500:
| | I'm considering setting up my server as a mirror site for the
| | freedb.org lookup database.  Unfortunately, I've seem to have
| | run into a stumbling block.  The server app requires over
| | 250,000 files in a single directory. Each file is about 2k
| | in size.
|
|It can be argued that the application is poorly designed...

There's no arguing.  It *is* poorly designed.   I would have used a hash to
spread the files across multiple directories.   Unfortunately, I didn't
design it.


|
| I already have UFS_DIRHASH specified in my kernel config file.  :-(
|
|Maybe you can tweak some sysctls:
|
|  atlas% sysctl -a | grep dirhash
|  vfs.ufs.dirhash_minsize: 2560
|  vfs.ufs.dirhash_maxmem: 2097152
|  vfs.ufs.dirhash_mem: 795239
|  vfs.ufs.dirhash_docheck: 0
|
|Try playing with with vfs.ufs.dirhash_maxmem.  I suppose it has to
|be at least as large as the size of the Directory From Hell.

Thanks, I'll give them a try.



|...
|It would probably be better to patch the dang server to use a
|multi-level directory structure instead.


That thought has crossed my mind.

Many thanks for the assist.


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-questions in the body of the message



Re: Lots of files in a directory

2003-01-02 Thread MikeM
On 1/2/03 at 11:38 AM Dan Nelson wrote:

|In the last episode (Jan 02), MikeM said:
| Is there anything I can do so that the file system works faster with
| such a large number of files?  I'm looking for an increase in the
| area of 5 to 1. For example, the command rm -rf misc where misc
| is the directory containing the 250,000 files takes a couple of hours
| to run.  If misc is my current working directory, and I type rm *
| I get the message that there are too many arguments being passed into
| rm.
|
|In addition to the other suggestions, enabling softupdates will make
|file creation and deletion much faster (your rm -rf will speed up
|~100x, for example).
 =

Thanks.   tunefs for enabling softupdates seems to require console access
and single user boot mode.  So I'll have to schedule that on my distant
server.  




To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-questions in the body of the message



Re: Lots of files in a directory

2003-01-02 Thread Dan Nelson
In the last episode (Jan 02), MikeM said:
 It was a surprise to me (but probably not to those on this list) that
 the file system does not handle that many files in an expeditious
 manner (I'm being kind here).
 
 Is there anything I can do so that the file system works faster with
 such a large number of files?  I'm looking for an increase in the
 area of 5 to 1. For example, the command rm -rf misc where misc
 is the directory containing the 250,000 files takes a couple of hours
 to run.  If misc is my current working directory, and I type rm *
 I get the message that there are too many arguments being passed into
 rm.

In addition to the other suggestions, enabling softupdates will make
file creation and deletion much faster (your rm -rf will speed up
~100x, for example).

If you're doing file operations directly on known filenames, then the
dirhash suggestions will really help.  If freedb is doing directory
scans or wildcard globbing, then no filesystem will save you :)

-- 
Dan Nelson
[EMAIL PROTECTED]

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-questions in the body of the message