Re: Lots of files in a directory
On 1/3/2003 at 9:25 AM MikeM wrote: | [snip] | |To you and others who have helped me, here's what I'm doing. | |- tomorrow I am upgrading from 4.5 to 4.7 |- once that proves stable, I'll increase the DIRHASH value appropriately. |- once that proves stable, I'll tunefs -n enable /usr to turn on soft |dependencies | |I'll report back with my experiences with the new configuration so that the |archive remains complete. = The upgrade to 4.7 went very smoothly. Many thanks to the documentation team for the excellent documentation on upgrading (http://www.freebsd.org/releases/4.7R/relnotes-i386.html#AEN577). This was my first major upgrade on a remote server (I have been keeping up-to-date with the security patches). Needless to say, I was a bit apprehensive when I typed that reboot command. I never thought I'd be so very happy to see the words Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 4.7-RELEASE-p2 ... appear on my ssh screen. :-) Next I did the tunefs step. I had to ask the techs at the datacenter to boot into single user mode and perform the tunefs on the local console. That also went smoothly. When I tried to tar -yxf the freedb archive, the server became unresponsive. I am not sure exactly what the root cause of the problem was, as my ssh session went dead and I could not re-establish a connection. Long story short, I found out the tech support in the data center (www.tera-byte.com) is excellent. The tech re-established the local console and ran the fsck required to bring the server back to life. He did mention the following in one of his emails, [the server] appears to be having some problems with the fsck due to some dirty inodes... . The server is back online now, and running well. Trying to duplicate the problem, I ran a similar sequence sequence of events on a server here at my house. It didn't crash, but I did get screenfuls of the following error message: /usr: create/symlink failed, no inodes free tar: misc/ed11d70f: Cannot open: No space left on device That message was repeated for each of the hundreds of thousands of files. I suspect 'something' on my remote server was not happy about dealing with such a flood of error messages. At this point, I'm not willing to newfs the file system to get the number of inodes that I need. So I think, so far as I am concerned, this experiment is over. :-) Once again, thanks to everyone who gave me help on this. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: Lots of files in a directory
On 2003-01-04 09:04, MikeM [EMAIL PROTECTED] wrote: The upgrade to 4.7 went very smoothly. [...] When I tried to tar -yxf the freedb archive, the server became unresponsive. [...] The server is back online now, and running well. Trying to duplicate the problem, I ran a similar sequence sequence of events on a server here at my house. It didn't crash, but I did get screenfuls of the following error message: /usr: create/symlink failed, no inodes free tar: misc/ed11d70f: Cannot open: No space left on device What does `df -i' report for your /usr filesystem? Every file needs an i-node of itself on a ufs filesystem, and having many thousands of files takes many thousands of i-nodes... At home, I can see something like the following: giorgos@gothmog[20:36]/home/giorgos$ df -i /usr Filesystem 1K-blocksUsedAvail Capacity iused ifree %iused Mounted on /dev/ad0s3g 4636750 1793606 247220442% 205145 377253 35% /usr The iused, ifree and %iused columns are those you're interested in. - Giorgos To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: Lots of files in a directory
On 2003-01-02 09:01, MikeM [EMAIL PROTECTED] wrote: I'm considering setting up my server as a mirror site for the freedb.org lookup database. Unfortunately, I've seem to have run into a stumbling block. The server app requires over 250,000 files in a single directory. Each file is about 2k in size. On 2003-01-02 12:13, MikeM [EMAIL PROTECTED] wrote: On 1/2/03 at 8:58 AM Mikko Ty?l?j?rvi wrote: It can be argued that the application is poorly designed... There's no arguing. It *is* poorly designed. I would have used a hash to spread the files across multiple directories. Unfortunately, I didn't design it. You are probably hitting the limit of fragments that are allowed on a filesystem and time optimization becomes space optimization. Are you getting entries in your system logs like the following? optimization changed from TIME to SPACE - Giorgos To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: Lots of files in a directory
On 1/3/03 at 6:45 AM Giorgos Keramidas wrote: |On 2003-01-02 09:01, MikeM [EMAIL PROTECTED] wrote: | I'm considering setting up my server as a mirror site for the freedb.org | lookup database. Unfortunately, I've seem to have run into a stumbling | block. The server app requires over 250,000 files in a single directory. | Each file is about 2k in size. | |On 2003-01-02 12:13, MikeM [EMAIL PROTECTED] wrote: | On 1/2/03 at 8:58 AM Mikko Ty?l?j?rvi wrote: | It can be argued that the application is poorly designed... | | There's no arguing. It *is* poorly designed. I would have used a | hash to spread the files across multiple directories. | Unfortunately, I didn't design it. | |You are probably hitting the limit of fragments that are allowed on a |filesystem and time optimization becomes space optimization. Are |you getting entries in your system logs like the following? | | optimization changed from TIME to SPACE = I just grep'd /var/log/* and the word optimization did not appear. To you and others who have helped me, here's what I'm doing. - tomorrow I am upgrading from 4.5 to 4.7 - once that proves stable, I'll increase the DIRHASH value appropriately. - once that proves stable, I'll tunefs -n enable /usr to turn on soft dependencies I'll report back with my experiences with the new configuration so that the archive remains complete. Thanks to everyone who helped. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Lots of files in a directory
I'm considering setting up my server as a mirror site for the freedb.org lookup database. Unfortunately, I've seem to have run into a stumbling block. The server app requires over 250,000 files in a single directory. Each file is about 2k in size. It was a surprise to me (but probably not to those on this list) that the file system does not handle that many files in an expeditious manner (I'm being kind here). Is there anything I can do so that the file system works faster with such a large number of files? I'm looking for an increase in the area of 5 to 1. For example, the command rm -rf misc where misc is the directory containing the 250,000 files takes a couple of hours to run. If misc is my current working directory, and I type rm * I get the message that there are too many arguments being passed into rm. Some details: FreeBSD 4.5, dmesg below. Filesystem containing the files: /dev/da0s1f on /usr (ufs, local, with quotas) Copyright (c) 1992-2002 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 4.5-RELEASE-p20 #12: Sat Sep 28 11:03:59 EDT 2002 [snip]:/usr/obj/usr/src/sys/STABLE4FW Timecounter i8254 frequency 1193182 Hz CPU: Pentium III/Pentium III Xeon/Celeron (797.48-MHz 686-class CPU) Origin = GenuineIntel Id = 0x686 Stepping = 6 Features=0x383fbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MC A,CMOV,PAT,PSE36,MMX,FXSR,SSE real memory = 805224448 (786352K bytes) avail memory = 778473472 (760228K bytes) Preloaded elf kernel kernel at 0xc0468000. Pentium Pro MTRR support enabled md0: Malloc disk npx0: math processor on motherboard npx0: INT 16 interface pcib0: ServerWorks NB6635 3.0LE host to PCI bridge on motherboard pci0: PCI bus on pcib0 pci0: S3 Savage 4 graphics accelerator at 1.0 fxp0: Intel Pro 10/100B/100+ Ethernet port 0x2200-0x223f mem 0xfea0-0xfeaf,0xfeb7f000-0xfeb7 irq 10 at device 2.0 on pci0 fxp0: Ethernet address 00:06:29:1f:27:61 inphy0: i82555 10/100 media interface on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto isab0: ServerWorks IB6566 PCI to ISA bridge at device 15.0 on pci0 isa0: ISA bus on isab0 atapci0: ServerWorks ROSB4 ATA33 controller port 0x700-0x70f at device 15.1 on pci0 ata0: at 0x1f0 irq 14 on atapci0 ohci0: OHCI (generic) USB controller mem 0xfeb7e000-0xfeb7efff irq 7 at device 15.2 on pci0 usb0: OHCI version 1.0, legacy support usb0: OHCI (generic) USB controller on ohci0 usb0: USB revision 1.0 uhub0: (unknown) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered pcib1: ServerWorks NB6635 3.0LE host to PCI bridge on motherboard pci1: PCI bus on pcib1 ahc0: Adaptec aic7892 Ultra160 SCSI adapter port 0x2300-0x23ff mem 0xe000-0xefff irq 9 at device 3.0 on pci1 aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/255 SCBs orm0: Option ROM at iomem 0xc-0xc9fff on isa0 fdc0: direction bit not set fdc0: cmd 3 failed at out byte 1 of 3 atkbdc0: Keyboard controller (i8042) at port 0x60,0x64 on isa0 vga0: Generic ISA VGA at port 0x3c0-0x3df iomem 0xa-0xb on isa0 sc0: System console at flags 0x100 on isa0 sc0: VGA 16 virtual consoles, flags=0x100 sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 sio0: type 16550A, console sio1: configured irq 3 not in bitmap of probed irqs 0 ppc0: parallel port not found. IP packet filtering initialized, divert disabled, rule-based forwarding disabled, default to deny, logging limited to 10 packets/entry by default acd0: CDROM CRN-8241B at ata0-master using PIO4 Waiting 15 seconds for SCSI devices to settle pass2 at ahc0 bus 0 target 8 lun 0 pass2: IBM FTlV1 S2 0 Fixed Processor SCSI-2 device pass2: 3.300MB/s transfers da0 at ahc0 bus 0 target 0 lun 0 da0: IBM-PSG DDYS-T18350M M S9AA Fixed Direct Access SCSI-3 device da0: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing Enabled da0: 17357MB (35548320 512 byte sectors: 255H 63S/T 2212C) da1 at ahc0 bus 0 target 1 lun 0 da1: IBM-ESXS ST336605LC!# B243 Fixed Direct Access SCSI-3 device da1: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing Enabled da1: 34715MB (71096640 512 byte sectors: 255H 63S/T 4425C) Mounting root from ufs:/dev/da0s1a To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: Lots of files in a directory
# [EMAIL PROTECTED] / 2003-01-02 09:01:27 -0500: I'm considering setting up my server as a mirror site for the freedb.org lookup database. Unfortunately, I've seem to have run into a stumbling block. The server app requires over 250,000 files in a single directory. Each file is about 2k in size. It was a surprise to me (but probably not to those on this list) that the file system does not handle that many files in an expeditious manner (I'm being kind here). Is there anything I can do so that the file system works faster with such a large number of files? I'm looking for an increase in the area of 5 to 1. For example, the command rm -rf misc where misc is the directory containing the 250,000 files takes a couple of hours to run. If misc is my current working directory, and I type rm * I get the message that there are too many arguments being passed into rm. Some details: FreeBSD 4.5, dmesg below. Filesystem containing the files: /dev/da0s1f on /usr (ufs, local, with quotas) would this help? (I don't know if this was available in 4.5, look in your /sys/i386/conf/LINT) roman@freepuppy ~ 1003:0 grep -B2 DIRHASH /sys/i386/conf/LINT # Directory hashing improves the speed of operations on very large # directories at the expense of some memory. options UFS_DIRHASH -- If you cc me or remove the list(s) completely I'll most likely ignore your message.see http://www.eyrie.org./~eagle/faqs/questions.html To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: Lots of files in a directory
From: MikeM [EMAIL PROTECTED] I'm considering setting up my server as a mirror site for the freedb.org lookup database. Unfortunately, I've seem to have run into a stumbling block. The server app requires over 250,000 files in a single directory. Each file is about 2k in size. It was a surprise to me (but probably not to those on this list) that the file system does not handle that many files in an expeditious manner (I'm being kind here). Is there anything I can do so that the file system works faster with such a large number of files? I'm looking for an increase in the area of 5 to 1. For example, the command rm -rf misc where misc is the directory containing the 250,000 files takes a couple of hours to run. If misc is my current working directory, and I type rm * I get the message that there are too many arguments being passed into rm. Try the dirhash option in your kernel (as suggested elsewhere). As for commands like rm, you have a few options (as I was recently taught). find(1) appears to handle large numbers of files, thus you can use it as a pipe for just about anything, i.e.: find /path/to/misc -name '*' -print0 | xargs -0 rm (although you don't really need the pipe for rm, but it's an example) I don't know how much of a problem this is for you, but it will work for mv, cp, grep ... just about anything. The quotes around * are important. Also, there is a sysctl (kern.argmax) that you can tweak to increase the length of command line arguments that can be processed, but I don't know how big you'd have to make it to handle 250,000 files! Good luck, Bill _ MSN 8 with e-mail virus protection service: 2 months FREE* http://join.msn.com/?page=features/virus To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: Lots of files in a directory
On 1/2/03 at 8:58 AM Mikko Työläjärvi wrote: |On Thu, 2 Jan 2003, MikeM wrote: | | On 1/2/03 at 3:13 PM Roman Neuhauser wrote: | | |# [EMAIL PROTECTED] / 2003-01-02 09:01:27 -0500: | | I'm considering setting up my server as a mirror site for the | | freedb.org lookup database. Unfortunately, I've seem to have | | run into a stumbling block. The server app requires over | | 250,000 files in a single directory. Each file is about 2k | | in size. | |It can be argued that the application is poorly designed... There's no arguing. It *is* poorly designed. I would have used a hash to spread the files across multiple directories. Unfortunately, I didn't design it. | | I already have UFS_DIRHASH specified in my kernel config file. :-( | |Maybe you can tweak some sysctls: | | atlas% sysctl -a | grep dirhash | vfs.ufs.dirhash_minsize: 2560 | vfs.ufs.dirhash_maxmem: 2097152 | vfs.ufs.dirhash_mem: 795239 | vfs.ufs.dirhash_docheck: 0 | |Try playing with with vfs.ufs.dirhash_maxmem. I suppose it has to |be at least as large as the size of the Directory From Hell. Thanks, I'll give them a try. |... |It would probably be better to patch the dang server to use a |multi-level directory structure instead. That thought has crossed my mind. Many thanks for the assist. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: Lots of files in a directory
On 1/2/03 at 11:38 AM Dan Nelson wrote: |In the last episode (Jan 02), MikeM said: | Is there anything I can do so that the file system works faster with | such a large number of files? I'm looking for an increase in the | area of 5 to 1. For example, the command rm -rf misc where misc | is the directory containing the 250,000 files takes a couple of hours | to run. If misc is my current working directory, and I type rm * | I get the message that there are too many arguments being passed into | rm. | |In addition to the other suggestions, enabling softupdates will make |file creation and deletion much faster (your rm -rf will speed up |~100x, for example). = Thanks. tunefs for enabling softupdates seems to require console access and single user boot mode. So I'll have to schedule that on my distant server. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: Lots of files in a directory
In the last episode (Jan 02), MikeM said: It was a surprise to me (but probably not to those on this list) that the file system does not handle that many files in an expeditious manner (I'm being kind here). Is there anything I can do so that the file system works faster with such a large number of files? I'm looking for an increase in the area of 5 to 1. For example, the command rm -rf misc where misc is the directory containing the 250,000 files takes a couple of hours to run. If misc is my current working directory, and I type rm * I get the message that there are too many arguments being passed into rm. In addition to the other suggestions, enabling softupdates will make file creation and deletion much faster (your rm -rf will speed up ~100x, for example). If you're doing file operations directly on known filenames, then the dirhash suggestions will really help. If freedb is doing directory scans or wildcard globbing, then no filesystem will save you :) -- Dan Nelson [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
