Thanks for your answer Bill! (and to Will as well),
Some more infos I gathered a few minutes ago:
[~/templates_c]$ date; du -s -m ; date
Wed May 6 13:35:15 CEST 2009
2652 .
Wed May 6 13:52:36 CEST 2009
[~/templates_c]$ date ; find . | wc -l ; date
Wed May 6 13:52:56 CEST 2009
305461
Wed May 6 14:09:39 CEST 2009
So this is on the system after a complete cache cleanup (at 00h00).
300'000 files and 2.6GB. So this night, there were probably around 3-4
million files to delete.
Deletion may take time, but 20 minutes juste to _count_ all the files
seems pretty long to me... I think I'll say a word to the developers
to let them tune their caching system a bit :)
On Wed, 2009-05-06 at 08:48 -0400, Bill Moran wrote:
> With lots of small files, the time involved is far less dependent on
> the size of data, and much more dependent on the number of files, and
> the resultant number of directory entries that need to be updated.
> "Lots" isn't a particularly accurate count of the # of files, but if
> you're talking web cache files, I'll guess they average 5k each, which
> means you had 6 million files. df -i would have been more useful in
> the output above.
Thanks, noted for next time. Now it looks like that:
Filesystem 1M-blocks Used Avail Capacity iusedifree %iused Mounted on
/dev/da0s1f128631 70544 4779560% 1913875 15114219 11% /usr
> This brings a number of questions up:
> * Are you _sure_ softupdates is enabled on that partition? That's
> going to make the biggest improvement in speed.
According to "mount" output, yes. I found no specific message about
that in the syslog or dmesg.
> * Are these 7200RPM disks or 15,000? Again, going to make a big
> difference.
HP 146GB 6G SAS 10K SFF DP ENT HDD (15k were not available at the time
the servers were ordered)
(
http://h18004.www1.hp.com/products/servers/proliantstorage/serial/sas/index.html
)
> * If apache was still running, is it possible that it was creating
> enough disk activity to slow the activity down? Running
> top -m io will show you how much disk IO each process is creating.
Yes, apache was still running, but the activity was quite low (it was
during the night, and the webpage doesn't get so many hits before 9 am
local time)
While watching "top -m io", the "du" or "find" takes between 80 and 99%,
so I guess it's not the probleme here:
PIDUID VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND
87996 100259 56 0 0 0 0 0.00% php
45389 100235 25 0 0 2 2 0.84% php
3964 1002 0 0 0 0 0 0 0.00% httpd
3822 1002 151 98 0 0 0 0 0.00% httpd
3005 1002 0 0 0 0 0 0 0.00% httpd
4129 1002 0 0 0 0 0 0 0.00% httpd
3971 1002 0 0 0 0 0 0 0.00% httpd
4231 1002 1 0 0 0 0 0 0.00% httpd
4132 0 234 5234 0 0234 97.91% find
98862 1002 1 0 0 0 0 0 0.00% top
609 0 0 0 0 0 0 0 0.00% snmpd
[...]
> * When you compared the speed to your laptop, did you delete 6 million
> files from the laptop? If you deleted a single 30G file, then you're
> comparing apples to atom bombs.
Yes sorry, I know :)
> If this is a directory that you blow away on a regular schedule, you'd
> do much better to make it a dedicated partition and simply reformat
> it.
Yes, it is one of the best options. My initial goal was to delete all
files older than N days by cron (find | xargs | rm, etc.), but if each
cronjob takes 2 hours (and takes so much cpu time), it's probably not
the best way.
I'll make some more tests on an test-server later this week and speak
with the devs. Thanks again for your very constructive feedback!
Regards,
Olivier
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
