Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-25 Thread Marc Deop
On Sunday 24 July 2011 10:13:30 R P Herrold wrote: #!/bin/sh # CANDIDATES=pix1.jpg pix2.jpg pix3.jpg for i in `echo ${CANDIDATES}`; do HASH=`echo $i | md5sum - | awk {'print $1'}` echo $i${HASH} done I know it absolutelly has nothing to do with

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-25 Thread Lamar Owen
On Sunday, July 24, 2011 05:29:23 AM yonatan pingle wrote: ... lately the server is under-preforming and load averages are high, mysql service keeps crashing and the server is hitting max memory usage ( so i added ram .. ) , after looking into the website folders, i have found one folder which

[CentOS] lots of small files in a folder on Linux centos

2011-07-25 Thread R P Herrold
On Mon, 25 Jul 2011, Marc Deop wrote: It's more than twice as fast than the previous sh script. In part this is /bin/sh v /bin/bash and using 'bashisms' matter, but yes, I did not seek to optimize a teaching throwaway 1- m5sum the file we need ... actually the NAME of the file, to make it

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Eero Volotinen
2011/7/24 yonatan pingle yonatan.pin...@gmail.com: Hello, I have a rather annoying issue on going with one of my centos virtual servers. the server hosts a website using apache and mysql ,there are three persons involved with keeping the site up and running. and i am his root due to the fact

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Alexander Dalloz
Am 24.07.2011 13:03, schrieb Eero Volotinen: 2011/7/24 yonatan pingle yonatan.pin...@gmail.com: uploads]# ls | wc -l 3123 I assume that you are using ext3 or ext4 filesystems? Both ext3 and ext4 slows down, if there is too much files in same directory. XFS-fs is solution to fix this

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread yonatan pingle
On Sun, Jul 24, 2011 at 2:19 PM, Alexander Dalloz ad+li...@uni-x.org wrote: Am 24.07.2011 13:03, schrieb Eero Volotinen: 2011/7/24 yonatan pingle yonatan.pin...@gmail.com: uploads]# ls | wc -l 3123 I assume that you are using ext3 or ext4 filesystems? Both ext3 and ext4 slows down, if

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Ljubomir Ljubojevic
yonatan pingle wrote: On Sun, Jul 24, 2011 at 2:19 PM, Alexander Dalloz ad+li...@uni-x.org wrote: Am 24.07.2011 13:03, schrieb Eero Volotinen: 2011/7/24 yonatan pingle yonatan.pin...@gmail.com: uploads]# ls | wc -l 3123 I assume that you are using ext3 or ext4 filesystems? Both ext3 and

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Ryan Wagoner
On Sun, Jul 24, 2011 at 7:52 AM, yonatan pingle yonatan.pin...@gmail.com wrote: Hi, Alexander good suggestions, ill monitor I/O and mysql code, sounds like a code related issue and not a centos issue after all. it runs on ext3  ,i could only guess how to code deals with the dir, as it seems

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread yonatan pingle
Do you have cahcing turned on in CMS? That could help. -- Ljubomir Ljubojevic (Love is in the Air) PL Computers Serbia, Europe Google is the Mother, Google is the Father, and traceroute is your trusty Spiderman... StarOS, Mikrotik and CentOS/RHEL/Linux consultant

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread yonatan pingle
RHCT | RHCSA | CCNA1 If you are using phpMyAdmin the status page will aid you in tuning mySQL. Look for values in red. The description will usually tell you what to adjust to improve performance. Ryan ___ CentOS mailing list CentOS@centos.org

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Ryan Wagoner
On Sun, Jul 24, 2011 at 8:40 AM, yonatan pingle yonatan.pin...@gmail.com wrote: im good with mysqltuner.pl, as it seems there are slow queries on mysql and i have adjusted all values in my.cnf according to the application needs. looks like it's all in the code and the way the CMS handles the

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread yonatan pingle
On Sun, Jul 24, 2011 at 3:43 PM, Ryan Wagoner rswago...@gmail.com wrote: On Sun, Jul 24, 2011 at 8:40 AM, yonatan pingle yonatan.pin...@gmail.com wrote: im good with mysqltuner.pl, as it seems there are slow queries on mysql and i have adjusted all values in my.cnf according to the

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread John R. Dennison
On Sun, Jul 24, 2011 at 03:53:46PM +0300, yonatan pingle wrote: Yes Ryan, that exactly what i have done. he will get the log shortly and i will get some not free beer. While I'm all for mysql optimization it's clearly evident from an earlier posting that your disks are thrashing with insanely

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Diego Sanchez
2011/7/24 yonatan pingle yonatan.pin...@gmail.com: there is no caching system, its a home made CMS. You can use an accelerator too. http://en.wikipedia.org/wiki/PHP_accelerator http://en.wikipedia.org/wiki/List_of_PHP_accelerators Please, make a big backup before this! (I nevever had a

[CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread R P Herrold
On Sun, 24 Jul 2011, yonatan pingle wrote: the coder is not tech savvy as one might expect, so it's really hard for me to explain the issue of having lots of files in one folder to the site owner or to the coder. I do not expect coders to remain 'not tech savvy' If the coder is not willing

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread yonatan pingle
On Sun, Jul 24, 2011 at 4:02 PM, John R. Dennison j...@gerdesas.com wrote: On Sun, Jul 24, 2011 at 03:53:46PM +0300, yonatan pingle wrote: Yes Ryan, that exactly what i have done. he will get the log shortly and i will get some not free beer. While I'm all for mysql optimization it's clearly

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread yonatan pingle
On Sun, Jul 24, 2011 at 5:13 PM, R P Herrold herr...@owlriver.com wrote: On Sun, 24 Jul 2011, yonatan pingle wrote: the coder is not tech savvy as one might expect, so it's really hard for me to explain the issue of having lots of files in one folder to the site owner or to the coder. I do

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Always Learning
On Sun, Jul 24, 2011 at 5:13 PM, R P Herrold herr...@owlriver.com wrote: then, we look to the leading letter of the hask, to design our egg carton bins. We place pix1.jpg in directory: ./f/ and pix2.jpg in directory ./1/ and pix3.jpg in directory ./b/ and so forth -- if the

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Marian Marinov
On Sunday 24 July 2011 22:48:23 Always Learning wrote: On Sun, Jul 24, 2011 at 5:13 PM, R P Herrold herr...@owlriver.com wrote: then, we look to the leading letter of the hask, to design our egg carton bins. We place pix1.jpg in directory: ./f/ and pix2.jpg in directory ./1/

[CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread R P Herrold
On Sun, 24 Jul 2011, Always Learning wrote: If the pictures are named sequentially, why not store then at a 100 per directory structure something like this /pix/0/00/pix1.jpg /pix/0/26/pix02614.jpg /pix/6/72/pix67255.jpg Go read Knuth One does not do that because then one is

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Keith Roberts
On Sun, 24 Jul 2011, R P Herrold wrote: By using a hash, we remove those constraints, and also gain the virtuous effect for free of self-organizing a relatively level dispersion of files to the destination directories Not followed the whole thread, but a SQL database index of the actual

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Always Learning
On Sun, 2011-07-24 at 16:33 -0400, R P Herrold wrote: On Sun, 24 Jul 2011, Always Learning wrote: If the pictures are named sequentially, why not store then at a 100 per directory structure something like this /pix/0/00/pix1.jpg /pix/0/26/pix02614.jpg

[CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread R P Herrold
On Sun, 24 Jul 2011, Keith Roberts wrote: By using a hash, we remove those constraints, and also gain the virtuous effect for free of self-organizing a relatively level dispersion of files to the destination directories Not followed the whole thread, but a SQL database index of the actual

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Always Learning
On Sun, 2011-07-24 at 17:50 -0400, R P Herrold wrote: On Sun, 24 Jul 2011, Keith Roberts wrote: By using a hash, we remove those constraints, and also gain the virtuous effect for free of self-organizing a relatively level dispersion of files to the destination directories Not

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Rajagopal Swaminathan
Greetings, On Sun, Jul 24, 2011 at 2:59 PM, yonatan pingle yonatan.pin...@gmail.com wrote: Hello, after looking into the website folders, i have found one folder which from my point of view is one of the causes for the server loads. hmm... does mount dir -noatime -noadirtime help speed it

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread John R. Dennison
On Mon, Jul 25, 2011 at 06:38:33AM +0530, Rajagopal Swaminathan wrote: hmm... does mount dir -noatime -noadirtime help speed it up? Just an FYI: noatime is a superset that includes noadirtime. John -- You can safely assume you've

Re: [CentOS] lots of small files in a folder on Linux centos

2011-07-24 Thread Les Mikesell
On 7/24/11 4:08 PM, Keith Roberts wrote: On Sun, 24 Jul 2011, R P Herrold wrote: By using a hash, we remove those constraints, and also gain the virtuous effect for free of self-organizing a relatively level dispersion of files to the destination directories Not followed the whole thread,