Re: [SLUG] Linux and Apache limits on number of files in a directory
When I downloaded the players in a single directory on my computer after download the directory management took it toll on my application, following the same directory structure had a massive improvement on performance, a simple divide gave me the directory. It would be worth noting, that caching would make the situation slightly different on a webserver. Marghanita -- Marghanita da Cruz Ramin Communications (Sydney) Website: http://ramin.com.au Phone:(+612) 0414-869202 -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Linux and Apache limits on number of files in a directory
Subsequent access yes, but the first access no. In addition, I still think having 1000 directories (each with so many files under each of those directories) is vastly better compared to 1 million files in a single directory. Very easy choice for managing using the first example.. web proxy or not :) On Thu, Apr 5, 2012 at 9:40 AM, Marghanita da Cruz marghan...@ramin.com.auwrote: When I downloaded the players in a single directory on my computer after download the directory management took it toll on my application, following the same directory structure had a massive improvement on performance, a simple divide gave me the directory. It would be worth noting, that caching would make the situation slightly different on a webserver. Marghanita -- Marghanita da Cruz Ramin Communications (Sydney) Website: http://ramin.com.au Phone:(+612) 0414-869202 -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/**mailinglists.htmlhttp://slug.org.au/faq/mailinglists.html -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Linux and Apache limits on number of files in a directory
Michael Fox wrote: Subsequent access yes, but the first access no. In addition, I still think having 1000 directories (each with so many files under each of those directories) is vastly better compared to 1 million files in a single directory. Good point, I just checked I am at 1406 files in the directory, so probably time to start spreading files across multiple directories. The ftp ls was slow and it probably also slows writing (uploading replacing files). At some time in the past, there was a suggestion about keeping files of particular types together in separate directories. Does anyone have any comments on this? Very easy choice for managing using the first example.. web proxy or not :) On Thu, Apr 5, 2012 at 9:40 AM, Marghanita da Cruz marghan...@ramin.com.au mailto:marghan...@ramin.com.au wrote: When I downloaded the players in a single directory on my computer after download the directory management took it toll on my application, following the same directory structure had a massive improvement on performance, a simple divide gave me the directory. It would be worth noting, that caching would make the situation slightly different on a webserver. Marghanita -- Marghanita da Cruz Ramin Communications (Sydney) Website: http://ramin.com.au Phone:(+612) 0414-869202 -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/__mailinglists.html http://slug.org.au/faq/mailinglists.html -- Marghanita da Cruz Ramin Communications (Sydney) Website: http://ramin.com.au Phone:(+612) 0414-869202 -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Linux and Apache limits on number of files in a directory
I wouldn't expect 1500 files in a dir to be slow (from the file system POV, the ftp server is doing who knows what). slow would probably start to kick in ~10-30k files (and in general that should still be fairly quick, opening the file would probably take on the order of half a second). Look at all the maildir based email systems out there, with 60k+ emails in a folder. On 04/05/2012 12:50 PM, Marghanita da Cruz wrote: Michael Fox wrote: Subsequent access yes, but the first access no. In addition, I still think having 1000 directories (each with so many files under each of those directories) is vastly better compared to 1 million files in a single directory. Good point, I just checked I am at 1406 files in the directory, so probably time to start spreading files across multiple directories. The ftp ls was slow and it probably also slows writing (uploading replacing files). At some time in the past, there was a suggestion about keeping files of particular types together in separate directories. Does anyone have any comments on this? Very easy choice for managing using the first example.. web proxy or not :) On Thu, Apr 5, 2012 at 9:40 AM, Marghanita da Cruz marghan...@ramin.com.au mailto:marghan...@ramin.com.au wrote: When I downloaded the players in a single directory on my computer after download the directory management took it toll on my application, following the same directory structure had a massive improvement on performance, a simple divide gave me the directory. It would be worth noting, that caching would make the situation slightly different on a webserver. Marghanita -- Marghanita da Cruz Ramin Communications (Sydney) Website: http://ramin.com.au Phone:(+612) 0414-869202 -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/__mailinglists.html http://slug.org.au/faq/mailinglists.html -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Linux and Apache limits on number of files in a directory
imho if you have so many files, it might be worth looking at a CMS where a record can be created for each item and rendered on demand. On 4/5/12, Jake Anderson ya...@vapourforge.com wrote: I wouldn't expect 1500 files in a dir to be slow (from the file system POV, the ftp server is doing who knows what). slow would probably start to kick in ~10-30k files (and in general that should still be fairly quick, opening the file would probably take on the order of half a second). Look at all the maildir based email systems out there, with 60k+ emails in a folder. On 04/05/2012 12:50 PM, Marghanita da Cruz wrote: Michael Fox wrote: Subsequent access yes, but the first access no. In addition, I still think having 1000 directories (each with so many files under each of those directories) is vastly better compared to 1 million files in a single directory. Good point, I just checked I am at 1406 files in the directory, so probably time to start spreading files across multiple directories. The ftp ls was slow and it probably also slows writing (uploading replacing files). At some time in the past, there was a suggestion about keeping files of particular types together in separate directories. Does anyone have any comments on this? Very easy choice for managing using the first example.. web proxy or not :) On Thu, Apr 5, 2012 at 9:40 AM, Marghanita da Cruz marghan...@ramin.com.au mailto:marghan...@ramin.com.au wrote: When I downloaded the players in a single directory on my computer after download the directory management took it toll on my application, following the same directory structure had a massive improvement on performance, a simple divide gave me the directory. It would be worth noting, that caching would make the situation slightly different on a webserver. Marghanita -- Marghanita da Cruz Ramin Communications (Sydney) Website: http://ramin.com.au Phone:(+612) 0414-869202 -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/__mailinglists.html http://slug.org.au/faq/mailinglists.html -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
