Re: maximum number of files in directory
On Tuesday 11 April 2006 13:42, Gilbert Midonnet wrote: It is quite possible that there could be 15-20,000 images in this directory. Is that too many? I would say so. You may want to break it down into subdirs by initial character, 2nd character, and then the file or something. That is how high performance applications like Squid work anyway. You other schemes should work too. simpler to have all the graphics in one directory Only from the point of view of the single well-defined CFC encapsulating the getFilePathForItem() function, so not too much bother :-) -- Tom Chiverton Advanced ColdFusion Programmer ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:237409 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: maximum number of files in directory
or maybe categories (dresses, denim, jewelry) (the advantage of this would be fewer directories than designers) You could also use some better balanced subdivision. For instance, if you have some numerical Id nb, use the three last digits as the file name, and the rest for the directory. This way, when the directory has 1000 files, a new one is created, etc. Use something that cannot be changed in your database, so that you do not have to move the files when data is changed. Category would not be a good choice, since someone may choose the wrong category first, then update the product. -- ___ REUSE CODE! Use custom tags; See http://www.contentbox.com/claude/customtags/tagstore.cfm (Please send any spam to this address: [EMAIL PROTECTED]) Thanks. ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:237414 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
RE: maximum number of files in directory
I have also seen that if clusters on the hard drive go bad that affect the folder. All images in that folder are now inaccessible. -B -Original Message- From: Gilbert Midonnet [mailto:[EMAIL PROTECTED] Sent: Tuesday, April 11, 2006 5:42 AM To: CF-Talk Subject: maximum number of files in directory I'm creating a database for a store. It will probably have several thousand items (but will probably never come close to 10,000). The pictures of the items will be stored in a file directory (not as a blob in a database). What is the maximum amount of files that can be stored in a directory and still have an efficient response time. Each item will have at least two graphics if not three or four #itemID#-th #itemID# It is quite possible that there could be 15-20,000 images in this directory. Is that too many? The alternative is to store the graphics according to designer /graphics/items/#designerName# / #itemID#.jpg or maybe categories (dresses, denim, jewelry) (the advantage of this would be fewer directories than designers) /graphics/items/#categories# / #itemID#.jpg It would be so much simpler to have all the graphics in one directory. Will I be causing problems later on by doing that? Gilbert Midonnet 718.928.4524 www.glmdesigns.com ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:237418 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
RE: maximum number of files in directory
Thanks all, I was pretty sure that 10-20,000 was way too much but not certain how best to organize it. I'm still thinking it through. Thanks about the point regarding category. (Category would not be a good choice, since someone may choose the wrong category first, then update the product.) Since each item will have at least two accompanying photos I'll probably do something along the line you suggest but probably bring it down to 100 or 250 instead of a thousand. Thanks Gil -Original Message- From: Claude Schneegans [mailto:[EMAIL PROTECTED] Sent: Tuesday, April 11, 2006 9:58 AM To: CF-Talk Subject: Re: maximum number of files in directory or maybe categories (dresses, denim, jewelry) (the advantage of this would be fewer directories than designers) You could also use some better balanced subdivision. For instance, if you have some numerical Id nb, use the three last digits as the file name, and the rest for the directory. This way, when the directory has 1000 files, a new one is created, etc. Use something that cannot be changed in your database, so that you do not have to move the files when data is changed. Category would not be a good choice, since someone may choose the wrong category first, then update the product. -- ___ REUSE CODE! Use custom tags; See http://www.contentbox.com/claude/customtags/tagstore.cfm (Please send any spam to this address: [EMAIL PROTECTED]) Thanks. ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:237419 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
RE: maximum number of files in directory
I've served images from a Windows 2000 Server that was fine until it reached 20,000-30,000, at which point it became unstable. Mike -Original Message- From: Gilbert Midonnet [mailto:[EMAIL PROTECTED] Sent: Tuesday, April 11, 2006 10:19 AM To: CF-Talk Subject: RE: maximum number of files in directory Thanks all, I was pretty sure that 10-20,000 was way too much but not certain how best to organize it. I'm still thinking it through. Thanks about the point regarding category. (Category would not be a good choice, since someone may choose the wrong category first, then update the product.) Since each item will have at least two accompanying photos I'll probably do something along the line you suggest but probably bring it down to 100 or 250 instead of a thousand. Thanks Gil -Original Message- From: Claude Schneegans [mailto:[EMAIL PROTECTED] Sent: Tuesday, April 11, 2006 9:58 AM To: CF-Talk Subject: Re: maximum number of files in directory or maybe categories (dresses, denim, jewelry) (the advantage of this would be fewer directories than designers) You could also use some better balanced subdivision. For instance, if you have some numerical Id nb, use the three last digits as the file name, and the rest for the directory. This way, when the directory has 1000 files, a new one is created, etc. Use something that cannot be changed in your database, so that you do not have to move the files when data is changed. Category would not be a good choice, since someone may choose the wrong category first, then update the product. -- ___ REUSE CODE! Use custom tags; See http://www.contentbox.com/claude/customtags/tagstore.cfm (Please send any spam to this address: [EMAIL PROTECTED]) Thanks. ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:237422 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
RE: maximum number of files in directory
I'm creating a database for a store. It will probably have several thousand items (but will probably never come close to 10,000). The pictures of the items will be stored in a file directory (not as a blob in a database). What is the maximum amount of files that can be stored in a directory and still have an efficient response time. Each item will have at least two graphics if not three or four #itemID#-th #itemID# It is quite possible that there could be 15-20,000 images in this directory. Is that too many? Yes, I would strongly recommend using some sort of storage hierarchy. You are very likely to see performance degrade with that many files in a single directory. The specific number of files that'll cause a problem will vary, of course, depending on your hardware and OS, but I've run into this problem several times, and would recommend that you plan for the worst. Dave Watts, CTO, Fig Leaf Software http://www.figleaf.com/ Fig Leaf Software provides the highest caliber vendor-authorized instruction at our training centers in Washington DC, Atlanta, Chicago, Baltimore, Northern Virginia, or on-site at your location. Visit http://training.figleaf.com/ for more information! ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:237423 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: maximum number of files in directory
I often use the itemID to break down the folders, since these seldom change. Since it is the computer and not humans that do the sorting, the rules can be simple for the computer, even though weird for people. The last 3 digits of the zero padded id has worked well for me. ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:237425 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: maximum number of files in directory
A customer of mine had a Windows 2K server on a cheesy little box. It had about 15,000 images in one folder and was still serving them up without much if any problem. However, I made the mistake of trying to open the folder in Explorer one day and the machine laughed maniacally then went unconscious. After the reboot, I used the command line to move some things about and reorganize, changed the code in the application and it's continued to work very well to this day with that load spread over about 20 directories arranged by product line and category. A couple of the categories still have folders with 600-700 images in them, but nothing more than that. --Ferg Mike Klostermeyer wrote: I've served images from a Windows 2000 Server that was fine until it reached 20,000-30,000, at which point it became unstable. Mike -Original Message- From: Gilbert Midonnet [mailto:[EMAIL PROTECTED] Sent: Tuesday, April 11, 2006 10:19 AM To: CF-Talk Subject: RE: maximum number of files in directory ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:237426 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: maximum number of files in directory
On 4/11/06, Gilbert Midonnet [EMAIL PROTECTED] wrote: I'm creating a database for a store. It will probably have several thousand items (but will probably never come close to 10,000). The pictures of the items will be stored in a file directory (not as a blob in a database). What is the maximum amount of files that can be stored in a directory and still have an efficient response time. Efficient response time is very much a nebulous concept -- hardware, load, and user-dependent. As far as maximum amounts go, that is *filesystem* dependent. Some MS specifics are here (eg 4+ billion files for NTFS) http://www.microsoft.com/technet/prodtechnol/winxppro/reskit/c13621675.mspx Linux specifics vary all over the map, since there are so many filesystems, some designed for closer to petabyte storage arrays. It would be so much simpler to have all the graphics in one directory. Will I be causing problems later on by doing that? The smartest thing to do would be to abstract out the storage details so you can change the implementation when you need to. You want a function like PathToItem(itemId) that gives you back the location of the file. But you want the implementation to be a black box to the rest of the app. That way you can start with the simplest possible approach and hard-code a path function PathToItem(itemId) return '/path/to/my/file pathToItem(8202) - c:/wwwroot/myapp/images Then later decide that you need to partition out numerically by the item id into groups of 1000 as someone suggested in this thread function PathToItem(itemId) return '/rootpath/' (itemId DIV 1000) '/' itemId pathToItem(8202) - c:/wwwroot/myapp/images/8/8202 Then later you find that breaking the directories by month/year of entry makes more sense function PathToItem(itemId) return '/rootpath/' Year(createddate) '/' Month(createddate) itemId pathToItem(8202) - c:/wwwroot/myapp/images/2006/04/8202 etc. If you use this to create the path when you write the image and when you read it, you're set. -- John Paul Ashenfelter CTO/Transitionpoint (blog) http://www.ashenfelter.com (email) [EMAIL PROTECTED] ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:237430 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
RE: maximum number of files in directory
This begs another question that I haven't run into a problem with yet but makes me feel uneasy. What about folders within a folder? I have a folder where I create a new folder for every user on the site and then anything they upload goes into their folder. This is working well so far as we only have around 1000 users. We're using a Windows 2003 Server. As it grows am I going to hit the same kind of limitations or does windows handle the folders well enough that it won't be a problem? Just curious because I've never dealt with the issue before when it comes to folder. Definitely had some headaches with the number of files (especially images) in a folder. Anyone have experiences or thoughts? John Burns Certified Advanced ColdFusion MX Developer Wyle Laboratories, Inc. | Web Developer -Original Message- From: Ken Ferguson [mailto:[EMAIL PROTECTED] Sent: Tuesday, April 11, 2006 12:01 PM To: CF-Talk Subject: Re: maximum number of files in directory A customer of mine had a Windows 2K server on a cheesy little box. It had about 15,000 images in one folder and was still serving them up without much if any problem. However, I made the mistake of trying to open the folder in Explorer one day and the machine laughed maniacally then went unconscious. After the reboot, I used the command line to move some things about and reorganize, changed the code in the application and it's continued to work very well to this day with that load spread over about 20 directories arranged by product line and category. A couple of the categories still have folders with 600-700 images in them, but nothing more than that. --Ferg Mike Klostermeyer wrote: I've served images from a Windows 2000 Server that was fine until it reached 20,000-30,000, at which point it became unstable. Mike -Original Message- From: Gilbert Midonnet [mailto:[EMAIL PROTECTED] Sent: Tuesday, April 11, 2006 10:19 AM To: CF-Talk Subject: RE: maximum number of files in directory ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:237431 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: maximum number of files in directory
I'll probably do something along the line you suggest but probably bring it down to 100 or 250 instead of a thousand. Time to use the MOD operator then ;-) -- ___ REUSE CODE! Use custom tags; See http://www.contentbox.com/claude/customtags/tagstore.cfm (Please send any spam to this address: [EMAIL PROTECTED]) Thanks. ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:237438 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: maximum number of files in directory
Gilbert Midonnet wrote: I'm creating a database for a store. It will probably have several thousand items (but will probably never come close to 10,000). The pictures of the items will be stored in a file directory (not as a blob in a database). What is the maximum amount of files that can be stored in a directory and still have an efficient response time. Which filesystem? Jochem ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:237445 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
RE: maximum number of files in directory
It's a shared server space. Will probably have to move to a dedicated server -- at least for the client's sake I hope so. Its on an NT 2003 server. -Original Message- From: Jochem van Dieten [mailto:[EMAIL PROTECTED] Sent: Tuesday, April 11, 2006 2:18 PM To: CF-Talk Subject: Re: maximum number of files in directory Gilbert Midonnet wrote: I'm creating a database for a store. It will probably have several thousand items (but will probably never come close to 10,000). The pictures of the items will be stored in a file directory (not as a blob in a database). What is the maximum amount of files that can be stored in a directory and still have an efficient response time. Which filesystem? Jochem ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:237450 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
