On Aug 17, 2010, at 5:44 PM, Stuart Smith wrote: > I started to break the files into subdirectories out of habit (from working > on ntfs/etc), but it occurred to me that maybe (from a performance > perspective), it doesn't really matter on hdfs. > > Does it? Is there some recommended limit on the number of files to store in > one directory on hdfs? I'm thinking thousands to millions, so we're not > talking about INT_MAX or anything, but a lot. > > Or is it only limited by my sanity :) ?
We have a directory with several thousand files in it. It is always a pain when we hit it because the client heap size needs to be increased to do anything in it: directory listings, web uis, distcp, etc, etc, etc. Doing any sort of manipulation in that dir is also slower. My recommendation: don't do it. Directories, AFAIK, are relatively cheap resource wise vs. lots of files in one. [Hopefully these files are large. Otherwise they should be joined together... if not, you're going to take a performance hit processing them *and* storing them...]
