Please also note that the current design only allows you up to a certain number of files given a limited amount of RAM for namenode. The number depends on a various factors (e.g. # of dirs, # of blocks, etc.) but a through analysis found in http://issues.apache.org/jira/browse/HADOOP-1687shows it's around 1M files per 1G of RAM.
Of course, if you could have an infinite amount of RAM, then you could store an infinite number of files. As for the reliabilities of the HDFS, as long as you take care of your namenode alright (e.g. your namenode being the most powerful and reliable machine with a failover machine assigned to it), I find it shouldn't be a much of a problem. I wouldn't use HDFS for any front end, time-critical service, just yet, but I think it's good enough to use for back-end system ( e.g. log storage and analysis) On 9/6/07, Dongsheng Wang <[EMAIL PROTECTED]> wrote: > > > We are looking at using HDFS as a long term storage solution. We want to > use it to stored lots of files. The file could be big and small, they are > images, videos etc... We only write the files once, and may read them many > times. Sounds like it is perfect to use HDFS. > > The concern is that since it's been engineered to support MapReduce there > may be fundamental assumptions that the data being stored by HDFS is > transient in nature. Obviously for our scalable storage solution zero data > loss or corruption is a heavy requirement. > > Is anybody using HDFS as a long term storage solution? Interested in any > info. Thanks > > - ds > > > --------------------------------- > Yahoo! oneSearch: Finally, mobile search that gives answers, not web > links. -- Taeho Kang [tkang.blogspot.com] Software Engineer, NHN Corporation, Korea
