Hi, In our system, we plan to upload data into Hadoop from external sources and use it later on for analysis tasks. The interface to the external repositories allows us to fetch pieces of data in chunks. E.g. get n records at a time. Records are relatively small, though the overall amount of data is assumed to be large. For each repository, we fetch pieces of data in a serial manner. Number of repositories is small (few of them).
My first step is to put the data in plain files in HDFS. My question is what is the optimized file sizes to use. Many small files (to the extent of each record in a file) ? - guess not. Few huge files each holding all data of same type ? Or maybe put each chunk we get in a separate file, and close it right after a chunk was uploaded ? How would HFDS perform best, with few large files or more smaller files ? As I wrote we plan to run MapReduce jobs over the data in the files in order to organize the data and analyze it. Thanks for any help, Naama -- oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo "If you want your children to be intelligent, read them fairy tales. If you want them to be more intelligent, read them more fairy tales." (Albert Einstein)
