On Mon, 10 Mar 2008, Naama Kraus wrote:
Hi,
In our system, we plan to upload data into Hadoop from external sources and
use it later on for analysis tasks. The interface to the external
repositories allows us to fetch pieces of data in chunks. E.g. get n records
at a time. Records are relatively small, though the overall amount of data
is assumed to be large. For each repository, we fetch pieces of data in a
serial manner. Number of repositories is small (few of them).
My first step is to put the data in plain files in HDFS. My question is what
is the optimized file sizes to use. Many small files (to the extent of each
record in a file) ? - guess not. Few huge files each holding all data of
same type ? Or maybe put each chunk we get in a separate file, and close it
right after a chunk was uploaded ?
I think it should be more based on the size of the data you want to
process in a map which I think here is the chunk size, no?
Larger the file less the replicas and hence more the network transfers in
case of more maps. In case of smaller file size the NN will be bottleneck
but you will end up having more replicas for each map task and hence more locality.
Amar
How would HFDS perform best, with few large files or more smaller files ? As
I wrote we plan to run MapReduce jobs over the data in the files in order to
organize the data and analyze it.
Thanks for any help,
Naama
