Please use a new subject when starting a new topic. jafarim wrote:
Sorry if being off topic, but we experienced a very low bandwidth with hadoop while copying files to/from the cluster (some 1/100 comparing to plain samba share). The bandwidth did not improve at all by adding nodes to the cluster. At that time I thought that hadoop is not supposed to be used for this purpose and did not use it for my project. I am just curious how much scalable hadoop is and how bandwidth should grow as nodes are added to the cluster.
It's not clear to me what you tried. Are you running HDFS? On how large of a cluster? What version of Hadoop? What operating system? How were you copying files to/from the cluster?
The 'bin/hadoop distcp' command should scale to consume available network bandwidth and disk i/o.
Doug
