I ran the put command on 3 of the nodes simultaneously to copy files
that were local on those machines into the hdfs.
Brian Bockelman wrote:
What'd you do for the tests? Was it a single stream or a multiple
stream test?
Brian
On Jun 12, 2009, at 6:48 AM, Scott wrote:
So is ~ 1GB/minute transfer rate a reasonable performance benchmark?
Our test cluster consists of 4 quad core xeon machines with 2
non-raided drives each. My initial tests show a transfer rate of
around 1GB/minute, and that was slower that I expected it to be.
Thanks,
Scott
Brian Bockelman wrote:
Hey Sugandha,
Transfer rates depend on the quality/quantity of your hardware and
the quality of your client disk that is generating the data. I
usually say that you should expect near-hardware-bottleneck speeds
for an otherwise idle cluster.
There should be no "make it fast" required (though you should
reviewi the logs for errors if it's going slow). I would expect a
5GB file to take around 3-5 minutes to write on our cluster, but
it's a well-tuned and operational cluster.
As Todd (I think) mentioned before, we can't help any when you say
"I want to make it faster". You need to provide diagnostic
information - logs, Ganglia plots, stack traces, something - that
folks can look at.
Brian
On Jun 10, 2009, at 2:25 AM, Sugandha Naolekar wrote:
But if I want to make it fast, then??? I want to place the data in
HDFS and
reoplicate it in fraction of seconds. Can that be possible. and How?
On Wed, Jun 10, 2009 at 2:47 PM, kartik saxena
<kartik....@gmail.com> wrote:
I would suppose about 2-3 hours. It took me some 2 days to load a
160 Gb
file.
Secura
On Wed, Jun 10, 2009 at 11:56 AM, Sugandha Naolekar
<sugandha....@gmail.com>wrote:It
Hello!
If I try to transfer a 5GB VDI file from a remote host(not a part of
hadoop
cluster) into HDFS, and get it back, how much time is it supposed to
take?
No map-reduce involved. Simply Writing files in and out from HDFS
through
a
simple code of java (usage of API's).
--
Regards!
Sugandha
--
Regards!
Sugandha
