I test upload and download files(bunch of files , from 1M~ N*100M) with the
FSShell . and There are two problems:
1. write is much faster than read.
with single DataNode and single client , average UPLOAD speed is 6m/s ,
this is the speed that my network has( i test the real network performance
with ftp protocol ,aslo 6m/s ). but average DOWNLOAD speed is only 3~4 m/s.
Is this normal ? Why this happened ? as i know , the write and read
process is almost same , connect to the Master , get info , then communicate
with DataNode. i think CRC won't take so much time .
2. Cluster is slower than single
with 3 DataNode , set 3 replication , upload and download again . (Is
replication upload to other datanodes immediately , or just let
SecondNameNode to do this ?)
this time , in same network , the average UPLOAD speed is 1~2 m/s , and
DOWNLOAD speed is 700~900 KB/s .
so , both single and cluster , Download ~= Upload * 0.5 .
Isn't that crazy ? the cluster is eating performance.
any body could told me about these problems , this real confused me .
And , a question :
what's those in hadoop may affect performance .
"blocks seek , CRC , Network latency" ?
--
View this message in context:
http://www.nabble.com/About-read-write-performance-tf4023552.html#a11428257
Sent from the Hadoop Users mailing list archive at Nabble.com.
