What is the level of concurrency? I find that HDFS gets worse performing with more concurrent read threads.
-ryan 2011/7/30 seven garfee <[email protected]>: > hi,all > I set up a cluster on 4 machine.(1 HMaster,4RegionServer). > Each Machine has a 16G mem, one 2T Sata disk,CentOS 5.3,XFS . > I write 100M data in Hbase,only 1M key/value,so all keys are in only *one* > Store,and only 2 HFiles in that Store.(I used Large Data Set,but random read > performance is bad,so i try 100M) > Then I test random read with existing keys with only 1 client. But I got a > Bad Performance,About >100ms latency per read. > > Why did I get so long latency? > > I dig into the metric log,and found that, > in metrics_rpc.log: > get_avg_time=146, get_num_ops=22 > get_avg_time=94, get_num_ops=127 > get_avg_time=113, get_num_ops=182 > ....... > > and in metrics_hbase.log: > fsReadLatency_avg_time=140, fsReadLatency_num_ops=31 > fsReadLatency_avg_time=93, fsReadLatency_num_ops=137 > fsReadLatency_avg_time=109, fsReadLatency_num_ops=188 > ....... > > Is that say the fsReadLatency causes the random get latency so long? > > But i didn't tuning hdfs almost,except some ips/ports and max.xcievers, > Should I tuning for hdfs?And How? >
