> Currently, to store bill records, we can achieve about 30K record/second. >
Can you use bulk load? See http://hbase.apache.org/bulk-loads.html > However, the query performance is quite poor. We can only achieve > about 600~700 month_report/second. That is, each region server can > only serve query for about 100 row/second. Block cache hit ratio is > about 20%. > This is random accesses? Why random accesses and not scans? > Do you have any advice on how to improve the query performance? > See above cited performance section from website book. > Below is some metrics info reported by region server: > 2011-04-26T10:56:12 hbase.regionserver: > RegionServer=regionserver50820, blockCacheCount=40969, > blockCacheEvictedCount=216359, blockCacheFree=671152504, > blockCacheHitCachingRatio=20, blockCacheHitCount=67936, > blockCacheHitRatio=20, blockCacheMissCount=257675, > blockCacheSize=2743351688, compactionQueueSize=0, > compactionSize_avg_time=0, compactionSize_num_ops=7, > compactionTime_avg_time=0, compactionTime_num_ops=7, flushQueueSize=0, > flushSize_avg_time=0, flushSize_num_ops=0, flushTime_avg_time=0, > flushTime_num_ops=0, fsReadLatency_avg_time=46, > fsReadLatency_num_ops=257905, fsSyncLatency_avg_time=0, > fsSyncLatency_num_ops=1726, fsWriteLatency_avg_time=0, > fsWriteLatency_num_ops=0, memstoreSizeMB=0, regions=169, > requests=82.1, storefileIndexSizeMB=188, storefiles=343, stores=169 > 2011-04-26T10:56:22 hbase.regionserver: > RegionServer=regionserver50820, blockCacheCount=42500, > blockCacheEvictedCount=216359, blockCacheFree=569659040, > blockCacheHitCachingRatio=20, blockCacheHitCount=68418, > blockCacheHitRatio=20, blockCacheMissCount=259206, > blockCacheSize=2844845152, compactionQueueSize=0, > compactionSize_avg_time=0, compactionSize_num_ops=7, > compactionTime_avg_time=0, compactionTime_num_ops=7, flushQueueSize=0, > flushSize_avg_time=0, flushSize_num_ops=0, flushTime_avg_time=0, > flushTime_num_ops=0, fsReadLatency_avg_time=44, > fsReadLatency_num_ops=259547, fsSyncLatency_avg_time=0, > fsSyncLatency_num_ops=1736, fsWriteLatency_avg_time=0, > fsWriteLatency_num_ops=0, memstoreSizeMB=0, regions=169, > requests=92.2, storefileIndexSizeMB=188, storefiles=343, stores=169 > 2011-04-26T10:56:32 hbase.regionserver: > RegionServer=regionserver50820, blockCacheCount=39238, > blockCacheEvictedCount=221509, blockCacheFree=785944072, > blockCacheHitCachingRatio=20, blockCacheHitCount=69043, > blockCacheHitRatio=20, blockCacheMissCount=261095, > blockCacheSize=2628560120, compactionQueueSize=0, > compactionSize_avg_time=0, compactionSize_num_ops=7, > compactionTime_avg_time=0, compactionTime_num_ops=7, flushQueueSize=0, > flushSize_avg_time=0, flushSize_num_ops=0, flushTime_avg_time=0, > flushTime_num_ops=0, fsReadLatency_avg_time=39, > fsReadLatency_num_ops=261070, fsSyncLatency_avg_time=0, > fsSyncLatency_num_ops=1746, fsWriteLatency_avg_time=0, > fsWriteLatency_num_ops=0, memstoreSizeMB=0, regions=169, > requests=128.77777, storefileIndexSizeMB=188, storefiles=343, > stores=169 > This is hard to read but I don't see anything obnoxious. > > And we also tried to disable block cache, it seems the performance is > even a little bit better. And it we use the configuration 6 DN servers > + 3 RS servers, we can get better throughput at about 1000 > month_report/second. I am confused. Can any one explain the reason? > Sounds like you are doing all random reads? Do you have to? St.Ack
