Do you check your cluster has flushed many little hfiles? How much memory is used ?
It seems that region server has too many active regions You can use this command check it: grep "Finished memstore flush" *regionserver* -c grep "global heap pressure" *regionserver* -c -----邮件原件----- 发件人: Xu-Feng Mao [mailto:[email protected]] 发送时间: 2011年9月6日 16:16 收件人: [email protected]; [email protected] 主题: big compaction queue size Hi, We're running a 33-regionserver hbase cluster on top of cdh3u0 suites. On average, we have 2400 regions hosted on each regionserver. (hbase.hregion.max.filesize is 1.5GB, and we have value size up to 4MB per object). I check the log of regionserver, it seems like the compaction queue size is about 1700, and every the compaction action takes about 1 minute, and more over, most of the compaction are triggered to a major one. My question are, 1. Would this cause the performance degradation? It seems like "GET" action in the interval that two minutes before/after the compaction takes much longer time than usual. I thought the compaction is a asynchronous operation. 2. Any issue would cause long-term compaction? 3. It seems like HBASE-1476 is going to implement multi-threaded compaction, I guess it would help to reduce the size of compaction queue. Thanks and regards, Mao Xu-Feng
