On Wed, Sep 26, 2012 at 1:13 AM, Yusup Ashrap <[email protected]> wrote: > hi Stack, thanks for reply. > *hbase version* is 0.90.2.
This is an extremely old version. we use ganglia to monitor our cluster.write/read > is normal/equally distributed all day long. 1k write , 4k read. > it's kinda impossible to upgrade at current moment , and we dont have > extra machines to cope with this situation.(migrate=>upgrade) More machines at least would get you breathing room. > those metrics are region server metrics.high load time is about 10:30am ~ > 11:00 am (I'm from China) today. > all servers in the cluster are dedicated for hbase storage and we dont have > any other jobs or programs running on those servers. > > I found that I have one major compaction going on that period,is it the > main reason of high load situation? Probably. And compactions in 0.90.x are not too smart compared to newer versions. > and why does this " Block cache LRU eviction " happen so frequently? > #org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction > started; Attempting to free 634.89 MB of total=5.27 GB > That's 'normal'. Its a cache. Its evicting least recently used blocks to make room for more recently read blocks. > *10:30's major compaction log*: > 2012-09-26 10:31:36,581 DEBUG org.apache.hadoop.hbase.regionserver.Store: > Major compaction triggered on store data; time since last major compaction > 81842121ms > 2012-09-26 10:31:36,581 DEBUG > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction > requested for > Look at your ganglia. Do you see your iowait go up and your read latency too at this time when the major compaction runs? St.Ack
