How many column families does your table have ? Can you show us the output of " describe 'ttsi_order:ttsi_order_process' " ?
Thanks On Tue, May 19, 2015 at 4:56 AM, Louis Hust <[email protected]> wrote: > Hi, all > > My cluster is cdh5.2.0 with hbase 0.98.6, and recently some region server > load is very high, > about 50, and the CPU usage is about 2400%(with 24 core). > > I check the log in region server, and found frequent memstore flush and > compactions, is this the reason for the high load? > > {log} > 2015-05-19 08:40:21,749 INFO [MemStoreFlusher.0] regionserver.HRegion: > Started memstore flush for > > ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093., > current region memstore size 1.1 M > 2015-05-19 08:40:21,917 INFO [MemStoreFlusher.0] > regionserver.DefaultStoreFlusher: Flushed, sequenceid=11905428, memsize=1.1 > M, hasBloomFilter=true, into tmp file > > hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82 > 2015-05-19 08:40:21,934 DEBUG [MemStoreFlusher.0] > regionserver.HRegionFileSystem: Committing store file > > hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82 > as > > hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82 > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] regionserver.HStore: > Added > > hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82, > entries=1441, sequenceid=11905428, filesize=847.8 K > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] regionserver.HRegion: > Finished memstore flush of ~1.1 M/1126904, currentsize=0/0 for region > > ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093. > in 192ms, sequenceid=11905428, compaction requested=true > 2015-05-19 08:40:21,941 DEBUG [MemStoreFlusher.0] > regionserver.CompactSplitThread: Small Compaction requested: system; > Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, > merge_queue=0 > 2015-05-19 08:40:21,941 DEBUG > [regionserver60020-smallCompactions-1431261234450] > compactions.RatioBasedCompactionPolicy: Selecting compaction from 4 store > files, 0 compacting, 4 eligible, 30 blocking > 2015-05-19 08:40:21,941 DEBUG > [regionserver60020-smallCompactions-1431261234450] > compactions.ExploringCompactionPolicy: Exploring compaction algorithm has > selected 3 files of size 36749798 starting at candidate #1 after > considering 3 permutations with 1 in ratio > 2015-05-19 08:40:21,941 DEBUG > [regionserver60020-smallCompactions-1431261234450] regionserver.HStore: > 378c4f2ebc93770ef6cbf956de60d093 - orderinfo: Initiating minor compaction > 2015-05-19 08:40:21,941 INFO > [regionserver60020-smallCompactions-1431261234450] regionserver.HRegion: > Starting compaction on orderinfo in region > > ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093. > 2015-05-19 08:40:21,942 INFO > [regionserver60020-smallCompactions-1431261234450] regionserver.HStore: > Starting compaction of 3 file(s) in orderinfo of > > ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093. > into > > tmpdir=hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp, > totalSize=35.0 M > 2015-05-19 08:40:21,942 DEBUG > [regionserver60020-smallCompactions-1431261234450] compactions.Compactor: > Compacting > > hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/a89297d7bbc34db7aa9f23e42789e2bd, > keycount=22628, bloomtype=ROW, size=33.8 M, encoding=NONE, seqNum=11902843 > 2015-05-19 08:40:21,942 DEBUG > [regionserver60020-smallCompactions-1431261234450] compactions.Compactor: > Compacting > > hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/d1f8531b00fa4f77ad901001ebe39644, > keycount=512, bloomtype=ROW, size=468.8 K, encoding=NONE, seqNum=11903863 > 2015-05-19 08:40:21,942 DEBUG > [regionserver60020-smallCompactions-1431261234450] compactions.Compactor: > Compacting > > hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82, > keycount=597, bloomtype=ROW, size=847.8 K, encoding=NONE, seqNum=11905428 > {/log} > > Why the flush so small memstore which just 1.1 M, and many other flush > even small than 1M。 > > > My configuration: > > RegionServer: MaxHeap: 64G > hbase.hregion.memstore.flush.size = 128M > hbase.regionserver.global.memstore.upperLimit=0.4 > hbase.regionserver.global.memstore.lowerLimit=0.38 > hbase.hregion.memstore.block.multiplier=4 > hbase.hregion.memstore.mslab.enabled=true > hbase.hregion.max.filesize = 10G > hbase.hstore.compaction.max=10 > hbase.hstore.compactionThreshold=3 > hbase.hstore.compaction.kv.max=10 > > And the region server has 223 regions. >
