any ideas?
2015-05-19 19:56 GMT+08:00 Louis Hust <[email protected]>:
> Hi, all
>
> My cluster is cdh5.2.0 with hbase 0.98.6, and recently some region server
> load is very high,
> about 50, and the CPU usage is about 2400%(with 24 core).
>
> I check the log in region server, and found frequent memstore flush and
> compactions, is this the reason for the high load?
>
> {log}
> 2015-05-19 08:40:21,749 INFO [MemStoreFlusher.0] regionserver.HRegion:
> Started memstore flush for
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.,
> current region memstore size 1.1 M
> 2015-05-19 08:40:21,917 INFO [MemStoreFlusher.0]
> regionserver.DefaultStoreFlusher: Flushed, sequenceid=11905428, memsize=1.1
> M, hasBloomFilter=true, into tmp file
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
> 2015-05-19 08:40:21,934 DEBUG [MemStoreFlusher.0]
> regionserver.HRegionFileSystem: Committing store file
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
> as
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82
> 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] regionserver.HStore:
> Added
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
> entries=1441, sequenceid=11905428, filesize=847.8 K
> 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] regionserver.HRegion:
> Finished memstore flush of ~1.1 M/1126904, currentsize=0/0 for region
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> in 192ms, sequenceid=11905428, compaction requested=true
> 2015-05-19 08:40:21,941 DEBUG [MemStoreFlusher.0]
> regionserver.CompactSplitThread: Small Compaction requested: system;
> Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0,
> merge_queue=0
> 2015-05-19 08:40:21,941 DEBUG
> [regionserver60020-smallCompactions-1431261234450]
> compactions.RatioBasedCompactionPolicy: Selecting compaction from 4 store
> files, 0 compacting, 4 eligible, 30 blocking
> 2015-05-19 08:40:21,941 DEBUG
> [regionserver60020-smallCompactions-1431261234450]
> compactions.ExploringCompactionPolicy: Exploring compaction algorithm has
> selected 3 files of size 36749798 starting at candidate #1 after
> considering 3 permutations with 1 in ratio
> 2015-05-19 08:40:21,941 DEBUG
> [regionserver60020-smallCompactions-1431261234450] regionserver.HStore:
> 378c4f2ebc93770ef6cbf956de60d093 - orderinfo: Initiating minor compaction
> 2015-05-19 08:40:21,941 INFO
> [regionserver60020-smallCompactions-1431261234450] regionserver.HRegion:
> Starting compaction on orderinfo in region
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> 2015-05-19 08:40:21,942 INFO
> [regionserver60020-smallCompactions-1431261234450] regionserver.HStore:
> Starting compaction of 3 file(s) in orderinfo of
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> into
> tmpdir=hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp,
> totalSize=35.0 M
> 2015-05-19 08:40:21,942 DEBUG
> [regionserver60020-smallCompactions-1431261234450] compactions.Compactor:
> Compacting
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/a89297d7bbc34db7aa9f23e42789e2bd,
> keycount=22628, bloomtype=ROW, size=33.8 M, encoding=NONE, seqNum=11902843
> 2015-05-19 08:40:21,942 DEBUG
> [regionserver60020-smallCompactions-1431261234450] compactions.Compactor:
> Compacting
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/d1f8531b00fa4f77ad901001ebe39644,
> keycount=512, bloomtype=ROW, size=468.8 K, encoding=NONE, seqNum=11903863
> 2015-05-19 08:40:21,942 DEBUG
> [regionserver60020-smallCompactions-1431261234450] compactions.Compactor:
> Compacting
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
> keycount=597, bloomtype=ROW, size=847.8 K, encoding=NONE, seqNum=11905428
> {/log}
>
> Why the flush so small memstore which just 1.1 M, and many other flush
> even small than 1M。
>
>
> My configuration:
>
> RegionServer: MaxHeap: 64G
> hbase.hregion.memstore.flush.size = 128M
> hbase.regionserver.global.memstore.upperLimit=0.4
> hbase.regionserver.global.memstore.lowerLimit=0.38
> hbase.hregion.memstore.block.multiplier=4
> hbase.hregion.memstore.mslab.enabled=true
> hbase.hregion.max.filesize = 10G
> hbase.hstore.compaction.max=10
> hbase.hstore.compactionThreshold=3
> hbase.hstore.compaction.kv.max=10
>
> And the region server has 223 regions.
>
>
>
>
