Hi, @ted, It seems not only affect the table ttsi_order:ttsi_order_process, some other table's region also flushed and compact
2015-05-20 11:04 GMT+08:00 Louis Hust <[email protected]>: > Hi, ted > > hbase(main):001:0> describe 'ttsi_order:ttsi_order_process' > DESCRIPTION > ENABLED > 'ttsi_order:ttsi_order_process', {NAME => 'orderinfo', > DATA_BLOCK_ENCODING => 'NO true > NE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', > COMPRESSION > => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => > 'FALSE', > BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} > 1 row(s) in 1.0260 seconds > > > 2015-05-20 10:59 GMT+08:00 Ted Yu <[email protected]>: > >> How many column families does your table have ? >> >> Can you show us the output of " describe 'ttsi_order:ttsi_order_process' >> " ? >> >> Thanks >> >> On Tue, May 19, 2015 at 4:56 AM, Louis Hust <[email protected]> wrote: >> >> > Hi, all >> > >> > My cluster is cdh5.2.0 with hbase 0.98.6, and recently some region >> server >> > load is very high, >> > about 50, and the CPU usage is about 2400%(with 24 core). >> > >> > I check the log in region server, and found frequent memstore flush and >> > compactions, is this the reason for the high load? >> > >> > {log} >> > 2015-05-19 08:40:21,749 INFO [MemStoreFlusher.0] regionserver.HRegion: >> > Started memstore flush for >> > >> > >> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093., >> > current region memstore size 1.1 M >> > 2015-05-19 08:40:21,917 INFO [MemStoreFlusher.0] >> > regionserver.DefaultStoreFlusher: Flushed, sequenceid=11905428, >> memsize=1.1 >> > M, hasBloomFilter=true, into tmp file >> > >> > >> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82 >> > 2015-05-19 08:40:21,934 DEBUG [MemStoreFlusher.0] >> > regionserver.HRegionFileSystem: Committing store file >> > >> > >> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82 >> > as >> > >> > >> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82 >> > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] regionserver.HStore: >> > Added >> > >> > >> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82, >> > entries=1441, sequenceid=11905428, filesize=847.8 K >> > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] regionserver.HRegion: >> > Finished memstore flush of ~1.1 M/1126904, currentsize=0/0 for region >> > >> > >> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093. >> > in 192ms, sequenceid=11905428, compaction requested=true >> > 2015-05-19 08:40:21,941 DEBUG [MemStoreFlusher.0] >> > regionserver.CompactSplitThread: Small Compaction requested: system; >> > Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, >> > merge_queue=0 >> > 2015-05-19 08:40:21,941 DEBUG >> > [regionserver60020-smallCompactions-1431261234450] >> > compactions.RatioBasedCompactionPolicy: Selecting compaction from 4 >> store >> > files, 0 compacting, 4 eligible, 30 blocking >> > 2015-05-19 08:40:21,941 DEBUG >> > [regionserver60020-smallCompactions-1431261234450] >> > compactions.ExploringCompactionPolicy: Exploring compaction algorithm >> has >> > selected 3 files of size 36749798 starting at candidate #1 after >> > considering 3 permutations with 1 in ratio >> > 2015-05-19 08:40:21,941 DEBUG >> > [regionserver60020-smallCompactions-1431261234450] regionserver.HStore: >> > 378c4f2ebc93770ef6cbf956de60d093 - orderinfo: Initiating minor >> compaction >> > 2015-05-19 08:40:21,941 INFO >> > [regionserver60020-smallCompactions-1431261234450] >> regionserver.HRegion: >> > Starting compaction on orderinfo in region >> > >> > >> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093. >> > 2015-05-19 08:40:21,942 INFO >> > [regionserver60020-smallCompactions-1431261234450] regionserver.HStore: >> > Starting compaction of 3 file(s) in orderinfo of >> > >> > >> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093. >> > into >> > >> > >> tmpdir=hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp, >> > totalSize=35.0 M >> > 2015-05-19 08:40:21,942 DEBUG >> > [regionserver60020-smallCompactions-1431261234450] >> compactions.Compactor: >> > Compacting >> > >> > >> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/a89297d7bbc34db7aa9f23e42789e2bd, >> > keycount=22628, bloomtype=ROW, size=33.8 M, encoding=NONE, >> seqNum=11902843 >> > 2015-05-19 08:40:21,942 DEBUG >> > [regionserver60020-smallCompactions-1431261234450] >> compactions.Compactor: >> > Compacting >> > >> > >> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/d1f8531b00fa4f77ad901001ebe39644, >> > keycount=512, bloomtype=ROW, size=468.8 K, encoding=NONE, >> seqNum=11903863 >> > 2015-05-19 08:40:21,942 DEBUG >> > [regionserver60020-smallCompactions-1431261234450] >> compactions.Compactor: >> > Compacting >> > >> > >> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82, >> > keycount=597, bloomtype=ROW, size=847.8 K, encoding=NONE, >> seqNum=11905428 >> > {/log} >> > >> > Why the flush so small memstore which just 1.1 M, and many other flush >> > even small than 1M。 >> > >> > >> > My configuration: >> > >> > RegionServer: MaxHeap: 64G >> > hbase.hregion.memstore.flush.size = 128M >> > hbase.regionserver.global.memstore.upperLimit=0.4 >> > hbase.regionserver.global.memstore.lowerLimit=0.38 >> > hbase.hregion.memstore.block.multiplier=4 >> > hbase.hregion.memstore.mslab.enabled=true >> > hbase.hregion.max.filesize = 10G >> > hbase.hstore.compaction.max=10 >> > hbase.hstore.compactionThreshold=3 >> > hbase.hstore.compaction.kv.max=10 >> > >> > And the region server has 223 regions. >> > >> > >
