Louis: I don't see -XX:G1HeapRegionSize option in the command line. Please refer to Bryan's comment : http://search-hadoop.com/m/YGbbupEDoKTrDo
Cheers On Tue, May 19, 2015 at 11:50 PM, Louis Hust <[email protected]> wrote: > Hi, ted, > > today i counter the problem again, and I find the metrics of GCCOUNT G1 > YOUNG GENERATION is related to the > CPU LOAD, so this means it's GC problem? > > The options for regionserver like below: > > /home/q/java/default/bin/java -Dproc_regionserver > -XX:OnOutOfMemoryError=kill -9 %p -Xmx8192m -Xms8g -Xmx8g -XX:NewSize=4g > -XX:MaxNewSize=4g -XX:PermSize=1g -XX:MaxPermSize=1g -server > -XX:+DisableExplicitGC -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC > -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly > -XX:CMSInitiatingOccupancyFraction=75 -XX:SoftRefLRUPolicyMSPerMB=0 > -XX:+PrintClassHistogram -XX:+PrintGCDetails -XX:+PrintGCDateStamps > -XX:+PrintHeapAtGC > > -Xloggc:/home/q/hbase/hbase-0.98.6-cdh5.2.0/logs/gc-l-hbase1.dba.cn1-hbase.log > -XX:ParallelGCThreads=10 -Xms64g -Xmx64g -XX:NewSize=4g -XX:MaxNewSize=4g > -XX:PermSize=1g -XX:MaxPermSize=1g -server -XX:+DisableExplicitGC > -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:+UseFastAccessorMethods > -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=75 > -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+PrintClassHistogram -XX:+PrintGCDetails > -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC > > -Xloggc:/home/q/hbase/hbase-0.98.6-cdh5.2.0/logs/gc-l-hbase1.dba.cn1-hbase.log > -XX:ParallelGCThreads=10 -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.authenticate=false > -Dcom.sun.management.jmxremote.port=10102 > -Dhbase.log.dir=/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/../logs > -Dhbase.log.file=hbase-hadoop-regionserver-l-hbase1.dba.cn1.log > -Dhbase.home.dir=/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/.. > -Dhbase.id.str=hadoop -Dhbase.root.logger=INFO,RFA > > -Djava.library.path=/home/q/hadoop/hadoop-2.3.0-cdh5.1.0/lib/native:/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/../lib/native/Linux-amd64-64 > -Dhbase.security.logger=INFO,RFAS > org.apache.hadoop.hbase.regionserver.HRegionServer start > > > Any idea? > > > > 2015-05-20 11:07 GMT+08:00 Louis Hust <[email protected]>: > > > Hi, @ted, > > > > It seems not only affect the table ttsi_order:ttsi_order_process, some > > other table's region also flushed and compact > > > > 2015-05-20 11:04 GMT+08:00 Louis Hust <[email protected]>: > > > >> Hi, ted > >> > >> hbase(main):001:0> describe 'ttsi_order:ttsi_order_process' > >> DESCRIPTION > >> ENABLED > >> 'ttsi_order:ttsi_order_process', {NAME => 'orderinfo', > >> DATA_BLOCK_ENCODING => 'NO true > >> NE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', > >> COMPRESSION > >> => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS > => > >> 'FALSE', > >> BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} > >> 1 row(s) in 1.0260 seconds > >> > >> > >> 2015-05-20 10:59 GMT+08:00 Ted Yu <[email protected]>: > >> > >>> How many column families does your table have ? > >>> > >>> Can you show us the output of " describe > 'ttsi_order:ttsi_order_process' > >>> " ? > >>> > >>> Thanks > >>> > >>> On Tue, May 19, 2015 at 4:56 AM, Louis Hust <[email protected]> > >>> wrote: > >>> > >>> > Hi, all > >>> > > >>> > My cluster is cdh5.2.0 with hbase 0.98.6, and recently some region > >>> server > >>> > load is very high, > >>> > about 50, and the CPU usage is about 2400%(with 24 core). > >>> > > >>> > I check the log in region server, and found frequent memstore flush > and > >>> > compactions, is this the reason for the high load? > >>> > > >>> > {log} > >>> > 2015-05-19 08:40:21,749 INFO [MemStoreFlusher.0] > regionserver.HRegion: > >>> > Started memstore flush for > >>> > > >>> > > >>> > ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093., > >>> > current region memstore size 1.1 M > >>> > 2015-05-19 08:40:21,917 INFO [MemStoreFlusher.0] > >>> > regionserver.DefaultStoreFlusher: Flushed, sequenceid=11905428, > >>> memsize=1.1 > >>> > M, hasBloomFilter=true, into tmp file > >>> > > >>> > > >>> > hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82 > >>> > 2015-05-19 08:40:21,934 DEBUG [MemStoreFlusher.0] > >>> > regionserver.HRegionFileSystem: Committing store file > >>> > > >>> > > >>> > hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82 > >>> > as > >>> > > >>> > > >>> > hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82 > >>> > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] > regionserver.HStore: > >>> > Added > >>> > > >>> > > >>> > hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82, > >>> > entries=1441, sequenceid=11905428, filesize=847.8 K > >>> > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] > regionserver.HRegion: > >>> > Finished memstore flush of ~1.1 M/1126904, currentsize=0/0 for region > >>> > > >>> > > >>> > ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093. > >>> > in 192ms, sequenceid=11905428, compaction requested=true > >>> > 2015-05-19 08:40:21,941 DEBUG [MemStoreFlusher.0] > >>> > regionserver.CompactSplitThread: Small Compaction requested: system; > >>> > Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, > >>> > merge_queue=0 > >>> > 2015-05-19 08:40:21,941 DEBUG > >>> > [regionserver60020-smallCompactions-1431261234450] > >>> > compactions.RatioBasedCompactionPolicy: Selecting compaction from 4 > >>> store > >>> > files, 0 compacting, 4 eligible, 30 blocking > >>> > 2015-05-19 08:40:21,941 DEBUG > >>> > [regionserver60020-smallCompactions-1431261234450] > >>> > compactions.ExploringCompactionPolicy: Exploring compaction algorithm > >>> has > >>> > selected 3 files of size 36749798 starting at candidate #1 after > >>> > considering 3 permutations with 1 in ratio > >>> > 2015-05-19 08:40:21,941 DEBUG > >>> > [regionserver60020-smallCompactions-1431261234450] > regionserver.HStore: > >>> > 378c4f2ebc93770ef6cbf956de60d093 - orderinfo: Initiating minor > >>> compaction > >>> > 2015-05-19 08:40:21,941 INFO > >>> > [regionserver60020-smallCompactions-1431261234450] > >>> regionserver.HRegion: > >>> > Starting compaction on orderinfo in region > >>> > > >>> > > >>> > ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093. > >>> > 2015-05-19 08:40:21,942 INFO > >>> > [regionserver60020-smallCompactions-1431261234450] > >>> regionserver.HStore: > >>> > Starting compaction of 3 file(s) in orderinfo of > >>> > > >>> > > >>> > ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093. > >>> > into > >>> > > >>> > > >>> > tmpdir=hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp, > >>> > totalSize=35.0 M > >>> > 2015-05-19 08:40:21,942 DEBUG > >>> > [regionserver60020-smallCompactions-1431261234450] > >>> compactions.Compactor: > >>> > Compacting > >>> > > >>> > > >>> > hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/a89297d7bbc34db7aa9f23e42789e2bd, > >>> > keycount=22628, bloomtype=ROW, size=33.8 M, encoding=NONE, > >>> seqNum=11902843 > >>> > 2015-05-19 08:40:21,942 DEBUG > >>> > [regionserver60020-smallCompactions-1431261234450] > >>> compactions.Compactor: > >>> > Compacting > >>> > > >>> > > >>> > hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/d1f8531b00fa4f77ad901001ebe39644, > >>> > keycount=512, bloomtype=ROW, size=468.8 K, encoding=NONE, > >>> seqNum=11903863 > >>> > 2015-05-19 08:40:21,942 DEBUG > >>> > [regionserver60020-smallCompactions-1431261234450] > >>> compactions.Compactor: > >>> > Compacting > >>> > > >>> > > >>> > hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82, > >>> > keycount=597, bloomtype=ROW, size=847.8 K, encoding=NONE, > >>> seqNum=11905428 > >>> > {/log} > >>> > > >>> > Why the flush so small memstore which just 1.1 M, and many other > flush > >>> > even small than 1M。 > >>> > > >>> > > >>> > My configuration: > >>> > > >>> > RegionServer: MaxHeap: 64G > >>> > hbase.hregion.memstore.flush.size = 128M > >>> > hbase.regionserver.global.memstore.upperLimit=0.4 > >>> > hbase.regionserver.global.memstore.lowerLimit=0.38 > >>> > hbase.hregion.memstore.block.multiplier=4 > >>> > hbase.hregion.memstore.mslab.enabled=true > >>> > hbase.hregion.max.filesize = 10G > >>> > hbase.hstore.compaction.max=10 > >>> > hbase.hstore.compactionThreshold=3 > >>> > hbase.hstore.compaction.kv.max=10 > >>> > > >>> > And the region server has 223 regions. > >>> > > >>> > >> > >> > > >
