hbase logs show the performance of compact isnt good. any suggstion to find out the reason?

[Lu Dillon]

Hi All,


I'm a newbie against Hbase. When I use YCSB to load data into Hbase, the 
compaction in two cluster looks different.

DEV: 6 nodes. A and B are master and standby for HDFS and Hbase. C/D/E/F are 
the HDFS datanode and regionservers.

PROD: 8 nodes. A and B are master and standby for HDFS and Hbase. C/D/E/F/G/H 
are the HDFS datanode and regionservers.

Network is 1000mbs.

Hadoop: 2.7.3

Hbase: 1.2.6

JDK:1.8

Zookeep: install on C/D/E



In DEV, I notice the compaction 4G files are about 30 seconds; however, it 
tooks about 30 seconds to process 400MB files in PROD.

For example, this is logs from one node of DEV:

2018-08-10 16:30:47,535 INFO  
[regionserver/node103/172.28.200.103:16020-shortCompactions-1533803656579] 
regionserver.HStore: Completed compaction of 6 file(s) in family of 
usertable_nosplit,,1533878791857.be420543a609c32475386855dee93a82. into 
abe3a5c0dbbd4708ae2523f9b4a53ad8(size=1.4 G), total size for store is 4.0 G. 
This selection was in queue for 0sec, and took 18sec to execute.


This is from PROD:

2018-08-17 23:29:28,751 INFO  
[regionserver/YJB-HADOOP-74-23/192.168.74.23:16020-shortCompactions-1534519629607]
 regionserver.HStore: Completed compaction of 3 (all) file(s) in family of 
usertable_nosplit,,1534518587365.95595277af1c56260b85de113c9687f1. into 
934669bf6a8a407aa6ee385439c4a7b5(size=430.8 M), total size for store is 430.8 
M. This selection was in queue for 0sec, and took 30sec to execute.


I thought this might be a problem of GC. So I change to use CG1 with this 
article "https://product.hubspot.com/blog/g1gc-tuning-your-hbase-cluster"; . The 
performance improve a little, not to good.


Is there any suggestion to find out what causes the problem and improve the 
performance?


BTW: I'm not sure what kind information can help so I attach my hbase-site.xml 
and hbase-env.sh first.

BTW2: to use GC1 against regionserver, I create a speraeted hbase-env.sh for 
regionserver.


Thanks,

Dillon

hbase-env(regionserver).sh
Description: hbase-env(regionserver).sh

hbase-env(master).sh
Description: hbase-env(master).sh

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
/**
 *
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
-->
<configuration>
        <property>
            <name>hbase.rootdir</name>
            <value>hdfs://HaHdfs/apps/hbase/data</value>
        </property>

        <property>
            <name>hbase.cluster.distributed</name>
            <value>true</value>
        </property>

        <property>
            <name>hbase.master</name>
            <value>16000</value>
        </property>

        <property>
            <name>hbase.tmp.dir</name>
            <value>/opt/app/hbase/tmp</value>
        </property>

        <property>
            <name>hbase.zookeeper.quorum</name>
            <value>YJB-HADOOP-74-23,YJB-HADOOP-74-24,YJB-HADOOP-74-25</value>
        </property>

        <property>
            <name>hbase.zookeeper.property.dataDir</name>
            <value>/opt/app/zookeeper-3.4.9/data</value>
        </property>

        <property>
            <name>hbase.zookeeper.property.clientPort</name>
            <value>2181</value>
        </property>

        <property>
            <name>zookeeper.session.timeout</name>
            <value>120000</value>
        </property>

        <property>
            <name>hbase.regionserver.restart.on.zk.expire</name>
            <value>true</value>
        </property>


<property>
<name>hbase.ipc.server.callqueue.handler.factor</name>
<value>0.5</value>
</property>

<property>
<name>hbase.regionserver.region.split.policy</name>
<value>org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy</value>
</property>

<property>
<name>hbase.regionserver.handler.count</name>
<value>30</value>
</property>
<property>
<name>hbase.hregion.memstore.flush.size</name>
<value>268435456</value>
</property>

<property>
<name>hbase.hstore.compaction.max</name>
<value>20</value>
</property>

<property>
<name>hbase.hstore.compaction.min</name>
<value>3</value>
</property>

<property>
<name>hbase.hstore.compaction.min.size</name>
<value>268435456</value>
</property>
</configuration>