hbase1.3.1 (build hadoop2.7.4) shortcircuit problem
Online HBase services, large requests， hbase version: 1.3.1 hadoop version: 2.7.4 os linux version: centos6.5 hdfs-site.xml conf: dfs.client.read.shortcircuittruehdfs-site.xmldfs.domain.socket.path/var/run/hadoop-hdfs/dn_sockethdfs-site.xml os linux command : netstat -an unix 2 [ ACC ] STREAM LISTENING 10576083 /var/run/hadoop-hdfs/dn_socket problem : /var/run/hadoop-hdfs/dn_socket only one LISTENING， but no CONNECTED
hbase1.3.1 (build hadoop2.7.4) shortcircuit problem
Online HBase services, large requests， hbase version: 1.3.1 hadoop version: 2.7.4 os linux version: centos6.5 hdfs-site.xml conf: dfs.client.read.shortcircuittruehdfs-site.xmldfs.domain.socket.path/var/run/hadoop-hdfs/dn_sockethdfs-site.xml os linux command : netstat -an unix 2 [ ACC ] STREAM LISTENING 10576083 /var/run/hadoop-hdfs/dn_socket problem : /var/run/hadoop-hdfs/dn_socket only one LISTENING， but no CONNECTED
hbase shortcircuit problem
Online HBase services, large requests， hbase version: 1.3.1 hadoop version: 2.7.4 os linux version: centos6.5 hdfs-site.xml conf: dfs.client.read.shortcircuittruehdfs-site.xmldfs.domain.socket.path/var/run/hadoop-hdfs/dn_sockethdfs-site.xml os linux command : netstat -an unix 2 [ ACC ] STREAM LISTENING 10576083 /var/run/hadoop-hdfs/dn_socket have a LISTENING， but no CONNECTED
回复：hbase CMS gc pause serious program
No use bucket cache， I Can try testing environment。 hbase-site.xml: hbase.master qihe015005:6 hbase.rootdir hdfs://fscluster/hbase hbase.cluster.distributed true hbase.zookeeper.quorum 10.15.5.120:2181,10.15.5.107:2181,10.15.5.55:2181,10.15.5.56:2181,10.15.2.31:2181 hbase.zookeeper.property.dataDir /home/hadoop/data/hbase/zookeeper zookeeper.znode.parent /hbasecluster hbase.tmp.dir /home/hadoop/data/hbase/tmp hbase.fs.tmp.dir /home/hadoop/data/hbase/tmp/hbase-staging hbase.local.dir /home/hadoop/data/hbase/local hbase.master.logcleaner.ttl 60 hbase.regionserver.logroll.period 360 hbase.regionserver.global.memstore.size 0.4 hbase.regionserver.global.memstore.size.lower.limit 0.35 hbase.regionserver.region.split.policy org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy hbase.regionserver.regionSplitLimit 1600 zookeeper.session.timeout 12 hbase.normalizer.enabled true hbase.normalizer.period 360 hbase.server.thread.wakefrequency 1 hbase.server.versionfile.writeattempts 3 hbase.hregion.memstore.flush.size 134217728 hbase.hregion.memstore.block.multiplier 4 hbase.hregion.memstore.mslab.enabled true hbase.hregion.max.filesize 6442450944 hbase.hregion.majorcompaction 60480 hbase.hstore.compactionThreshold 5 hbase.hstore.flusher.count 8 hbase.hstore.blockingStoreFiles 16 hbase.hstore.blockingWaitTime 3 hbase.hstore.compaction.min 6 hbase.hstore.compaction.max 12 hbase.hstore.compaction.min.size 134217728 hbase.hstore.compaction.ratio 1.2F hbase.regionserver.thread.compaction.throttle 2684354560 hbase.hstore.compaction.kv.max 100 hbase.storescanner.parallel.seek.enable true hbase.storescanner.parallel.seek.threads 10 hfile.block.cache.size 0.4 hbase.rpc.timeout 9 hbase.server.compactchecker.interval.multiplier 1000 hbase.security.authentication simple hbase.regionserver.storefile.refresh.period 15000 hbase.region.replica.replication.enabled true hbase.replication true hbase.ipc.warn.response.time 3000 hbase.ipc.warn.response.size 10485760 hbase.quota.enabled true hbase.regionserver.handler.count 180 hbase.snapshot.enabled true hbase.rest.port 20550 hbase.rest.info.port 8085 >hbase.rest.readonly false hbase.rest.threads.min 2 hbase.thrift.minWorkerThreads 200 hbase.thrift.info.port 9095 thrift.accept-backlog 511 --发件人：Ted Yu <yuzhih...@gmail.com>发送时间：2017年3月10日(星期五) 21:31收件人：user <user@hbase.apache.org>抄 送：gehaijiang <gehaiji...@aliyun.com>主 题：Re: hbase CMS gc pause serious program Attachment didn't go through. Do you use bucket cache ? It would reduce GC pause. On Mar 9, 2017, at 9:24 PM, gehaijiang <gehaiji...@aliyun.com> wrote: CMS GC program: 2017-03-10T10:15:25.741+0800: 4555916.378: [GC2017-03-10T10:15:25.741+0800: 4555916.378: [ParNew: 3067136K->340736K(3067136K), 2.0813220 secs] 79945091K->77675170K(100322560K), 2.0816590 secs] [Times: user=4.07 sys=0.35, real=2.09 secs]2017-03-10T10:15:29.524+0800: 4555920.160: [GC2017-03-10T10:15:29.524+0800: 4555920.160: [ParNew: 3067133K->340736K(3067136K), 2.0586980 secs] 80328431K->78058138K(100322560K), 2.0590280 secs] [Times: user=3.94 sys=0.34, real=2.05 secs]2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-sweep: 1441.773/1618.869 secs] [Times: user=2518.60 sys=59.25, real=1618.62 secs]2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-reset-start]2017-03-10T10:15:33.126+0800: 4555923.762: [CMS-concurrent-reset: 0.215/0.215
回复：hbase CMS gc pause serious program
hbase 1.1.2 ， JDK8 compatibility issues ？ --发件人：Sean Busbey <bus...@apache.org>发送时间：2017年3月10日(星期五) 23:48收件人：user <user@hbase.apache.org>; gehaijiang <gehaiji...@aliyun.com>主 题：Re: hbase CMS gc pause serious program For heap sizes larger than ~16GiB you should be using Java 8 and the G1GC collector. You'll need to plan some time to tune G1GC for your workload. There are some pointers doing so in this blog post: https://blogs.apache.org/hbase/entry/tuning_g1gc_for_your_hbase On Thu, Mar 9, 2017 at 11:24 PM, gehaijiang <gehaiji...@aliyun.com> wrote: > CMS GC program: > > 2017-03-10T10:15:25.741+0800: 4555916.378: [GC2017-03-10T10:15:25.741+0800: > 4555916.378: [ParNew: 3067136K->340736K(3067136K), 2.0813220 secs] > 79945091K->77675170K(100322560K), 2.0816590 secs] [Times: user=4.07 > sys=0.35, real=2.09 secs] > 2017-03-10T10:15:29.524+0800: 4555920.160: [GC2017-03-10T10:15:29.524+0800: > 4555920.160: [ParNew: 3067133K->340736K(3067136K), 2.0586980 secs] > 80328431K->78058138K(100322560K), 2.0590280 secs] [Times: user=3.94 > sys=0.34, real=2.05 secs] > 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-sweep: > 1441.773/1618.869 secs] [Times: user=2518.60 sys=59.25, real=1618.62 secs] > 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-reset-start] > > 2017-03-10T10:15:33.126+0800: 4555923.762: [CMS-concurrent-reset: > 0.215/0.215 secs] [Times: user=1.23 sys=0.08, real=0.22 secs] > 2017-03-10T10:15:33.236+0800: 4555923.873: [GC2017-03-10T10:15:33.237+0800: > 4555923.873: [ParNew: 3067011K->340736K(3067136K), 2.4140270 secs] > 80615855K->78315999K(100322560K), 2.4144230 secs] [Times: user=4.63 > sys=0.36, real=2.41 secs] > 2017-03-10T10:15:35.655+0800: 4555926.292: [GC [1 CMS-initial-mark: > 77975263K(97255424K)] 78316286K(100322560K), 0.0149650 secs] [Times: > user=0.01 sys=0.00, real=0.01 secs] > 2017-03-10T10:15:35.671+0800: 4555926.307: [CMS-concurrent-mark-start] > 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-mark: 0.427/0.427 > secs] [Times: user=5.72 sys=0.05, real=0.43 secs] > 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-preclean-start] > 2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-preclean: > 0.192/0.193 secs] [Times: user=0.80 sys=0.03, real=0.19 secs] > 2017-03-10T10:15:36.291+0800: 4555926.928: > [CMS-concurrent-abortable-preclean-start] > 2017-03-10T10:15:37.378+0800: 4555928.014: [GC2017-03-10T10:15:37.378+0800: > 4555928.014: [ParNew: 3067083K->340736K(3067136K), 2.6221190 secs] > 81042347K->78771078K(100322560K), 2.6224970 secs] [Times: user=4.79 > sys=0.48, real=2.62 secs] > 2017-03-10T10:15:41.012+0800: 4555931.648: > [CMS-concurrent-abortable-preclean: 2.083/4.721 secs] [Times: user=13.51 > sys=0.87, real=4.72 secs] > 2017-03-10T10:15:41.015+0800: 4555931.652: [GC[YG occupancy: 2011637 K > (3067136 K)]2017-03-10T10:15:41.016+0800: 4555931.652: > [GC2017-03-10T10:15:41.016+0800: 4555931.652: [ParNew: > 2011637K->340736K(3067136K), 2.0773980 secs] > 80441979K->79117650K(100322560K), 2.0777380 secs] [Times: user=4.09 > sys=0.38, real=2.07 secs] > > > regionserver JVM config: > > export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=256m > -XX:MaxPermSize=256m -Xms96G -Xmx96G" > export HBASE_OPTS="$HBASE_OPTS -Djava.net.preferIPv4Stack=true > -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60 > -XX:+CMSParallelRemarkEnabled -XX:+CMSConcurrentMTEnabled > -XX:ParallelGCThreads=40 -XX:+DisableExplicitGC -XX:+PrintGCDetails > -XX:+PrintGCDateStamps -verbose:gc > -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1 > -XX:+CMSScavengeBeforeRemark -XX:+HeapDumpOnOutOfMemoryError > > > attachment：hdfs-site.xml
hbase CMS gc pause serious program
CMS GC program: 2017-03-10T10:15:25.741+0800: 4555916.378: [GC2017-03-10T10:15:25.741+0800: 4555916.378: [ParNew: 3067136K->340736K(3067136K), 2.0813220 secs] 79945091K->77675170K(100322560K), 2.0816590 secs] [Times: user=4.07 sys=0.35, real=2.09 secs]2017-03-10T10:15:29.524+0800: 4555920.160: [GC2017-03-10T10:15:29.524+0800: 4555920.160: [ParNew: 3067133K->340736K(3067136K), 2.0586980 secs] 80328431K->78058138K(100322560K), 2.0590280 secs] [Times: user=3.94 sys=0.34, real=2.05 secs]2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-sweep: 1441.773/1618.869 secs] [Times: user=2518.60 sys=59.25, real=1618.62 secs]2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-reset-start]2017-03-10T10:15:33.126+0800: 4555923.762: [CMS-concurrent-reset: 0.215/0.215 secs] [Times: user=1.23 sys=0.08, real=0.22 secs] 2017-03-10T10:15:33.236+0800: 4555923.873: [GC2017-03-10T10:15:33.237+0800: 4555923.873: [ParNew: 3067011K->340736K(3067136K), 2.4140270 secs] 80615855K->78315999K(100322560K), 2.4144230 secs] [Times: user=4.63 sys=0.36, real=2.41 secs] 2017-03-10T10:15:35.655+0800: 4555926.292: [GC [1 CMS-initial-mark: 77975263K(97255424K)] 78316286K(100322560K), 0.0149650 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 2017-03-10T10:15:35.671+0800: 4555926.307: [CMS-concurrent-mark-start] 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-mark: 0.427/0.427 secs] [Times: user=5.72 sys=0.05, real=0.43 secs] 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-preclean-start] 2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-preclean: 0.192/0.193 secs] [Times: user=0.80 sys=0.03, real=0.19 secs] 2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-abortable-preclean-start] 2017-03-10T10:15:37.378+0800: 4555928.014: [GC2017-03-10T10:15:37.378+0800: 4555928.014: [ParNew: 3067083K->340736K(3067136K), 2.6221190 secs] 81042347K->78771078K(100322560K), 2.6224970 secs] [Times: user=4.79 sys=0.48, real=2.62 secs] 2017-03-10T10:15:41.012+0800: 4555931.648: [CMS-concurrent-abortable-preclean: 2.083/4.721 secs] [Times: user=13.51 sys=0.87, real=4.72 secs] 2017-03-10T10:15:41.015+0800: 4555931.652: [GC[YG occupancy: 2011637 K (3067136 K)]2017-03-10T10:15:41.016+0800: 4555931.652: [GC2017-03-10T10:15:41.016+0800: 4555931.652: [ParNew: 2011637K->340736K(3067136K), 2.0773980 secs] 80441979K->79117650K(100322560K), 2.0777380 secs] [Times: user=4.09 sys=0.38, real=2.07 secs] regionserver JVM config:export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=256m -XX:MaxPermSize=256m -Xms96G -Xmx96G" export HBASE_OPTS="$HBASE_OPTS -Djava.net.preferIPv4Stack=true -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSParallelRemarkEnabled -XX:+CMSConcurrentMTEnabled -XX:ParallelGCThreads=40 -XX:+DisableExplicitGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -verbose:gc -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1 -XX:+CMSScavengeBeforeRemark -XX:+HeapDumpOnOutOfMemoryError attachment： hdfs-site.xml
