[jira] [Created] (HBASE-17182) Memory leak from openScanner of HBase thrift2

[JIRA]

易剑 created HBASE-17182:
--

 Summary: Memory leak from openScanner of HBase thrift2
 Key: HBASE-17182
 URL: https://issues.apache.org/jira/browse/HBASE-17182
 Project: HBase
  Issue Type: Bug
  Components: Thrift
Reporter: 易剑


Client call openScanner, but client (coredump or others) not closeScanner, the 
scanner will not be removed from scannerMap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-17180) Let HBase thrift2 support TThreadedSelectorServer

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-17180.
---
Resolution: Invalid

Same with HBASE-17181.

> Let HBase thrift2 support TThreadedSelectorServer
> -
>
> Key: HBASE-17180
> URL: https://issues.apache.org/jira/browse/HBASE-17180
> Project: HBase
>  Issue Type: New Feature
>  Components: Thrift
>Affects Versions: 1.2.3
>Reporter: 易剑
>Priority: Minor
>  Labels: features
> Fix For: 1.2.3
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Add TThreadedSelectorServer for HBase Thrift2



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17180) Let HBase thrift2 support TThreadedSelectorServer

[JIRA]

易剑 created HBASE-17180:
--

 Summary: Let HBase thrift2 support TThreadedSelectorServer
 Key: HBASE-17180
 URL: https://issues.apache.org/jira/browse/HBASE-17180
 Project: HBase
  Issue Type: New Feature
  Components: Thrift
Affects Versions: 1.2.3
Reporter: 易剑
Priority: Minor
 Fix For: 1.2.3


Add TThreadedSelectorServer for HBase Thrift2



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17181) Let HBase thrift2 support TThreadedSelectorServer

[JIRA]

易剑 created HBASE-17181:
--

 Summary: Let HBase thrift2 support TThreadedSelectorServer
 Key: HBASE-17181
 URL: https://issues.apache.org/jira/browse/HBASE-17181
 Project: HBase
  Issue Type: New Feature
  Components: Thrift
Affects Versions: 1.2.3
Reporter: 易剑
Priority: Minor
 Fix For: 1.2.3


Add TThreadedSelectorServer for HBase Thrift2



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17179) the chunkpool are not reclamed correctly

[chenxu (JIRA)]

chenxu created HBASE-17179:
--

 Summary: the chunkpool are not reclamed correctly
 Key: HBASE-17179
 URL: https://issues.apache.org/jira/browse/HBASE-17179
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.1.2
Reporter: chenxu


in our cluster, the RS's config is as below
{quote}
heap size(25G)
hbase.regionserver.global.memstore.size(0.5)
hbase.hregion.memstore.chunkpool.maxsize(0.8)
hbase.hregion.memstore.chunkpool.initialsize(1.0)
{quote}
so the chunkpool's size is 25*0.5*0.8 = 10GB
in our monitor system, the _memStoreSize_ never up to 3G，so the _reuseRatio_ of 
the chunkpool should always 100%, but it's not, see the logs below:
{quote}
2016-11-23 11:47:34,302 DEBUG 
[StoreOpener-33a77b971aff15e5e6be3cc614870e43-1-MemStoreChunkPool Statistics] 
regionserver.MemStoreChunkPool: Stats: current pool size=5009,created chunk 
count=0,reused chunk count=81,reuseRatio=100.00%
2016-11-28 11:47:34,302 DEBUG 
[StoreOpener-33a77b971aff15e5e6be3cc614870e43-1-MemStoreChunkPool Statistics] 
regionserver.MemStoreChunkPool: Stats: current pool size=3,created chunk 
count=1325,reused chunk count=67209,reuseRatio=98.07%
{quote}
5 days later, the _reuseRatio_ is below 100%



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: mini cluster not starting due to unhandled com.lmax.disruptor.dsl.Disruptor. (eclipse only, not in mvn)

[Stephen Jiang]

Thanks, forget the 'mvn eclipse:eclipse' trick.  After doing this, things
working again.

Thanks
Stephen

On Sun, Nov 27, 2016 at 6:51 PM, 张铎  wrote:

> Try mvn eclispe:eclipse again? We upgraded disruptor to 3.3.6 recently.
>
> 2016-11-28 10:48 GMT+08:00 Stephen Jiang :
>
> > I had problem to start mini cluster in eclipse in the last few days.  In
> > master branch (without any additional change), I got the following FATAL
> > error (running the UT using maven has NO problem):
> >
> > 2016-11-27 18:44:58,102 FATAL [M:0;10.10.0.153:59578]
> > master.HMaster(2241):
> > Master server abort: loaded coprocessors are: []
> > 2016-11-27 18:44:58,102 FATAL [M:0;10.10.0.153:59578]
> > master.HMaster(2244):
> > Unhandled:
> > com.lmax.disruptor.dsl.Disruptor.(Lcom/lmax/
> > disruptor/EventFactory;ILjava/util/concurrent/ThreadFactory;
> > Lcom/lmax/disruptor/dsl/ProducerType;Lcom/lmax/disruptor/WaitStrategy;)V
> > java.lang.NoSuchMethodError:
> > com.lmax.disruptor.dsl.Disruptor.(Lcom/lmax/
> > disruptor/EventFactory;ILjava/util/concurrent/ThreadFactory;
> > Lcom/lmax/disruptor/dsl/ProducerType;Lcom/lmax/disruptor/WaitStrategy;)V
> > at org.apache.hadoop.hbase.regionserver.wal.FSHLog.
> (FSHLog.java:230)
> > at
> > org.apache.hadoop.hbase.wal.FSHLogProvider.createWAL(
> > FSHLogProvider.java:80)
> > at
> > org.apache.hadoop.hbase.wal.FSHLogProvider.createWAL(
> > FSHLogProvider.java:39)
> > at
> > org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(
> > AbstractFSWALProvider.java:132)
> > at
> > org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(
> > AbstractFSWALProvider.java:52)
> > at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:242)
> > at
> > org.apache.hadoop.hbase.regionserver.HRegionServer.
> > getWAL(HRegionServer.java:1929)
> > at
> > org.apache.hadoop.hbase.regionserver.HRegionServer.
> > buildServerLoad(HRegionServer.java:1247)
> > at
> > org.apache.hadoop.hbase.regionserver.HRegionServer.
> tryRegionServerReport(
> > HRegionServer.java:1205)
> > at
> > org.apache.hadoop.hbase.regionserver.HRegionServer.
> > run(HRegionServer.java:1022)
> > at java.lang.Thread.run(Thread.java:745)
> >
>

[jira] [Created] (HBASE-17178) Add region balance throttling

[Guanghao Zhang (JIRA)]

Guanghao Zhang created HBASE-17178:
--

 Summary: Add region balance throttling
 Key: HBASE-17178
 URL: https://issues.apache.org/jira/browse/HBASE-17178
 Project: HBase
  Issue Type: Improvement
  Components: Balancer
Reporter: Guanghao Zhang


Our online cluster serves dozens of  tables and different tables serve for 
different services. If the balancer moves too many regions in the same time, 
it will decrease the availability for some table or some services. So we add 
region balance throttling on our online serve cluster. 
We introduce a new config hbase.balancer.max.balancing.regions, which means the 
max number of regions in transition when balancing.
If we config this to 1 and a table have 100 regions, then the table will have 
99 regions available at any time. It helps a lot for our use case and it has 
been running a long time
our production cluster.

But for some use case, we need the balancer run faster. If a cluster has 100 
regionservers, then it add 50 new regionservers for peak requests. Then it need 
balancer run as soon as
possible and let the cluster reach a balance state soon. Our idea is compute 
max number of regions in transition by the max balancing time and the average 
time of region in transition.
Then the balancer use the computed value to throttling.

Examples for understanding.
A cluster has 100 regionservers, each regionserver has 200 regions and the 
average time of region in transition is 1 seconds, we config the max balancing 
time is 10 * 60 seconds.
Case 1. One regionserver crash, the cluster at most need balance 200 regions. 
Then 200 / (10 * 60s / 1s) < 1, it means the max number of regions in 
transition is 1 when balancing. Then the balancer can move region one by one 
and the cluster will have high availability  when balancing.
Case 2. Add other 100 regionservers, the cluster at most need balance 1 
regions. Then 1 / (10 * 60s / 1s) = 16.7, it means the max number of 
regions in transition is 17 when balancing. Then the cluster can reach a 
balance state within the max balancing time.

Any suggestions are welcomed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: mini cluster not starting due to unhandled com.lmax.disruptor.dsl.Disruptor. (eclipse only, not in mvn)

[张铎]

Try mvn eclispe:eclipse again? We upgraded disruptor to 3.3.6 recently.

2016-11-28 10:48 GMT+08:00 Stephen Jiang :

> I had problem to start mini cluster in eclipse in the last few days.  In
> master branch (without any additional change), I got the following FATAL
> error (running the UT using maven has NO problem):
>
> 2016-11-27 18:44:58,102 FATAL [M:0;10.10.0.153:59578]
> master.HMaster(2241):
> Master server abort: loaded coprocessors are: []
> 2016-11-27 18:44:58,102 FATAL [M:0;10.10.0.153:59578]
> master.HMaster(2244):
> Unhandled:
> com.lmax.disruptor.dsl.Disruptor.(Lcom/lmax/
> disruptor/EventFactory;ILjava/util/concurrent/ThreadFactory;
> Lcom/lmax/disruptor/dsl/ProducerType;Lcom/lmax/disruptor/WaitStrategy;)V
> java.lang.NoSuchMethodError:
> com.lmax.disruptor.dsl.Disruptor.(Lcom/lmax/
> disruptor/EventFactory;ILjava/util/concurrent/ThreadFactory;
> Lcom/lmax/disruptor/dsl/ProducerType;Lcom/lmax/disruptor/WaitStrategy;)V
> at org.apache.hadoop.hbase.regionserver.wal.FSHLog.(FSHLog.java:230)
> at
> org.apache.hadoop.hbase.wal.FSHLogProvider.createWAL(
> FSHLogProvider.java:80)
> at
> org.apache.hadoop.hbase.wal.FSHLogProvider.createWAL(
> FSHLogProvider.java:39)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(
> AbstractFSWALProvider.java:132)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(
> AbstractFSWALProvider.java:52)
> at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:242)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.
> getWAL(HRegionServer.java:1929)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.
> buildServerLoad(HRegionServer.java:1247)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(
> HRegionServer.java:1205)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.
> run(HRegionServer.java:1022)
> at java.lang.Thread.run(Thread.java:745)
>

mini cluster not starting due to unhandled com.lmax.disruptor.dsl.Disruptor. (eclipse only, not in mvn)

[Stephen Jiang]

I had problem to start mini cluster in eclipse in the last few days.  In
master branch (without any additional change), I got the following FATAL
error (running the UT using maven has NO problem):

2016-11-27 18:44:58,102 FATAL [M:0;10.10.0.153:59578] master.HMaster(2241):
Master server abort: loaded coprocessors are: []
2016-11-27 18:44:58,102 FATAL [M:0;10.10.0.153:59578] master.HMaster(2244):
Unhandled:
com.lmax.disruptor.dsl.Disruptor.(Lcom/lmax/disruptor/EventFactory;ILjava/util/concurrent/ThreadFactory;Lcom/lmax/disruptor/dsl/ProducerType;Lcom/lmax/disruptor/WaitStrategy;)V
java.lang.NoSuchMethodError:
com.lmax.disruptor.dsl.Disruptor.(Lcom/lmax/disruptor/EventFactory;ILjava/util/concurrent/ThreadFactory;Lcom/lmax/disruptor/dsl/ProducerType;Lcom/lmax/disruptor/WaitStrategy;)V
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.(FSHLog.java:230)
at
org.apache.hadoop.hbase.wal.FSHLogProvider.createWAL(FSHLogProvider.java:80)
at
org.apache.hadoop.hbase.wal.FSHLogProvider.createWAL(FSHLogProvider.java:39)
at
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:132)
at
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:52)
at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:242)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:1929)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1247)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1205)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1022)
at java.lang.Thread.run(Thread.java:745)

Still Failing: HBase Generate Website

[Apache Jenkins Server]

Build status: Still Failing

If successful, the website and docs have been generated. To update the live 
site, follow the instructions below. If failed, skip to the bottom of this 
email.

Use the following commands to download the patch and apply it to a clean branch 
based on origin/asf-site. If you prefer to keep the hbase-site repo around 
permanently, you can skip the clone step.

  git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git

  cd hbase-site
  wget -O- 
https://builds.apache.org/job/hbase_generate_website/418/artifact/website.patch.zip
 | funzip > ${GIT_SHA}.patch
  git fetch
  git checkout -b asf-site-${GIT_SHA} origin/asf-site
  git am --whitespace=fix $GIT_SHA.patch

At this point, you can preview the changes by opening index.html or any of the 
other HTML pages in your local asf-site-${GIT_SHA} branch.

There are lots of spurious changes, such as timestamps and CSS styles in 
tables, so a generic git diff is not very useful. To see a list of files that 
have been added, deleted, renamed, changed type, or are otherwise interesting, 
use the following command:

  git diff --name-status --diff-filter=ADCRTXUB origin/asf-site

To see only files that had 100 or more lines changed:

  git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'

When you are satisfied, publish your changes to origin/asf-site using these 
commands:

  git commit --allow-empty -m "Empty commit" # to work around a current ASF 
INFRA bug
  git push origin asf-site-${GIT_SHA}:asf-site
  git checkout asf-site
  git branch -D asf-site-${GIT_SHA}

Changes take a couple of minutes to be propagated. You can verify whether they 
have been propagated by looking at the Last Published date at the bottom of 
http://hbase.apache.org/. It should match the date in the index.html on the 
asf-site branch in Git.

As a courtesy- reply-all to this email to let other committers know you pushed 
the site.



If failed, see https://builds.apache.org/job/hbase_generate_website/418/console