[
https://issues.apache.org/jira/browse/HBASE-19320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16735638#comment-16735638
]
wengliemiao commented on HBASE-19320:
-------------------------------------
Our hbase client version is 1.2.3. Recently we run into direct memory leak too.
Used eclipse MAT analysis the dump file and looked into hbase client's
implement of RPC write and read. I found that the leak is resulted by these
conditions' effect together:
* The default implementation of RpcClient (RpcClientImpl) using
_HeapByteBuffer_ for Network IO
* The _sun.nio.ch.Util_ Object uses _ThreadLocal_ to cache _DirectByteBuffer_
* The _batchPool_ created when _getTable,_ set _allowCoreThreadTimeOut_ to __
true
* The rate of hbase write, such as one write in one second
* There are no FGC of JVM
With the conditions listed above, I recurrent the leak phenomenon at our
performance server.
About the rate of write, when i tested without limit of the rate, the JVM
occurred FGC. The direct memory occupied only reached to 3.5g, and then down to
2g.
In the end, I upgrade the hbase client to 2.1.0, and the leak phenomenon gone.
But according to the conditions listed above, there are many other solutions.
Such as use AsyncRpcClient，limit the rate of hbase write, and so on.
> document the mysterious direct memory leak in hbase
> ----------------------------------------------------
>
> Key: HBASE-19320
> URL: https://issues.apache.org/jira/browse/HBASE-19320
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 1.2.6, 2.0.0
> Reporter: huaxiang sun
> Assignee: huaxiang sun
> Priority: Major
> Attachments: HBASE-19320-master-v001.patch, Screen Shot 2017-11-21 at
> 4.43.36 PM.png, Screen Shot 2017-11-21 at 4.44.22 PM.png
>
>
> Recently we run into a direct memory leak case, which takes some time to
> trace and debug. Internally discussed with our [[email protected]], we
> thought we had some findings and want to share with the community.
> Basically, it is the issue described in
> http://www.evanjones.ca/java-bytebuffer-leak.html and it happened to one of
> our hbase clusters.
> Create the jira first and will fill in more details later.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
