[
https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273688#comment-15273688
]
deepankar commented on HBASE-15525:
-----------------------------------
We pulled in this patch and ran it on one of our production systems, except
couple of places where we missed returning back the buffers, the patch solves
this issue in an excellent way, and only allocates the exact optimal amount of
buffers needed (with limit of 4096 only allocating ~300 only) , this is the
difference that we had to add to this patch to fix the minor leaks
{code}
diff --git
a/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java
b/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java
index 1584a40..378459c 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java
@@ -158,7 +158,13 @@ public class IPCUtil {
assert pool != null;
PoolAwareByteBuffersOutputStream pbbos = new
PoolAwareByteBuffersOutputStream(pool);
encodeCellsTo(pbbos, cellScanner, codec, compressor);
- if (pbbos.size() == 0) return null;
+ if (pbbos.size() == 0) {
+ pbbos.closeAndPutbackBuffers();
+ return null;
+ }
return pbbos;
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
index 517339a..93fac4e 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
@@ -1184,6 +1184,7 @@ public class RpcServer implements RpcServerInterface {
if (error) {
LOG.debug(getName() + call.toShortString() + ": output error --
closing");
closeConnection(call.connection);
+ call.done();
}
}
{code}
Thanks [~anoop.hbase] for the excellent patch and help in debugging the above
issue.
> OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
> --------------------------------------------------------------------------
>
> Key: HBASE-15525
> URL: https://issues.apache.org/jira/browse/HBASE-15525
> Project: HBase
> Issue Type: Sub-task
> Components: IPC/RPC
> Reporter: deepankar
> Assignee: Anoop Sam John
> Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15525_V1.patch, HBASE-15525_V2.patch,
> HBASE-15525_WIP.patch, WIP.patch
>
>
> After HBASE-13819 the system some times run out of direct memory whenever
> there is some network congestion or some client side issues.
> This was because of pending RPCs in the RPCServer$Connection.responseQueue
> and since all the responses in this queue hold a buffer for cellblock from
> BoundedByteBufferPool this could takeup a lot of memory if the
> BoundedByteBufferPool's moving average settles down towards a higher value
> See the discussion here
> [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
