[
https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15231863#comment-15231863
]
Hiroshi Ikeda commented on HBASE-15525:
---------------------------------------
It is better to keep the pool in BufferChain or something in order to return
each borrowed buffer just after its completion to write, especially when the
pool is exhausted. This also gives chances to replace heap buffers with direct
buffers anytime; Concretely, it is better to check the possibility before you
actually write it to GatheringByteChannel, because the Oracle implementation
temporarily copies heaps to direct buffers and the copying is occurred in
multiple times for the same heap buffer unless it can be written at one time.
Under the condition that the buffers to send are filled with 64KB or something
(>= native socket buffer size) except the first and last buffers, it is enough
to create an array of 2 buffers to write them into GatheringByteChannel. That
means the above checking/replacing is enough for the 2 buffers.
When the final checking fails, the Oracle implementation uses thread-local
direct buffers and feedbacks to the heap buffers. (That is regardless of the
sum of the available data, but the existing restriction about 'chunk' keeps
safe because of the above condition.) That involves a bit of off-heap
fragmentation until the thread-local pool is filled with enough large buffers,
and you might want to create you own thread-local buffers with 64KB or
something.
> OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
> --------------------------------------------------------------------------
>
> Key: HBASE-15525
> URL: https://issues.apache.org/jira/browse/HBASE-15525
> Project: HBase
> Issue Type: Sub-task
> Components: IPC/RPC
> Reporter: deepankar
> Assignee: Anoop Sam John
> Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15525_V1.patch, HBASE-15525_V2.patch,
> HBASE-15525_WIP.patch, WIP.patch
>
>
> After HBASE-13819 the system some times run out of direct memory whenever
> there is some network congestion or some client side issues.
> This was because of pending RPCs in the RPCServer$Connection.responseQueue
> and since all the responses in this queue hold a buffer for cellblock from
> BoundedByteBufferPool this could takeup a lot of memory if the
> BoundedByteBufferPool's moving average settles down towards a higher value
> See the discussion here
> [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
