[
https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182851#comment-13182851
]
Lars Hofhansl commented on HBASE-4956:
--------------------------------------
Looking at the stacktrace that Jonathan posted to the asynchhabse group:
{code}
java.nio.Buffer.<init>(Buffer.java:172)
java.nio.ByteBuffer.<init>(ByteBuffer.java:259)
java.nio.ByteBuffer.<init>(ByteBuffer.java:267)
java.nio.MappedByteBuffer.<init>(MappedByteBuffer.java:64)
java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:97)
java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:155)
sun.nio.ch.IOUtil.write(IOUtil.java:37)
sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
--> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
--> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
java.io.DataOutputStream.flush(DataOutputStream.java:106)
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:518)
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:751)
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
$Proxy11.getProtocolVersion(<Unknown Source>:Unknown line)
{code}
Since the writes go through a BufferOutputStream they are already limited to 8k
chunks (default buffer size).
So I do not really know what the problem is... In order to make this any kind
of problem you need have buffers cached for 1m threads (to get to 8GB).
Anyway with the optional global ExecutorService of HBASE-4805 this should no
longer be a problem.
> Control direct memory buffer consumption by HBaseClient
> -------------------------------------------------------
>
> Key: HBASE-4956
> URL: https://issues.apache.org/jira/browse/HBASE-4956
> Project: HBase
> Issue Type: New Feature
> Reporter: Ted Yu
>
> As Jonathan explained here
> https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1
> , standard hbase client inadvertently consumes large amount of direct memory.
> We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
