[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13428750#comment-13428750 ] Hudson commented on HBASE-4956: --- Integrated in HBase-0.94-security-on-Hadoop-23 #6 (See [https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/6/]) HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363533) Result = FAILURE tedyu : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Result.java Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu Assignee: Bob Copeland Fix For: 0.96.0, 0.94.1 Attachments: 4956.txt, thread_get.rb As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13419652#comment-13419652 ] Hudson commented on HBASE-4956: --- Integrated in HBase-0.94-security #44 (See [https://builds.apache.org/job/HBase-0.94-security/44/]) HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363533) Result = FAILURE tedyu : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Result.java Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu Assignee: Bob Copeland Fix For: 0.96.0, 0.94.1 Attachments: 4956.txt, thread_get.rb As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418330#comment-13418330 ] Bob Copeland commented on HBASE-4956: - Just a follow-up -- we've been running with this patch (8k window size) for a couple of months now, and have no more OOMs, and no noticeable performance or functionality regressions. Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu Attachments: thread_get.rb As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418383#comment-13418383 ] Hadoop QA commented on HBASE-4956: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537187/4956.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 12 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//console This message is automatically generated. Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu Fix For: 0.96.0, 0.94.1 Attachments: 4956.txt, thread_get.rb As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418384#comment-13418384 ] Lars Hofhansl commented on HBASE-4956: -- Thanks Bob! I think we should commit this. To 0.94 and 0.96 at least. If there're no objections I'll do that. From your analysis it looks like 32k would be the sweet spot, but since you ran it in production with an 8k window, let's commit this. Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu Fix For: 0.96.0, 0.94.1 Attachments: 4956.txt, thread_get.rb As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418394#comment-13418394 ] Bob Copeland commented on HBASE-4956: - Yeah, buffer size doesn't really matter to our workload so out of laziness I stuck with 8k, but a larger one should be OK. Worth noting that 8k is better than the full length (i.e. unpatched) in my tests, presumably because of the additional allocation overhead. Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu Fix For: 0.96.0, 0.94.1 Attachments: 4956.txt, thread_get.rb As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418646#comment-13418646 ] Zhihong Ted Yu commented on HBASE-4956: --- Patch integrated to trunk. Thanks for the patch, Bob. Thanks for the review, Lars. Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu Fix For: 0.96.0, 0.94.1 Attachments: 4956.txt, thread_get.rb As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418655#comment-13418655 ] Zhihong Ted Yu commented on HBASE-4956: --- Integrated to 0.94 since this JIRA is marked against 0.94.1 release. Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu Fix For: 0.96.0, 0.94.1 Attachments: 4956.txt, thread_get.rb As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418657#comment-13418657 ] Lars Hofhansl commented on HBASE-4956: -- Thanks Ted. Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu Fix For: 0.96.0, 0.94.1 Attachments: 4956.txt, thread_get.rb As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418680#comment-13418680 ] Hudson commented on HBASE-4956: --- Integrated in HBase-TRUNK #3153 (See [https://builds.apache.org/job/HBase-TRUNK/3153/]) HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363526) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Result.java Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu Fix For: 0.96.0, 0.94.1 Attachments: 4956.txt, thread_get.rb As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418705#comment-13418705 ] Hudson commented on HBASE-4956: --- Integrated in HBase-0.94 #341 (See [https://builds.apache.org/job/HBase-0.94/341/]) HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363533) Result = SUCCESS tedyu : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Result.java Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu Assignee: Bob Copeland Fix For: 0.96.0, 0.94.1 Attachments: 4956.txt, thread_get.rb As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418797#comment-13418797 ] Hudson commented on HBASE-4956: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #100 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/100/]) HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363526) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Result.java Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu Assignee: Bob Copeland Fix For: 0.96.0, 0.94.1 Attachments: 4956.txt, thread_get.rb As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[
https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13268435#comment-13268435
]
Bob Copeland commented on HBASE-4956:
-
With regard to readFully() being everywhere -- perhaps it would be less
intrusive to subclass DataInput, or to modify one of the read functions further
down in the stack like org.apache.hadoop.net.SocketInputStream.read() to do the
chunking. You guys know the code better than I.
These numbers are for standalone mode. I can't easily do a test on our
distributed cluster but I don't think it would change the numbers much. I'll
upload my test script and then anyone who wants can repeat the experiment on
their setup.
Each trial consists of a run of hbase org.jruby.Main thread_get.rb 1, which
does 100 gets of a ~10 MB row, 3 times in a row, computing avg stddev of
wall-clock time. Maxread variable is changed for every run.
{code}
sizeavg stddev
~~ ~~
133.690 .10
1k3.072 .18
2k3.395 .19
4k3.493 .16
8k3.306 .19
16k 2.918 .21
32k 2.835 .06
64k 2.689 .08
128k 3.048 .03
len 3.328 .08
{code}
Control direct memory buffer consumption by HBaseClient
---
Key: HBASE-4956
URL: https://issues.apache.org/jira/browse/HBASE-4956
Project: HBase
Issue Type: New Feature
Reporter: Ted Yu
As Jonathan explained here
https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1
, standard hbase client inadvertently consumes large amount of direct memory.
We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[
https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267797#comment-13267797
]
Bob Copeland commented on HBASE-4956:
-
I have this bug on our servers (0.90.4-cdh3). I think Johnathan's analysis was
spot on, but it matters much more for read path since readFully() allocates a
direct buffer as large as the target read.
Here's an easy way to reproduce:
1) create a ~10 meg row in some table
2) edit bin/hbase script to have max direct memory size 10m, e.g.:
{code}
+JAVA_HEAP_MAX=-Xmx1000m -XX:MaxDirectMemorySize=1m
{code}
3) now have any client try to read this row. If read side were chunking with
1m pieces, it should still be able to read.
Obviously this example is contrived, but we hit the problem with around 1-2000
threads (we have large rows in our DB). Here's what our stack trace looks like:
{code}
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:633)
at java.nio.DirectByteBuffer.init(DirectByteBuffer.java:98)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:155)
at sun.nio.ch.IOUtil.read(IOUtil.java:174)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
at
org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:299)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at org.apache.hadoop.hbase.client.Result.readFields(Result.java:504)
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:259)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:554)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:477)
{code}
The following works around the problem by only doing 8k allocations:
{code}
diff --git a/src/main/java/org/apache/hadoop/hbase/client/Result.java
b/src/main/java/org/apache/hadoop/hbase/client/Result.java
index 8a0c1a9..378ca88 100644
--- a/src/main/java/org/apache/hadoop/hbase/client/Result.java
+++ b/src/main/java/org/apache/hadoop/hbase/client/Result.java
@@ -502,10 +502,18 @@ public class Result implements Writable, WritableWithSize
{
return;
}
byte [] raw = new byte[totalBuffer];
-in.readFully(raw, 0, totalBuffer);
+readChunked(in, raw, 0, totalBuffer);
bytes = new ImmutableBytesWritable(raw, 0, totalBuffer);
}
+ private void readChunked(final DataInput in, byte[] dest, int ofs, int len)
+ throws IOException {
+int maxread = 8192;
+
+for (; ofs len; ofs += maxread)
+ in.readFully(dest, ofs, Math.min(len - ofs, maxread));
+ }
+
//Create KeyValue[] when needed
private void readFields() {
if (bytes == null) {
{code}
Control direct memory buffer consumption by HBaseClient
---
Key: HBASE-4956
URL: https://issues.apache.org/jira/browse/HBASE-4956
Project: HBase
Issue Type: New Feature
Reporter: Ted Yu
As Jonathan explained here
https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1
, standard hbase client inadvertently consumes large amount of direct memory.
We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267832#comment-13267832 ] Lars Hofhansl commented on HBASE-4956: -- This looks like a good approach. I'd be OK committing this (after passing tests, etc). Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267838#comment-13267838 ] Lars Hofhansl commented on HBASE-4956: -- Potentially every call to readFully over the wire is suspect. Just did a quick check. It's used in Put/Append/KeyValue/Bytes/etc/etc. Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267849#comment-13267849 ] Lars Hofhansl commented on HBASE-4956: -- Yeah, we should quantify this. 8k seems like a very reasonable chunk size, though (probably anything between 1k and 64k would be OK :) ). @Bob: Any chance you can whip up a quick performance test for this? Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[
https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182851#comment-13182851
]
Lars Hofhansl commented on HBASE-4956:
--
Looking at the stacktrace that Jonathan posted to the asynchhabse group:
{code}
java.nio.Buffer.init(Buffer.java:172)
java.nio.ByteBuffer.init(ByteBuffer.java:259)
java.nio.ByteBuffer.init(ByteBuffer.java:267)
java.nio.MappedByteBuffer.init(MappedByteBuffer.java:64)
java.nio.DirectByteBuffer.init(DirectByteBuffer.java:97)
java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:155)
sun.nio.ch.IOUtil.write(IOUtil.java:37)
sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
-- java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
-- java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
java.io.DataOutputStream.flush(DataOutputStream.java:106)
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:518)
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:751)
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
$Proxy11.getProtocolVersion(Unknown Source:Unknown line)
{code}
Since the writes go through a BufferOutputStream they are already limited to 8k
chunks (default buffer size).
So I do not really know what the problem is... In order to make this any kind
of problem you need have buffers cached for 1m threads (to get to 8GB).
Anyway with the optional global ExecutorService of HBASE-4805 this should no
longer be a problem.
Control direct memory buffer consumption by HBaseClient
---
Key: HBASE-4956
URL: https://issues.apache.org/jira/browse/HBASE-4956
Project: HBase
Issue Type: New Feature
Reporter: Ted Yu
As Jonathan explained here
https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1
, standard hbase client inadvertently consumes large amount of direct memory.
We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[
https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182908#comment-13182908
]
stack commented on HBASE-4956:
--
Hmmm... Looking in code, we allocate a BufferedOutputStream using default
buffer size:
{code}
this.out = new DataOutputStream
(new BufferedOutputStream(NetUtils.getOutputStream(socket)));
{code}
... so, it seems that yeah, at any one time, you'd think the maximum allocation
8k as you say Lars.
Jonathan says 3 of these buffers allocated when reading. Presume similar
writing (easy to check I suppose. I havent'). Thats 6*8k per thread since
these are thread local which still don't seem like that much. You'd need
lots of threads to run into trouble.
We need to reproduce.
Control direct memory buffer consumption by HBaseClient
---
Key: HBASE-4956
URL: https://issues.apache.org/jira/browse/HBASE-4956
Project: HBase
Issue Type: New Feature
Reporter: Ted Yu
As Jonathan explained here
https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1
, standard hbase client inadvertently consumes large amount of direct memory.
We should consider using netty for NIO-related tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181872#comment-13181872 ] Lars Hofhansl commented on HBASE-4956: -- @Stack and Ted: Yes, if the application uses the API introduced in HBASE-4805 the number of threads doing network ops is limited to a single thread pools (that can be sized accordingly). Still not ideal. The regionserver side of things does chunking (8k chunks by default), so even if a lot of buffers are cached for a lot of threads it won't add to much memory - even for 1000 threads each caching 3 buffer you'd only use 24mb, and since they are the same size they are reusable). Maybe the client do the same. Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13174884#comment-13174884 ] stack commented on HBASE-4956: -- @Lars So supposition is that we bound the number of direct buffers client allocates by bounding the number of threads in the thread pool (and have all accesses go via the HTable thread pool)? If so, sounds good. What happens if as many HTable instances as there are application-level threads and say the application spawns lots of threads? The application should then go via HTablePool? Or rather, you are suggesting that application uses new HBASE-4805 api and pass common executor to all HTable instances? Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13174892#comment-13174892 ] Zhihong Yu commented on HBASE-4956: --- Currently lily-project uses a hack to share ExecutorService among all HTable instances. Once HBASE-5084 is integrated, client application would have full control over reusing ExecutorService across all HTable instances. Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173917#comment-13173917 ] Lars Hofhansl commented on HBASE-4956: -- Just checked the HTable code in trunk... Puts (even single put operations) already go through a thread from the HTable's thread pool. A single get request is issuing a request directly, but get(ListGet) is also using the pool. So limiting the number of threads in a single thread pool and using the new HTable constructor from HBASE-4805, this can be controlled. Need to make one is using only get(ListGet) in addition to a relatively small global threadpool. At Salesforce we'll shoot for pool with ~100 threads and a waiting queue of about 100 as well (for very large clusters that may be too small though). Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164798#comment-13164798 ] Zhihong Yu commented on HBASE-4956: --- Since the proposal involves asynchronous communication, we should devise new API which can be used to validate the reduction in use of direct memory buffer. Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
