[jira] [Commented] (HBASE-4863) Make HBase Thrift server more configurable and add a command-line UI test
[ https://issues.apache.org/jira/browse/HBASE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13157126#comment-13157126 ] Hadoop QA commented on HBASE-4863: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505097/D531.4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -162 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 67 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.client.TestInstantSchemaChange Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/366//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/366//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/366//console This message is automatically generated. Make HBase Thrift server more configurable and add a command-line UI test - Key: HBASE-4863 URL: https://issues.apache.org/jira/browse/HBASE-4863 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: 0001-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch, 0002-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch, D531.1.patch, D531.2.patch, D531.3.patch, D531.4.patch This started as an internal hotfix where we found out that the Thrift server spawned 15000 threads. To bound the thread pool size I added a custom thread pool server implementation called HBaseThreadPoolServer into HBase codebase, and made the following parameters configurable from both command line and as config settings: minWorkerThreads, maxWorkerThreads, and maxQueuedRequests. Under an increasing load, the server creates new threads for every connection before the pool size reaches minWorkerThreads. After that, the server puts new connections into the queue and only creates a new thread when the queue is full. If an attempt to create a new thread fails, the server drops connection. The default TThreadPoolServer would crash in that case, but it never happened because the thread pool was unbounded, so the server would hang indefinitely, consume a lot of memory, and cause huge latency spikes on the client side. Another part of this fix is refactoring and unit testing of the command-line part of the Thrift server. The logic there is sufficiently complicated, and the existing ThriftServer class does not test that part at all. The new TestThriftServerCmdLine test starts the Thrift server on a random port with various combinations of options and talks to it through the client API from another thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4863) Make HBase Thrift server more configurable and add a command-line UI test
[
https://issues.apache.org/jira/browse/HBASE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156821#comment-13156821
]
Ted Yu commented on HBASE-4863:
---
I got compilation error:
{code}
testRunThriftServer[0](org.apache.hadoop.hbase.thrift.TestThriftServerCmdLine)
Time elapsed: 2.047 sec ERROR!
java.lang.Error: Unresolved compilation problem:
Cannot make a static reference to the non-static method
getColumnDescriptors() from the type TestThriftServer
at
org.apache.hadoop.hbase.thrift.TestThriftServer.createDropTable(TestThriftServer.java:111)
{code}
Since HBaseThreadPoolServer extends TServer, I think a better name for the
class would be TThreadPoolServer.
Make HBase Thrift server more configurable and add a command-line UI test
-
Key: HBASE-4863
URL: https://issues.apache.org/jira/browse/HBASE-4863
Project: HBase
Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: D531.1.patch, D531.2.patch
This started as an internal hotfix where we found out that the Thrift server
spawned 15000 threads. To bound the thread pool size I added a custom thread
pool server implementation called HBaseThreadPoolServer into HBase codebase,
and made the following parameters configurable from both command line and as
config settings: minWorkerThreads, maxWorkerThreads, and maxQueuedRequests.
Under an increasing load, the server creates new threads for every connection
before the pool size reaches minWorkerThreads. After that, the server puts
new connections into the queue and only creates a new thread when the queue
is full. If an attempt to create a new thread fails, the server drops
connection. The default TThreadPoolServer would crash in that case, but it
never happened because the thread pool was unbounded, so the server would
hang indefinitely, consume a lot of memory, and cause huge latency spikes on
the client side.
Another part of this fix is refactoring and unit testing of the command-line
part of the Thrift server. The logic there is sufficiently complicated, and
the existing ThriftServer class does not test that part at all. The new
TestThriftServerCmdLine test starts the Thrift server on a random port with
various combinations of options and talks to it through the client API from
another thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4863) Make HBase Thrift server more configurable and add a command-line UI test
[ https://issues.apache.org/jira/browse/HBASE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156828#comment-13156828 ] Phabricator commented on HBASE-4863: tedyu has commented on the revision [jira] [HBASE-4863] Make HBase Thrift server more configurable and add a command-line UI test. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/thrift/HBaseThreadPoolServer.java:64 Please add javadoc for the keys. These keys should be placed into hbase-default.xml src/main/java/org/apache/hadoop/hbase/thrift/HBaseThreadPoolServer.java:80 Is TIME_TO_WAIT_AFTER_SHUTDOWN_MS a better name for this constant ? REVISION DETAIL https://reviews.facebook.net/D531 Make HBase Thrift server more configurable and add a command-line UI test - Key: HBASE-4863 URL: https://issues.apache.org/jira/browse/HBASE-4863 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D531.1.patch, D531.2.patch This started as an internal hotfix where we found out that the Thrift server spawned 15000 threads. To bound the thread pool size I added a custom thread pool server implementation called HBaseThreadPoolServer into HBase codebase, and made the following parameters configurable from both command line and as config settings: minWorkerThreads, maxWorkerThreads, and maxQueuedRequests. Under an increasing load, the server creates new threads for every connection before the pool size reaches minWorkerThreads. After that, the server puts new connections into the queue and only creates a new thread when the queue is full. If an attempt to create a new thread fails, the server drops connection. The default TThreadPoolServer would crash in that case, but it never happened because the thread pool was unbounded, so the server would hang indefinitely, consume a lot of memory, and cause huge latency spikes on the client side. Another part of this fix is refactoring and unit testing of the command-line part of the Thrift server. The logic there is sufficiently complicated, and the existing ThriftServer class does not test that part at all. The new TestThriftServerCmdLine test starts the Thrift server on a random port with various combinations of options and talks to it through the client API from another thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4863) Make HBase Thrift server more configurable and add a command-line UI test
[ https://issues.apache.org/jira/browse/HBASE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156883#comment-13156883 ] Phabricator commented on HBASE-4863: tedyu has commented on the revision [jira] [HBASE-4863] Make HBase Thrift server more configurable and add a command-line UI test. Should similar changes in thrift/ThriftServer.java be applied to thrift2/ThriftServer.java ? INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/thrift/HBaseThreadPoolServer.java:111 Should this become a parameter user can adjust ? src/main/java/org/apache/hadoop/hbase/thrift/HBaseThreadPoolServer.java:263 Should ttx.getType() be logged ? src/main/java/org/apache/hadoop/hbase/thrift/ThriftServer.java:179 Should read 'Exactly one ' REVISION DETAIL https://reviews.facebook.net/D531 Make HBase Thrift server more configurable and add a command-line UI test - Key: HBASE-4863 URL: https://issues.apache.org/jira/browse/HBASE-4863 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D531.1.patch, D531.2.patch This started as an internal hotfix where we found out that the Thrift server spawned 15000 threads. To bound the thread pool size I added a custom thread pool server implementation called HBaseThreadPoolServer into HBase codebase, and made the following parameters configurable from both command line and as config settings: minWorkerThreads, maxWorkerThreads, and maxQueuedRequests. Under an increasing load, the server creates new threads for every connection before the pool size reaches minWorkerThreads. After that, the server puts new connections into the queue and only creates a new thread when the queue is full. If an attempt to create a new thread fails, the server drops connection. The default TThreadPoolServer would crash in that case, but it never happened because the thread pool was unbounded, so the server would hang indefinitely, consume a lot of memory, and cause huge latency spikes on the client side. Another part of this fix is refactoring and unit testing of the command-line part of the Thrift server. The logic there is sufficiently complicated, and the existing ThriftServer class does not test that part at all. The new TestThriftServerCmdLine test starts the Thrift server on a random port with various combinations of options and talks to it through the client API from another thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4863) Make HBase Thrift server more configurable and add a command-line UI test
[ https://issues.apache.org/jira/browse/HBASE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156923#comment-13156923 ] Hadoop QA commented on HBASE-4863: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505038/0001-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -162 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 67 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.util.TestThreads Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/364//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/364//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/364//console This message is automatically generated. Make HBase Thrift server more configurable and add a command-line UI test - Key: HBASE-4863 URL: https://issues.apache.org/jira/browse/HBASE-4863 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: 0001-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch, D531.1.patch, D531.2.patch, D531.3.patch This started as an internal hotfix where we found out that the Thrift server spawned 15000 threads. To bound the thread pool size I added a custom thread pool server implementation called HBaseThreadPoolServer into HBase codebase, and made the following parameters configurable from both command line and as config settings: minWorkerThreads, maxWorkerThreads, and maxQueuedRequests. Under an increasing load, the server creates new threads for every connection before the pool size reaches minWorkerThreads. After that, the server puts new connections into the queue and only creates a new thread when the queue is full. If an attempt to create a new thread fails, the server drops connection. The default TThreadPoolServer would crash in that case, but it never happened because the thread pool was unbounded, so the server would hang indefinitely, consume a lot of memory, and cause huge latency spikes on the client side. Another part of this fix is refactoring and unit testing of the command-line part of the Thrift server. The logic there is sufficiently complicated, and the existing ThriftServer class does not test that part at all. The new TestThriftServerCmdLine test starts the Thrift server on a random port with various combinations of options and talks to it through the client API from another thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4863) Make HBase Thrift server more configurable and add a command-line UI test
[
https://issues.apache.org/jira/browse/HBASE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156956#comment-13156956
]
Ted Yu commented on HBASE-4863:
---
In thrift2/ThriftServer.java:
{code}
} else {
server = getTThreadPoolServer(protocolFactory, processor,
transportFactory, inetSocketAddress);
{code}
where
{code}
TThreadPoolServer.Args serverArgs = new
TThreadPoolServer.Args(serverTransport);
{code}
It would be nice to incorporate TBoundedThreadPoolServer into the above module.
This can be done in a separate JIRA.
Make HBase Thrift server more configurable and add a command-line UI test
-
Key: HBASE-4863
URL: https://issues.apache.org/jira/browse/HBASE-4863
Project: HBase
Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments:
0001-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch, D531.1.patch,
D531.2.patch, D531.3.patch
This started as an internal hotfix where we found out that the Thrift server
spawned 15000 threads. To bound the thread pool size I added a custom thread
pool server implementation called HBaseThreadPoolServer into HBase codebase,
and made the following parameters configurable from both command line and as
config settings: minWorkerThreads, maxWorkerThreads, and maxQueuedRequests.
Under an increasing load, the server creates new threads for every connection
before the pool size reaches minWorkerThreads. After that, the server puts
new connections into the queue and only creates a new thread when the queue
is full. If an attempt to create a new thread fails, the server drops
connection. The default TThreadPoolServer would crash in that case, but it
never happened because the thread pool was unbounded, so the server would
hang indefinitely, consume a lot of memory, and cause huge latency spikes on
the client side.
Another part of this fix is refactoring and unit testing of the command-line
part of the Thrift server. The logic there is sufficiently complicated, and
the existing ThriftServer class does not test that part at all. The new
TestThriftServerCmdLine test starts the Thrift server on a random port with
various combinations of options and talks to it through the client API from
another thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4863) Make HBase Thrift server more configurable and add a command-line UI test
[
https://issues.apache.org/jira/browse/HBASE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156964#comment-13156964
]
Ted Yu commented on HBASE-4863:
---
{code}
testSleepWithoutInterrupt(org.apache.hadoop.hbase.util.TestThreads) Time
elapsed: 5.004 sec FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at
org.apache.hadoop.hbase.util.TestThreads.testSleepWithoutInterrupt(TestThreads.java:57)
{code}
points to this line:
{code}
assertTrue(sleeper.isInterrupted());
{code}
Make HBase Thrift server more configurable and add a command-line UI test
-
Key: HBASE-4863
URL: https://issues.apache.org/jira/browse/HBASE-4863
Project: HBase
Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments:
0001-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch, D531.1.patch,
D531.2.patch, D531.3.patch
This started as an internal hotfix where we found out that the Thrift server
spawned 15000 threads. To bound the thread pool size I added a custom thread
pool server implementation called HBaseThreadPoolServer into HBase codebase,
and made the following parameters configurable from both command line and as
config settings: minWorkerThreads, maxWorkerThreads, and maxQueuedRequests.
Under an increasing load, the server creates new threads for every connection
before the pool size reaches minWorkerThreads. After that, the server puts
new connections into the queue and only creates a new thread when the queue
is full. If an attempt to create a new thread fails, the server drops
connection. The default TThreadPoolServer would crash in that case, but it
never happened because the thread pool was unbounded, so the server would
hang indefinitely, consume a lot of memory, and cause huge latency spikes on
the client side.
Another part of this fix is refactoring and unit testing of the command-line
part of the Thrift server. The logic there is sufficiently complicated, and
the existing ThriftServer class does not test that part at all. The new
TestThriftServerCmdLine test starts the Thrift server on a random port with
various combinations of options and talks to it through the client API from
another thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
