[jira] [Commented] (HBASE-10810) LoadTestTool should share the connection and connection pool
[
https://issues.apache.org/jira/browse/HBASE-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046749#comment-14046749
]
Hudson commented on HBASE-10810:
FAILURE: Integrated in HBase-TRUNK #5245 (See
[https://builds.apache.org/job/HBase-TRUNK/5245/])
HBASE-10810 LoadTestTool should share the connection and connection pool (enis:
rev 25baace0dec1fc4f3b5fb51292c8ec8a6da85ba0)
*
hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestTimeBoundedRequestsWithRegionReplicas.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReaderWithACL.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriterWithACL.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriterBase.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedUpdaterWithACL.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedUpdater.java
LoadTestTool should share the connection and connection pool
Key: HBASE-10810
URL: https://issues.apache.org/jira/browse/HBASE-10810
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.99.0, hbase-10070
Attachments:
0038-HBASE-10810-LoadTestTool-should-share-the-connection.patch,
hbase-10810_v1.patch
While running the IT test from HBASE-10572, we've noticed that the number of
threads jumps to 4K's when CM actions are going on.
Our [~ndimiduk] summarizes the problem quite good:
MultiThreadedReader creates this pool for each HTable:
{code}
ThreadPoolExecutor pool = new ThreadPoolExecutor(1, maxThreads,
keepAliveTime, TimeUnit.SECONDS,
new SynchronousQueueRunnable(),
Threads.newDaemonThreadFactory(htable));
{code}
This comes from the HTable creation
{code}
public HTable(Configuration conf, final TableName tableName)
{code}
As well the javadoc says Recommended.
This is wrong.
In this issue we can change the LTT sub classes to use the shared connection
object and initialize their tables using HConnection.getTable() rather than
new HTable().
This is relevant to trunk as well, but there since there is only one
outstanding RPC per thread, it is not such a big problem.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10810) LoadTestTool should share the connection and connection pool
[
https://issues.apache.org/jira/browse/HBASE-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046638#comment-14046638
]
Enis Soztutar commented on HBASE-10810:
---
Attaching rebased patch for master that is committed
LoadTestTool should share the connection and connection pool
Key: HBASE-10810
URL: https://issues.apache.org/jira/browse/HBASE-10810
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: hbase-10070
Attachments:
0038-HBASE-10810-LoadTestTool-should-share-the-connection.patch,
hbase-10810_v1.patch
While running the IT test from HBASE-10572, we've noticed that the number of
threads jumps to 4K's when CM actions are going on.
Our [~ndimiduk] summarizes the problem quite good:
MultiThreadedReader creates this pool for each HTable:
{code}
ThreadPoolExecutor pool = new ThreadPoolExecutor(1, maxThreads,
keepAliveTime, TimeUnit.SECONDS,
new SynchronousQueueRunnable(),
Threads.newDaemonThreadFactory(htable));
{code}
This comes from the HTable creation
{code}
public HTable(Configuration conf, final TableName tableName)
{code}
As well the javadoc says Recommended.
This is wrong.
In this issue we can change the LTT sub classes to use the shared connection
object and initialize their tables using HConnection.getTable() rather than
new HTable().
This is relevant to trunk as well, but there since there is only one
outstanding RPC per thread, it is not such a big problem.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10810) LoadTestTool should share the connection and connection pool
[
https://issues.apache.org/jira/browse/HBASE-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046686#comment-14046686
]
Enis Soztutar commented on HBASE-10810:
---
Committed to master as part of hbase-10070 branch merge
LoadTestTool should share the connection and connection pool
Key: HBASE-10810
URL: https://issues.apache.org/jira/browse/HBASE-10810
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.99.0, hbase-10070
Attachments:
0038-HBASE-10810-LoadTestTool-should-share-the-connection.patch,
hbase-10810_v1.patch
While running the IT test from HBASE-10572, we've noticed that the number of
threads jumps to 4K's when CM actions are going on.
Our [~ndimiduk] summarizes the problem quite good:
MultiThreadedReader creates this pool for each HTable:
{code}
ThreadPoolExecutor pool = new ThreadPoolExecutor(1, maxThreads,
keepAliveTime, TimeUnit.SECONDS,
new SynchronousQueueRunnable(),
Threads.newDaemonThreadFactory(htable));
{code}
This comes from the HTable creation
{code}
public HTable(Configuration conf, final TableName tableName)
{code}
As well the javadoc says Recommended.
This is wrong.
In this issue we can change the LTT sub classes to use the shared connection
object and initialize their tables using HConnection.getTable() rather than
new HTable().
This is relevant to trunk as well, but there since there is only one
outstanding RPC per thread, it is not such a big problem.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10810) LoadTestTool should share the connection and connection pool
[
https://issues.apache.org/jira/browse/HBASE-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998271#comment-13998271
]
Devaraj Das commented on HBASE-10810:
-
+1
LoadTestTool should share the connection and connection pool
Key: HBASE-10810
URL: https://issues.apache.org/jira/browse/HBASE-10810
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: hbase-10070
Attachments: hbase-10810_v1.patch
While running the IT test from HBASE-10572, we've noticed that the number of
threads jumps to 4K's when CM actions are going on.
Our [~ndimiduk] summarizes the problem quite good:
MultiThreadedReader creates this pool for each HTable:
{code}
ThreadPoolExecutor pool = new ThreadPoolExecutor(1, maxThreads,
keepAliveTime, TimeUnit.SECONDS,
new SynchronousQueueRunnable(),
Threads.newDaemonThreadFactory(htable));
{code}
This comes from the HTable creation
{code}
public HTable(Configuration conf, final TableName tableName)
{code}
As well the javadoc says Recommended.
This is wrong.
In this issue we can change the LTT sub classes to use the shared connection
object and initialize their tables using HConnection.getTable() rather than
new HTable().
This is relevant to trunk as well, but there since there is only one
outstanding RPC per thread, it is not such a big problem.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10810) LoadTestTool should share the connection and connection pool
[
https://issues.apache.org/jira/browse/HBASE-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13986255#comment-13986255
]
Enis Soztutar commented on HBASE-10810:
---
Thanks Nick for the review.
bq. What's unclear to me is how the 100 reader threads will contend for the (up
to) 256 threads in the shared pool used by all HTables for the connection
In case this becomes a problem, we can increase the default pool side that the
LTT connection is using.
LoadTestTool should share the connection and connection pool
Key: HBASE-10810
URL: https://issues.apache.org/jira/browse/HBASE-10810
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: hbase-10070
Attachments: hbase-10810_v1.patch
While running the IT test from HBASE-10572, we've noticed that the number of
threads jumps to 4K's when CM actions are going on.
Our [~ndimiduk] summarizes the problem quite good:
MultiThreadedReader creates this pool for each HTable:
{code}
ThreadPoolExecutor pool = new ThreadPoolExecutor(1, maxThreads,
keepAliveTime, TimeUnit.SECONDS,
new SynchronousQueueRunnable(),
Threads.newDaemonThreadFactory(htable));
{code}
This comes from the HTable creation
{code}
public HTable(Configuration conf, final TableName tableName)
{code}
As well the javadoc says Recommended.
This is wrong.
In this issue we can change the LTT sub classes to use the shared connection
object and initialize their tables using HConnection.getTable() rather than
new HTable().
This is relevant to trunk as well, but there since there is only one
outstanding RPC per thread, it is not such a big problem.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10810) LoadTestTool should share the connection and connection pool
[
https://issues.apache.org/jira/browse/HBASE-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13943764#comment-13943764
]
Nick Dimiduk commented on HBASE-10810:
--
[~nkeywal] and I came to the same conclusion simultaneously. I think the test
should be using a single pool for all tables it creates, instead of using this
HTable constructor with it's managed pool. In general, I think the HTable
constructor is dangerous for anything but a one-off and should be removed (see
HBASE-9117).
LoadTestTool should share the connection and connection pool
Key: HBASE-10810
URL: https://issues.apache.org/jira/browse/HBASE-10810
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: hbase-10070
While running the IT test from HBASE-10572, we've noticed that the number of
threads jumps to 4K's when CM actions are going on.
Our [~ndimiduk] summarizes the problem quite good:
MultiThreadedReader creates this pool for each HTable:
{code}
ThreadPoolExecutor pool = new ThreadPoolExecutor(1, maxThreads,
keepAliveTime, TimeUnit.SECONDS,
new SynchronousQueueRunnable(),
Threads.newDaemonThreadFactory(htable));
{code}
This comes from the HTable creation
{code}
public HTable(Configuration conf, final TableName tableName)
{code}
As well the javadoc says Recommended.
This is wrong.
In this issue we can change the LTT sub classes to use the shared connection
object and initialize their tables using HConnection.getTable() rather than
new HTable().
This is relevant to trunk as well, but there since there is only one
outstanding RPC per thread, it is not such a big problem.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10810) LoadTestTool should share the connection and connection pool
[
https://issues.apache.org/jira/browse/HBASE-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13943810#comment-13943810
]
Nick Dimiduk commented on HBASE-10810:
--
+1 this should resolve the problem of creating so many pools with so many
threads in total.
What's unclear to me is how the 100 reader threads will contend for the (up to)
256 threads in the shared pool used by all HTables for the connection. I need a
more thorough understanding of AsyncProcess...
LoadTestTool should share the connection and connection pool
Key: HBASE-10810
URL: https://issues.apache.org/jira/browse/HBASE-10810
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: hbase-10070
Attachments: hbase-10810_v1.patch
While running the IT test from HBASE-10572, we've noticed that the number of
threads jumps to 4K's when CM actions are going on.
Our [~ndimiduk] summarizes the problem quite good:
MultiThreadedReader creates this pool for each HTable:
{code}
ThreadPoolExecutor pool = new ThreadPoolExecutor(1, maxThreads,
keepAliveTime, TimeUnit.SECONDS,
new SynchronousQueueRunnable(),
Threads.newDaemonThreadFactory(htable));
{code}
This comes from the HTable creation
{code}
public HTable(Configuration conf, final TableName tableName)
{code}
As well the javadoc says Recommended.
This is wrong.
In this issue we can change the LTT sub classes to use the shared connection
object and initialize their tables using HConnection.getTable() rather than
new HTable().
This is relevant to trunk as well, but there since there is only one
outstanding RPC per thread, it is not such a big problem.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
