[jira] [Commented] (HBASE-8218) pass HTable as a parameter to method of AggregationClient
[ https://issues.apache.org/jira/browse/HBASE-8218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13629845#comment-13629845 ] cuijianwei commented on HBASE-8218: --- Thanks for your concern. I make a new patch to introduce a 'close' flag. I think it is more reasonable to wait all executing tasks finish before returning from close(), so that I invoke 'awaitTermination' after 'shutdown'。I invoke 'close()' in corresponding unit test of 'max(...)', it works locally. pass HTable as a parameter to method of AggregationClient - Key: HBASE-8218 URL: https://issues.apache.org/jira/browse/HBASE-8218 Project: HBase Issue Type: Improvement Components: Client, Coprocessors Affects Versions: 0.94.3 Reporter: cuijianwei Attachments: HBASE-8218-0.94.3-v1.txt, HBASE-8218-0.94.3-v2.txt In AggregationClient, methods such as max(...), min(...) pass 'tableName' as a parameter, then a HTable will be created in the method, before the method return, the created HTable will be closed. The process above may be heavy because each call must create and close a HTable. The situation becomes worse when there is only one thread access HBase using AggregationClient. The underly HConnection of created HTable will also be created and then closed each time when we invoke these method because no other HTables using the HConnection. This operation is heavy. Therefore, can we add another group of methods which pass HTable or HTablePool as a parameter to methods defined in AggregationClient? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8218) pass HTable as a parameter to method of AggregationClient
[
https://issues.apache.org/jira/browse/HBASE-8218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13629850#comment-13629850
]
Ted Yu commented on HBASE-8218:
---
Thanks for the quick response.
{code}
+ this.threadPool.awaitTermination(Long.MAX_VALUE, TimeUnit.SECONDS);
{code}
Is the above waiting period too long ? How about waiting for 10 min ?
If awaitTermination() returns false, you should call this method:
http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/ExecutorService.html#shutdownNow()
Please generate next patch from trunk so that hadoop QA can test the patch.
pass HTable as a parameter to method of AggregationClient
-
Key: HBASE-8218
URL: https://issues.apache.org/jira/browse/HBASE-8218
Project: HBase
Issue Type: Improvement
Components: Client, Coprocessors
Affects Versions: 0.94.3
Reporter: cuijianwei
Attachments: HBASE-8218-0.94.3-v1.txt, HBASE-8218-0.94.3-v2.txt
In AggregationClient, methods such as max(...), min(...) pass 'tableName' as
a parameter, then a HTable will be created in the method, before the method
return, the created HTable will be closed.
The process above may be heavy because each call must create and close a
HTable. The situation becomes worse when there is only one thread access
HBase using AggregationClient. The underly HConnection of created HTable will
also be created and then closed each time when we invoke these method because
no other HTables using the HConnection. This operation is heavy. Therefore,
can we add another group of methods which pass HTable or HTablePool as a
parameter to methods defined in AggregationClient?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8218) pass HTable as a parameter to method of AggregationClient
[
https://issues.apache.org/jira/browse/HBASE-8218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13629748#comment-13629748
]
cuijianwei commented on HBASE-8218:
---
I add a patch which using HConnection and ThreadPool to create HTable in
AggregationClient. I'am not sure whether it's a reasonable way to alleviate the
problem. Consequently, I only use HConnection and ThreadPool to create HTable
in method {code} public R, S R max(final byte[] tableName, final
ColumnInterpreterR, S ci, final Scan scan) throws Throwable {code} 。I run
TestAggregationClient.java locally and it passes. I can provide a more
completed patch after understand your feedback.
pass HTable as a parameter to method of AggregationClient
-
Key: HBASE-8218
URL: https://issues.apache.org/jira/browse/HBASE-8218
Project: HBase
Issue Type: Improvement
Components: Client, Coprocessors
Affects Versions: 0.94.3
Reporter: cuijianwei
Attachments: HBASE-8218-0.94.3-v1.txt
In AggregationClient, methods such as max(...), min(...) pass 'tableName' as
a parameter, then a HTable will be created in the method, before the method
return, the created HTable will be closed.
The process above may be heavy because each call must create and close a
HTable. The situation becomes worse when there is only one thread access
HBase using AggregationClient. The underly HConnection of created HTable will
also be created and then closed each time when we invoke these method because
no other HTables using the HConnection. This operation is heavy. Therefore,
can we add another group of methods which pass HTable or HTablePool as a
parameter to methods defined in AggregationClient?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8218) pass HTable as a parameter to method of AggregationClient
[
https://issues.apache.org/jira/browse/HBASE-8218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13629754#comment-13629754
]
Ted Yu commented on HBASE-8218:
---
{code}
+ public void close() throws IOException {
+this.threadPool.shutdown();
{code}
Can you introduce a flag so that we know whether AggregationClient has been
closed ?
Do we need to wait for the shutdown to finish before returning from close() ?
pass HTable as a parameter to method of AggregationClient
-
Key: HBASE-8218
URL: https://issues.apache.org/jira/browse/HBASE-8218
Project: HBase
Issue Type: Improvement
Components: Client, Coprocessors
Affects Versions: 0.94.3
Reporter: cuijianwei
Attachments: HBASE-8218-0.94.3-v1.txt
In AggregationClient, methods such as max(...), min(...) pass 'tableName' as
a parameter, then a HTable will be created in the method, before the method
return, the created HTable will be closed.
The process above may be heavy because each call must create and close a
HTable. The situation becomes worse when there is only one thread access
HBase using AggregationClient. The underly HConnection of created HTable will
also be created and then closed each time when we invoke these method because
no other HTables using the HConnection. This operation is heavy. Therefore,
can we add another group of methods which pass HTable or HTablePool as a
parameter to methods defined in AggregationClient?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8218) pass HTable as a parameter to method of AggregationClient
[
https://issues.apache.org/jira/browse/HBASE-8218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13625093#comment-13625093
]
cuijianwei commented on HBASE-8218:
---
Maybe, we can add HConnection and ExecutorService as members of
AggregationClient. Then, we can invoke constructor: {code} public
HTable(final byte[] tableName, final HConnection connection, final
ExecutorService pool) throws IOException {code} to construct a HTable in each
method. The current signatures won't change, and this constructor of HTable is
lighter because we don't need to construct new HConnection and ExecutorService.
However, we need a close() method for user to close HConnection and
ExecutorService.
pass HTable as a parameter to method of AggregationClient
-
Key: HBASE-8218
URL: https://issues.apache.org/jira/browse/HBASE-8218
Project: HBase
Issue Type: Improvement
Components: Client, Coprocessors
Affects Versions: 0.94.3
Reporter: cuijianwei
In AggregationClient, methods such as max(...), min(...) pass 'tableName' as
a parameter, then a HTable will be created in the method, before the method
return, the created HTable will be closed.
The process above may be heavy because each call must create and close a
HTable. The situation becomes worse when there is only one thread access
HBase using AggregationClient. The underly HConnection of created HTable will
also be created and then closed each time when we invoke these method because
no other HTables using the HConnection. This operation is heavy. Therefore,
can we add another group of methods which pass HTable or HTablePool as a
parameter to methods defined in AggregationClient?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8218) pass HTable as a parameter to method of AggregationClient
[ https://issues.apache.org/jira/browse/HBASE-8218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618266#comment-13618266 ] cuijianwei commented on HBASE-8218: --- As HTablePool is being deprecated, is there alternative HTable pool to replace the old one? pass HTable as a parameter to method of AggregationClient - Key: HBASE-8218 URL: https://issues.apache.org/jira/browse/HBASE-8218 Project: HBase Issue Type: Improvement Components: Client, Coprocessors Affects Versions: 0.94.3 Reporter: cuijianwei In AggregationClient, methods such as max(...), min(...) pass 'tableName' as a parameter, then a HTable will be created in the method, before the method return, the created HTable will be closed. The process above may be heavy because each call must create and close a HTable. The situation becomes worse when there is only one thread access HBase using AggregationClient. The underly HConnection of created HTable will also be created and then closed each time when we invoke these method because no other HTables using the HConnection. This operation is heavy. Therefore, can we add another group of methods which pass HTable or HTablePool as a parameter to methods defined in AggregationClient? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8218) pass HTable as a parameter to method of AggregationClient
[ https://issues.apache.org/jira/browse/HBASE-8218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618360#comment-13618360 ] cuijianwei commented on HBASE-8218: --- Will we remove HTablePool in 0.95 as discussed in [HBASE-6651|https://issues.apache.org/jira/browse/HBASE-6651]? pass HTable as a parameter to method of AggregationClient - Key: HBASE-8218 URL: https://issues.apache.org/jira/browse/HBASE-8218 Project: HBase Issue Type: Improvement Components: Client, Coprocessors Affects Versions: 0.94.3 Reporter: cuijianwei In AggregationClient, methods such as max(...), min(...) pass 'tableName' as a parameter, then a HTable will be created in the method, before the method return, the created HTable will be closed. The process above may be heavy because each call must create and close a HTable. The situation becomes worse when there is only one thread access HBase using AggregationClient. The underly HConnection of created HTable will also be created and then closed each time when we invoke these method because no other HTables using the HConnection. This operation is heavy. Therefore, can we add another group of methods which pass HTable or HTablePool as a parameter to methods defined in AggregationClient? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8218) pass HTable as a parameter to method of AggregationClient
[ https://issues.apache.org/jira/browse/HBASE-8218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13617260#comment-13617260 ] Ted Yu commented on HBASE-8218: --- How about adding method setHTable(HTable) ? The passed HTable would be saved in a field. This keeps APIs for other methods same as current. AggregationClient would use the passed HTable if it is non-null. pass HTable as a parameter to method of AggregationClient - Key: HBASE-8218 URL: https://issues.apache.org/jira/browse/HBASE-8218 Project: HBase Issue Type: Improvement Components: Client, Coprocessors Affects Versions: 0.94.3 Reporter: cuijianwei In AggregationClient, methods such as max(...), min(...) pass 'tableName' as a parameter, then a HTable will be created in the method, before the method return, the created HTable will be closed. The process above may be heavy because each call must create and close a HTable. The situation becomes worse when there is only one thread access HBase using AggregationClient. The underly HConnection of created HTable will also be created and then closed each time when we invoke these method because no other HTables using the HConnection. This operation is heavy. Therefore, can we add another group of methods which pass HTable or HTablePool as a parameter to methods defined in AggregationClient? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8218) pass HTable as a parameter to method of AggregationClient
[
https://issues.apache.org/jira/browse/HBASE-8218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13617271#comment-13617271
]
Jean-Marc Spaggiari commented on HBASE-8218:
[~te...@apache.org] I'm not sure this is a clean solution.
This is the signature of AggregationClient.max():
public R, S R max(final byte[] tableName, final ColumnInterpreterR, S ci,
final Scan scan)
If we add setHTable(HTable) to ignore the tableName parameter, many will miss
that might wonder why tableName is not used. Is it nos better duplicte the
methods and make one calling the other one?
Something like that where we keep the current signature and introduce the new
option too?
{code}
/**
* It gives the maximum value of a column for a given column family for the
* given range. In case qualifier is null, a max of all values for the given
* family is returned.
* @param tableName
* @param ci
* @param scan
* @return max val R
* @throws Throwable
* The caller is supposed to handle the exception as they are thrown
*propagated to it.
*/
public R, S R max(final byte[] tableName, final ColumnInterpreterR, S ci,
final Scan scan) throws Throwable {
HTable table = null;
try {
table = new HTable(conf, tableName);
return max(table, ci, scan);
} finally {
if (table != null) {
table.close();
}
}
}
/**
* It gives the maximum value of a column for a given column family for the
* given range. In case qualifier is null, a max of all values for the given
* family is returned.
* @param table
* @param ci
* @param scan
* @return max val R
* @throws Throwable
* The caller is supposed to handle the exception as they are thrown
*propagated to it.
*/
public R, S R max(final HTable table, final ColumnInterpreterR, S ci,
final Scan scan) throws Throwable {
validateParameters(scan);
class MaxCallBack implements Batch.CallbackR {
R max = null;
R getMax() {
return max;
}
@Override
public synchronized void update(byte[] region, byte[] row, R result) {
max = (max == null || (result != null ci.compare(max, result) 0))
? result : max;
}
}
MaxCallBack aMaxCallBack = new MaxCallBack();
table.coprocessorExec(AggregateProtocol.class, scan.getStartRow(),
scan.getStopRow(), new Batch.CallAggregateProtocol, R() {
@Override
public R call(AggregateProtocol instance) throws IOException {
return instance.getMax(ci, scan);
}
}, aMaxCallBack);
return aMaxCallBack.getMax();
}
{code}
pass HTable as a parameter to method of AggregationClient
-
Key: HBASE-8218
URL: https://issues.apache.org/jira/browse/HBASE-8218
Project: HBase
Issue Type: Improvement
Components: Client, Coprocessors
Affects Versions: 0.94.3
Reporter: cuijianwei
In AggregationClient, methods such as max(...), min(...) pass 'tableName' as
a parameter, then a HTable will be created in the method, before the method
return, the created HTable will be closed.
The process above may be heavy because each call must create and close a
HTable. The situation becomes worse when there is only one thread access
HBase using AggregationClient. The underly HConnection of created HTable will
also be created and then closed each time when we invoke these method because
no other HTables using the HConnection. This operation is heavy. Therefore,
can we add another group of methods which pass HTable or HTablePool as a
parameter to methods defined in AggregationClient?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8218) pass HTable as a parameter to method of AggregationClient
[ https://issues.apache.org/jira/browse/HBASE-8218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13617276#comment-13617276 ] Ted Yu commented on HBASE-8218: --- I forgot to say that when tableName parameter points to the same table as HTable field does, we reuse HTable field. Otherwise we keep the current behavior. This allows the same AggregationClient to be used with different tables. pass HTable as a parameter to method of AggregationClient - Key: HBASE-8218 URL: https://issues.apache.org/jira/browse/HBASE-8218 Project: HBase Issue Type: Improvement Components: Client, Coprocessors Affects Versions: 0.94.3 Reporter: cuijianwei In AggregationClient, methods such as max(...), min(...) pass 'tableName' as a parameter, then a HTable will be created in the method, before the method return, the created HTable will be closed. The process above may be heavy because each call must create and close a HTable. The situation becomes worse when there is only one thread access HBase using AggregationClient. The underly HConnection of created HTable will also be created and then closed each time when we invoke these method because no other HTables using the HConnection. This operation is heavy. Therefore, can we add another group of methods which pass HTable or HTablePool as a parameter to methods defined in AggregationClient? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8218) pass HTable as a parameter to method of AggregationClient
[ https://issues.apache.org/jira/browse/HBASE-8218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13617376#comment-13617376 ] cuijianwei commented on HBASE-8218: --- I think setHTable(HTable) could alleviate the problem. The user is tend to access the same table when using AggregationClient. Maybe we can add HTablePool as a member of AggregationClient and construct the tablePool when constructing AggregationClient. Then, for methods in AggregationClient, we can using HTablePool to create and pool opened HTable. This could be helpful to keep the current signature. However, there must be a close() method to close HTablePool, and user must invoke close(...) after using AggregationClient. pass HTable as a parameter to method of AggregationClient - Key: HBASE-8218 URL: https://issues.apache.org/jira/browse/HBASE-8218 Project: HBase Issue Type: Improvement Components: Client, Coprocessors Affects Versions: 0.94.3 Reporter: cuijianwei In AggregationClient, methods such as max(...), min(...) pass 'tableName' as a parameter, then a HTable will be created in the method, before the method return, the created HTable will be closed. The process above may be heavy because each call must create and close a HTable. The situation becomes worse when there is only one thread access HBase using AggregationClient. The underly HConnection of created HTable will also be created and then closed each time when we invoke these method because no other HTables using the HConnection. This operation is heavy. Therefore, can we add another group of methods which pass HTable or HTablePool as a parameter to methods defined in AggregationClient? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8218) pass HTable as a parameter to method of AggregationClient
[ https://issues.apache.org/jira/browse/HBASE-8218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13617387#comment-13617387 ] Ted Yu commented on HBASE-8218: --- HTablePool is being deprecated. We'd better not introduce it into AggregationClient. When user application is done with using AggregationClient, setHTable(null) can be called where we call close on the previously passed HTable instance. pass HTable as a parameter to method of AggregationClient - Key: HBASE-8218 URL: https://issues.apache.org/jira/browse/HBASE-8218 Project: HBase Issue Type: Improvement Components: Client, Coprocessors Affects Versions: 0.94.3 Reporter: cuijianwei In AggregationClient, methods such as max(...), min(...) pass 'tableName' as a parameter, then a HTable will be created in the method, before the method return, the created HTable will be closed. The process above may be heavy because each call must create and close a HTable. The situation becomes worse when there is only one thread access HBase using AggregationClient. The underly HConnection of created HTable will also be created and then closed each time when we invoke these method because no other HTables using the HConnection. This operation is heavy. Therefore, can we add another group of methods which pass HTable or HTablePool as a parameter to methods defined in AggregationClient? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8218) pass HTable as a parameter to method of AggregationClient
[ https://issues.apache.org/jira/browse/HBASE-8218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13617492#comment-13617492 ] Himanshu Vashishtha commented on HBASE-8218: Imo, what JM suggest is better. It keeps the current behavior while also adding what this jira requires (and also using multiple tables). pass HTable as a parameter to method of AggregationClient - Key: HBASE-8218 URL: https://issues.apache.org/jira/browse/HBASE-8218 Project: HBase Issue Type: Improvement Components: Client, Coprocessors Affects Versions: 0.94.3 Reporter: cuijianwei In AggregationClient, methods such as max(...), min(...) pass 'tableName' as a parameter, then a HTable will be created in the method, before the method return, the created HTable will be closed. The process above may be heavy because each call must create and close a HTable. The situation becomes worse when there is only one thread access HBase using AggregationClient. The underly HConnection of created HTable will also be created and then closed each time when we invoke these method because no other HTables using the HConnection. This operation is heavy. Therefore, can we add another group of methods which pass HTable or HTablePool as a parameter to methods defined in AggregationClient? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
