Looking at Apache 0.98 code, you can follow the example in the class
javadoc (line 144 of HConnectionManager.java):
* HTableInterface table = connection.getTable("table1");
* try {
* // Use the table as needed, for a single operation and a single thread
* } finally {
* table.close();
* connection.close();
* }
Cheers
On Thu, Oct 16, 2014 at 9:03 PM, Fengyun RAO <[email protected]> wrote:
> Thanks, Ted,
>
> We use CDH 5.1 and the HBase version is 0.98.1-cdh5.1.0, in which the
> javadoc of HConnectionManager.java still recommends shutdown hook.
>
> I look into val table = Util.Connection.getTable("user"), and find it
> didn't invoke
>
> public HTable(Configuration conf, final byte[] tableName, final
> ExecutorService pool)
>
> but
>
> public HTable(TableName tableName, final HConnection connection,
> final ExecutorService pool) throws IOException {
> if (connection == null || connection.isClosed()) {
> throw new IllegalArgumentException("Connection is null or closed.");
> }
> this.tableName = tableName;
> this.cleanupPoolOnClose = this.cleanupConnectionOnClose = false;
> this.connection = connection;
> this.configuration = connection.getConfiguration();
> this.pool = pool;
>
> this.finishSetup();
> }
>
> in which cleanupConnectionOnClose is false
>
> 2014-10-16 22:51 GMT+08:00 Ted Yu <[email protected]>:
>
>> Which hbase release are you using ?
>>
>> Let me refer to 0.94 code hbase.
>>
>> Take a look at the following method
>> in src/main/java/org/apache/hadoop/hbase/client/HTable.java :
>>
>> public void close() throws IOException {
>> ...
>> if (cleanupConnectionOnClose) {
>> if (this.connection != null) {
>> this.connection.close();
>>
>> When Connection.getTable() is called, the following is invoked:
>> public HTable(Configuration conf, final byte[] tableName, final
>> ExecutorService pool)
>> which sets cleanupConnectionOnClose to true.
>>
>> w.r.t. javadoc, the paragraph on shutdown hook is
>> in HConnectionManager.java of 0.94
>> You don't need to use shutdown hook for 0.94+
>>
>> Cheers
>>
>> On Wed, Oct 15, 2014 at 11:41 PM, Fengyun RAO <[email protected]>
>> wrote:
>>
>>> I may have misunderstood your point.
>>>
>>> val result = rdd.map(line => {
>>> val table = Util.Connection.getTable("user")
>>> ...
>>> table.close()
>>> }
>>>
>>> Did you mean this is enough, and there’s no need to call
>>> Util.Connection.close(),
>>> or HConnectionManager.deleteAllConnections()?
>>>
>>> Where is the documentation that statesHconnectionManager would release
>>> underlying connection automatically?
>>> If that’s true, maybe the Javadoc which recommends a shutdown hook needs
>>> update
>>>
>>>
>>> 2014-10-16 14:20 GMT+08:00 Fengyun RAO <[email protected]>:
>>>
>>>> Thanks, Ted.
>>>> Util.Connection.close() should be called only once, so it can NOT be
>>>> in a map function
>>>>
>>>> val result = rdd.map(line => {
>>>> val table = Util.Connection.getTable("user")
>>>> ...
>>>> Util.Connection.close()
>>>> }
>>>>
>>>> As you mentioned:
>>>>
>>>> Calling table.close() is the recommended approach.
>>>> HConnectionManager does reference counting. When all references to the
>>>> underlying connection are gone, connection would be released.
>>>>
>>>> Yes, we should call table.close(), but it won’t remove HConnection in
>>>> HConnectionManager which is a HConnection pool.
>>>> As I look into the HconnectionManager Javadoc, it seems I have to
>>>> implement a shutdown hook
>>>>
>>>> * <p>Cleanup used to be done inside in a shutdown hook. On startup we'd
>>>> * register a shutdown hook that called {@link #deleteAllConnections()}
>>>> * on its way out but the order in which shutdown hooks run is not defined
>>>> so
>>>> * were problematic for clients of HConnection that wanted to register
>>>> their
>>>> * own shutdown hooks so we removed ours though this shifts the onus for
>>>> * cleanup to the client.
>>>>
>>>>
>>>>
>>>> 2014-10-15 22:31 GMT+08:00 Ted Yu <[email protected]>:
>>>>
>>>>> Pardon me - there was typo in previous email.
>>>>>
>>>>> Calling table.close() is the recommended approach.
>>>>> HConnectionManager does reference counting. When all references to the
>>>>> underlying connection are gone, connection would be released.
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Wed, Oct 15, 2014 at 7:13 AM, Ted Yu <[email protected]> wrote:
>>>>>
>>>>>> Have you tried the following ?
>>>>>>
>>>>>> val result = rdd.map(line => { val table =
>>>>>> Util.Connection.getTable("user")
>>>>>> ...
>>>>>> Util.Connection.close() }
>>>>>>
>>>>>> On Wed, Oct 15, 2014 at 6:09 AM, Fengyun RAO <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> In order to share an HBase connection pool, we create an object
>>>>>>>
>>>>>>> Object Util {
>>>>>>> val HBaseConf = HBaseConfiguration.create
>>>>>>> val Connection= HConnectionManager.createConnection(HBaseConf)
>>>>>>> }
>>>>>>>
>>>>>>> which would be shared among tasks on the same executor. e.g.
>>>>>>>
>>>>>>> val result = rdd.map(line => {
>>>>>>> val table = Util.Connection.getTable("user")
>>>>>>> ...
>>>>>>> }
>>>>>>>
>>>>>>> However, we don’t how to close the Util.Connection.
>>>>>>> If we write Util.Connection.close() in the main function,
>>>>>>> it’ll only run on the driver, not the executor.
>>>>>>>
>>>>>>> So, How to make sure every Connection closed before exist?
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
