Which hbase release are you using ?
Let me refer to 0.94 code hbase.
Take a look at the following method
in src/main/java/org/apache/hadoop/hbase/client/HTable.java :
public void close() throws IOException {
...
if (cleanupConnectionOnClose) {
if (this.connection != null) {
this.connection.close();
When Connection.getTable() is called, the following is invoked:
public HTable(Configuration conf, final byte[] tableName, final
ExecutorService pool)
which sets cleanupConnectionOnClose to true.
w.r.t. javadoc, the paragraph on shutdown hook is
in HConnectionManager.java of 0.94
You don't need to use shutdown hook for 0.94+
Cheers
On Wed, Oct 15, 2014 at 11:41 PM, Fengyun RAO <raofeng...@gmail.com> wrote:
> I may have misunderstood your point.
>
> val result = rdd.map(line => {
> val table = Util.Connection.getTable("user")
> ...
> table.close()
> }
>
> Did you mean this is enough, and there’s no need to call
> Util.Connection.close(),
> or HConnectionManager.deleteAllConnections()?
>
> Where is the documentation that statesHconnectionManager would release
> underlying connection automatically?
> If that’s true, maybe the Javadoc which recommends a shutdown hook needs
> update
>
>
> 2014-10-16 14:20 GMT+08:00 Fengyun RAO <raofeng...@gmail.com>:
>
>> Thanks, Ted.
>> Util.Connection.close() should be called only once, so it can NOT be in
>> a map function
>>
>> val result = rdd.map(line => {
>> val table = Util.Connection.getTable("user")
>> ...
>> Util.Connection.close()
>> }
>>
>> As you mentioned:
>>
>> Calling table.close() is the recommended approach.
>> HConnectionManager does reference counting. When all references to the
>> underlying connection are gone, connection would be released.
>>
>> Yes, we should call table.close(), but it won’t remove HConnection in
>> HConnectionManager which is a HConnection pool.
>> As I look into the HconnectionManager Javadoc, it seems I have to
>> implement a shutdown hook
>>
>> * <p>Cleanup used to be done inside in a shutdown hook. On startup we'd
>> * register a shutdown hook that called {@link #deleteAllConnections()}
>> * on its way out but the order in which shutdown hooks run is not defined so
>> * were problematic for clients of HConnection that wanted to register their
>> * own shutdown hooks so we removed ours though this shifts the onus for
>> * cleanup to the client.
>>
>>
>>
>> 2014-10-15 22:31 GMT+08:00 Ted Yu <yuzhih...@gmail.com>:
>>
>>> Pardon me - there was typo in previous email.
>>>
>>> Calling table.close() is the recommended approach.
>>> HConnectionManager does reference counting. When all references to the
>>> underlying connection are gone, connection would be released.
>>>
>>> Cheers
>>>
>>> On Wed, Oct 15, 2014 at 7:13 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>>>
>>>> Have you tried the following ?
>>>>
>>>> val result = rdd.map(line => { val table = Util.Connection.getTable("user")
>>>> ...
>>>> Util.Connection.close() }
>>>>
>>>> On Wed, Oct 15, 2014 at 6:09 AM, Fengyun RAO <raofeng...@gmail.com>
>>>> wrote:
>>>>
>>>>> In order to share an HBase connection pool, we create an object
>>>>>
>>>>> Object Util {
>>>>> val HBaseConf = HBaseConfiguration.create
>>>>> val Connection= HConnectionManager.createConnection(HBaseConf)
>>>>> }
>>>>>
>>>>> which would be shared among tasks on the same executor. e.g.
>>>>>
>>>>> val result = rdd.map(line => {
>>>>> val table = Util.Connection.getTable("user")
>>>>> ...
>>>>> }
>>>>>
>>>>> However, we don’t how to close the Util.Connection.
>>>>> If we write Util.Connection.close() in the main function,
>>>>> it’ll only run on the driver, not the executor.
>>>>>
>>>>> So, How to make sure every Connection closed before exist?
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
