HConnectionManager's shutdown hook interferes with client's operations
----------------------------------------------------------------------
Key: HBASE-2952
URL: https://issues.apache.org/jira/browse/HBASE-2952
Project: HBase
Issue Type: Improvement
Components: client
Affects Versions: 0.20.0
Reporter: Prakash Khemani
My HBase client calls incrementColValue() in pairs. If someone kills the client
(SIGINT or SIGTERM) I want my client's increment threads to gracefully exit. If
a thread has already done one of the incrementColValue() then I want that
thread to complete the other incrementColValue() and then exit.
For this purpose I installed my own shutdownHook(). My shitdownHook() thread
'sugnals' all the threads in my process that it is time to exit and then waits
for them to complete.
The problem is that HConnectionManager's shutdownHook thread also runs and
shuts down all connections and IPC threads.
My increment thread keeps waiting to increment and then times out after 240s.
Two problems with this - the incrementColValiue() didn't go through which will
increase the chances of inconsistency in my HBase data. And it too 240s to
exit. I am pasting some of the messages that the client thread outputs while it
tries contact the HBase server.
Signalled. Exiting ...
2010-09-01 12:11:14,769 DEBUG [HCM.shutdownHook]
zookeeper.ZooKeeperWrapper(787):
<localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Closed
connection with ZooKeeper; /hbase/root-region-server
flushing after 7899
2010-09-01 12:11:19,669 DEBUG [Line Processing Thread 0]
client.HConnectionManager$TableServers(903): Cache hit for row <> in tableName
.META.: location server hadoop2205.snc3.facebook.com:60020, location region
name .META.,,1.1028785192
2010-09-01 12:11:19,671 INFO [Line Processing Thread 0]
zookeeper.ZooKeeperWrapper(206): Reconnecting to zookeeper
2010-09-01 12:11:19,671 DEBUG [Line Processing Thread 0]
zookeeper.ZooKeeperWrapper(212):
<localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Connected
to zookeeper again
2010-09-01 12:11:24,679 DEBUG [Line Processing Thread 0]
client.HConnectionManager$TableServers(964): Removed .META.,,1.1028785192 for
tableName=.META. from cache because of content_action_url_metrics,\x080r&
B\xF7\x81_T\x07\x08\x16uOrcom.gigya 429934274290948,99999999999999
2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0]
client.HConnectionManager$TableServers(857): locateRegionInMeta attempt 0 of 4
failed; retrying after sleep of 5000 because: The client is stopped
2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0]
zookeeper.ZooKeeperWrapper(470):
<localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying to
read /hbase/root-region-server
2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0]
zookeeper.ZooKeeperWrapper(489):
<localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Read ZNode
/hbase/root-region-server got 10.26.119.190:60020
2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0]
client.HConnectionManager$TableServers(1116): Root region location changed.
Sleeping.
===
It might be a good idea to only run the HCM shutdown code when all the HTables
referring to it have been closed. That way the client can control when the
shutdown actually happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
