[
https://issues.apache.org/jira/browse/HBASE-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424308#comment-13424308
]
nkeywal commented on HBASE-6364:
--------------------------------
Thanks Suraj. So it seems that we have the root cause. You're right, on
ipc.Client it's less likely to occur. Only the NN could have the issue I think,
and as the hot failover is not yet deployed in production, it has not been an
issue so far. I will ping Todd on this to see what he thinks.
The only strange thing is that when tested on the 0.96 + minicluster, I have
the issue but many threads seem to be able to 'escape', i.e. manage to do the
'addCall' between two blocking calls to setupIOstreams. But it could be a pure
multithreading hazard. And I haven't been able to find a better cause than this.
I will propose a patch, likely end of next week. Suraj, If you have some time
available to validate it on your env, it would be great.
> Powering down the server host holding the .META. table causes HBase Client to
> take excessively long to recover and connect to reassigned .META. table
> -----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-6364
> URL: https://issues.apache.org/jira/browse/HBASE-6364
> Project: HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.90.6, 0.92.1, 0.94.0
> Reporter: Suraj Varma
> Assignee: nkeywal
> Labels: client
> Attachments: stacktrace.txt
>
>
> When a server host with a Region Server holding the .META. table is powered
> down on a live cluster, while the HBase cluster itself detects and reassigns
> the .META. table, connected HBase Client's take an excessively long time to
> detect this and re-discover the reassigned .META.
> Workaround: Decrease the ipc.socket.timeout on HBase Client side to a low
> value (default is 20s leading to 35 minute recovery time; we were able to get
> acceptable results with 100ms getting a 3 minute recovery)
> This was found during some hardware failure testing scenarios.
> Test Case:
> 1) Apply load via client app on HBase cluster for several minutes
> 2) Power down the region server holding the .META. server (i.e. power off ...
> and keep it off)
> 3) Measure how long it takes for cluster to reassign META table and for
> client threads to re-lookup and re-orient to the lesser cluster (minus the RS
> and DN on that host).
> Observation:
> 1) Client threads spike up to maxThreads size ... and take over 35 mins to
> recover (i.e. for the thread count to go back to normal) - no client calls
> are serviced - they just back up on a synchronized method (see #2 below)
> 2) All the client app threads queue up behind the
> oahh.ipc.HBaseClient#setupIOStreams method http://tinyurl.com/7js53dj
> After taking several thread dumps we found that the thread within this
> synchronized method was blocked on NetUtils.connect(this.socket,
> remoteId.getAddress(), getSocketTimeout(conf));
> The client thread that gets the synchronized lock would try to connect to the
> dead RS (till socket times out after 20s), retries, and then the next thread
> gets in and so forth in a serial manner.
> Workaround:
> -------------------
> Default ipc.socket.timeout is set to 20s. We dropped this to a low number
> (1000 ms, 100 ms, etc) on the client side hbase-site.xml. With this setting,
> the client threads recovered in a couple of minutes by failing fast and
> re-discovering the .META. table on a reassigned RS.
> Assumption: This ipc.socket.timeout is only ever used during the initial
> "HConnection" setup via the NetUtils.connect and should only ever be used
> when connectivity to a region server is lost and needs to be re-established.
> i.e it does not affect the normal "RPC" actiivity as this is just the connect
> timeout.
> During RS GC periods, any _new_ clients trying to connect will fail and will
> require .META. table re-lookups.
> This above timeout workaround is only for the HBase client side.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
