[
https://issues.apache.org/jira/browse/HADOOP-7472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067812#comment-13067812
]
Kihwal Lee commented on HADOOP-7472:
------------------------------------
Yes, it is a hard problem and I hope it is addressed in the HA work. Whereas
this JIRA is for improving the system behavior during certain class of
failures, specifically for the 0.20 lines, but not necessarily for achieving a
very high cluster availability through automatic failover within the cluster.
For the failure scenario described in this JIRA, the recovery involves manual
steps today. The consensus/fencing is forced manually and physically, and the
failure detection latencies in Hadoop can be relatively long (currently
infinite in some cases). When the cluster is restarted, the system loses a lot
of state and kindly asking users to resubmit their jobs is not exactly we are
thrilled about doing.
Here is what I think we can do (for 0.20):
- Improve the failure detection in software for the scenario described above.
If the NN became unavailable and would come back on the same node, there will
be no false alarms. Easy to do.
- Improve some components to do a smart recovery when such a failure is
detected. For example, it will be nice if this can be done for JT so that it
doesn't lose its state.
I wouldn't call it HA, but it still can improve MTTR and reduce the impact on
customers.
> RPC client should deal with the IP address changes
> --------------------------------------------------
>
> Key: HADOOP-7472
> URL: https://issues.apache.org/jira/browse/HADOOP-7472
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
> Affects Versions: 0.20.205.0
> Reporter: Kihwal Lee
> Assignee: Kihwal Lee
> Priority: Minor
>
> The current RPC client implementation and the client-side callers assume that
> the hostname-address mappings of servers never change. The resolved address
> is stored in an immutable InetSocketAddress object above/outside RPC, and the
> reconnect logic in the RPC Connection implementation also trusts the resolved
> address that was passed down.
> If the NN suffers a failure that requires migration, it may be started on a
> different node with a different IP address. In this case, even if the
> name-address mapping is updated in DNS, the cluster is stuck trying old
> address until the whole cluster is restarted.
> The RPC client-side should detect this situation and exit or try to recover.
> Updating ConnectionId within the Client implementation may get the system
> work for the moment, there always is a risk of the cached address:port become
> connectable again unintentionally. The real solution will be notifying upper
> layer of the address change so that they can re-resolve and retry or
> re-architecture the system as discussed in HDFS-34.
> For 0.20 lines, some type of compromise may be acceptable. For example, raise
> a custom exception for some well-defined high-impact upper layer to do
> re-resolve/retry, while other will have to restart. For TRUNK, the HA work
> will most likely determine what needs to be done. So this Jira won't cover
> the solutions for TRUNK.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
