[
https://issues.apache.org/jira/browse/HADOOP-15684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16611063#comment-16611063
]
Rong Tang commented on HADOOP-15684:
------------------------------------
Summary of what this patch does.
The purpose of that call() function is to use a round robin to find the active
NN. if it has trouble in communicating with a remote NN, or confirmed that
remote NN is standby ,then it should try next. the old logic is not, if it has
trouble in communication, it keeps trying the same one, thus, get stuck. From
code perspective, it throw all exceptions exception standby, and the outer
method calls it, will swallow them. That is how it finally tries the same one.
This call() method only throws IO exception. so my change catches the
IOException, and try next NN. [~lukmajercak] and I had discussion before, we
think the later is the right behavior.
the test case covers the behavior change I want to make. that is to try next
NN on exception, instead of trying the same one.
> triggerActiveLogRoll stuck on dead name node, when ConnectTimeoutException
> happens.
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-15684
> URL: https://issues.apache.org/jira/browse/HADOOP-15684
> Project: Hadoop Common
> Issue Type: Bug
> Components: ha
> Affects Versions: 3.0.0-alpha1
> Reporter: Rong Tang
> Assignee: Rong Tang
> Priority: Critical
> Attachments:
> 0001-RollEditLog-try-next-NN-when-exception-happens.patch,
> HADOOP-15684.000.patch, HADOOP-15684.001.patch, HADOOP-15684.002.patch,
> HADOOP-15684.003.patch, HADOOP-15684.004.patch,
> hadoop--rollingUpgrade-SourceMachine001.log
>
>
> When name node call triggerActiveLogRoll, and the cachedActiveProxy is a dead
> name node, it will throws a ConnectTimeoutException, expected behavior is to
> try next NN, but current logic doesn't do so, instead, it keeps trying the
> dead, mistakenly take it as active.
>
> 2018-08-17 10:02:12,001 WARN [Edit log tailer]
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to trigger a
> roll of the active NN
> org.apache.hadoop.net.ConnectTimeoutException: Call From
> SourceMachine001/SourceIP to001 TargetMachine001.ap.gbl:8020 failed on socket
> timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000
> millis timeout
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:298)
>
> C:\Users\rotang>ping TargetMachine001
> Pinging TargetMachine001[TargetIP001] with 32 bytes of data:
> Request timed out.
> Request timed out.
> Request timed out.
> Request timed out.
> Attachment is a log file saying how it repeatedly retries a dead name node,
> and a fix patch.
> I replaced the actual machine name/ip as SourceMachine001/SourceIP001 and
> TargetMachine001/TargetIP001.
>
> How to Repro:
> In a good running NNs, take down the active NN (don't let it come back during
> test), and then the stand by NNs will keep trying dead (old active) NN,
> because it is the cached one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
