[
https://issues.apache.org/jira/browse/HADOOP-15684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16586356#comment-16586356
]
Rong Tang edited comment on HADOOP-15684 at 8/20/18 6:36 PM:
-------------------------------------------------------------
[~lukmajercak] I agree with you.
The current logic is:
1. Try to RollEditLog for cachedAcitveProxy.
2. If it returns StandBy exception, it knows that the cached is a standby and
try next NN.
3. For other exceptions, it just re-throws, interesting is that the outer
method triggerActiveLogRoll just *swallow* the exceptions.
4. That means, next time, it will continue to try the same NN.
The issue I report here is that if a name node is
dead(ConnectTimeoutException), it will go to step 3 and 4, and indefinitely
takes the dead NN and active.
A better solution, I think, is like you said.
# Try to RollEditLog for cachedAcitveProxy.
# If it is StandBy exception, we should log it as INFO, and try next NN.
# For other exceptions, we should log it as Warning, and try next NN.
*It makes no sense to stuck trying the same NN, if it returns exception.*
I will provide the patch for it. Please let me know if you have more
suggestions.
[~jojochuang] for knowing.
was (Author: trjianjianjiao):
[~lukmajercak] I agree with you.
The current logic is:
1. Try to RollEditLog for cachedAcitveProxy.
2. If it returns StandBy exception, it knows that the cached is a standby and
try next NN.
3. For other exceptions, it just re-throws, interesting is that the outer
method triggerActiveLogRoll just *swallow* the exceptions.
4. That means, next time, it will continue to try the same NN.
The issue I report here is that if a name node is
dead(ConnectTimeoutException), it will go to step 3 and 4, and indefinitely
takes the dead NN and active.
A better solution, I think, is like you said.
# Try to RollEditLog for cachedAcitveProxy.
# If it is StandBy exception, we should log it as *INFO*, and try next NN.
# For other exceptions, we should log it as *Warning*, and try next NN.
It makes no sense to stuck trying the same NN, if it returns exception.
I will provide the patch for it. Please let me know if you have more
suggestions.
[~jojochuang] for knowing.
> triggerActiveLogRoll stuck on dead name node, when ConnectTimeoutException
> happens.
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-15684
> URL: https://issues.apache.org/jira/browse/HADOOP-15684
> Project: Hadoop Common
> Issue Type: Bug
> Components: ha
> Reporter: Rong Tang
> Priority: Critical
> Attachments: FixRollEditLog.patch,
> hadoop--rollingUpgrade-BN2SCH070021402.log
>
>
> When name node call triggerActiveLogRoll, and the cachedActiveProxy is a dead
> name node, it will throws a ConnectTimeoutException, expected behavior is to
> try next NN, but current logic doesn't do so, instead, it keeps trying the
> dead, mistakenly take it as active.
>
> 2018-08-17 10:02:12,001 WARN [Edit log tailer]
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to trigger a
> roll of the active NN
> org.apache.hadoop.net.ConnectTimeoutException: Call From
> BN2SCH070021402/25.126.188.193 to BN2SCH070041016.ap.gbl:8020 failed on
> socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException:
> 20000 millis timeout
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:298)
>
> C:\Users\rotang>ping BN2SCH070041016
> Pinging BN2SCH070041016 [25.126.141.79] with 32 bytes of data:
> Request timed out.
> Request timed out.
> Request timed out.
> Request timed out.
>
> Attachment is a log file saying how it repeatedly retries a dead name node,
> and a fix patch.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
