[
https://issues.apache.org/jira/browse/HDFS-16942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17698164#comment-17698164
]
ASF GitHub Bot commented on HDFS-16942:
---------------------------------------
Hexiaoqiao commented on code in PR #5460:
URL: https://github.com/apache/hadoop/pull/5460#discussion_r1130375460
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java:
##########
@@ -791,6 +792,9 @@ private void offerService() throws Exception {
shouldServiceRun = false;
return;
}
+ if (InvalidBlockReportLeaseException.class.getName().equals(reClass)) {
+ fullBlockReportLeaseId = 0;
Review Comment:
> Its really the isBlockReportDue() method that controls whether a new one
should be sent of not, and that is based on time since the last one. The the
blockReport(), it updates the time after a successful block report, but if it
gets an exception, like this change causes, it will not update the time and so
it will try again on the next heartbeat if it gets a new lease.
Thanks for the detailed explain. Make sense to me.
> perhaps we can add one liner log to indicate that the particular FBR went
through this trouble (i.e. log report id and lease id)
+1 from my side.
> Send error to datanode if FBR is rejected due to bad lease
> ----------------------------------------------------------
>
> Key: HDFS-16942
> URL: https://issues.apache.org/jira/browse/HDFS-16942
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
> Reporter: Stephen O'Donnell
> Assignee: Stephen O'Donnell
> Priority: Major
> Labels: pull-request-available
>
> When a datanode sends a FBR to the namenode, it requires a lease to send it.
> On a couple of busy clusters, we have seen an issue where the DN is somehow
> delayed in sending the FBR after requesting the least. Then the NN rejects
> the FBR and logs a message to that effect, but from the Datanodes point of
> view, it thinks the report was successful and does not try to send another
> report until the 6 hour default interval has passed.
> If this happens to a few DNs, there can be missing and under replicated
> blocks, further adding to the cluster load. Even worse, I have see the DNs
> join the cluster with zero blocks, so it is not obvious the under replication
> is caused by lost a FBR, as all DNs appear to be up and running.
> I believe we should propagate an error back to the DN if the FBR is rejected,
> that way, the DN can request a new lease and try again.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
