[
https://issues.apache.org/jira/browse/HDFS-16023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342653#comment-17342653
]
Stephen O'Donnell commented on HDFS-16023:
------------------------------------------
{quote}
When the NameNode receives the data (FBR) from the DataNode, it will put the
data in the queue (BlockReportProcessingThread#queue), and there will be
threads processing them thereafter.
{quote}
I don't think that is how FBRs work. The DN waits for the FBR to process before
the RPC call from the DN to NN returns,so it should not send another FBR until
the first one has returned. This is the code which processes it - it queues the
future and but waits for it to be done before returning:
{code}
// sync batch processing for a full BR.
public <T> T runBlockOp(final Callable<T> action)
throws IOException {
final FutureTask<T> future = new FutureTask<T>(action);
enqueueBlockOp(future);
try {
return future.get();
} catch (ExecutionException ee) {
Throwable cause = ee.getCause();
if (cause == null) {
cause = ee;
}
if (!(cause instanceof IOException)) {
cause = new IOException(cause);
}
throw (IOException)cause;
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
throw new IOException(ie);
}
}
{code}
I guess the RPC can timeout on the DN side and leave the FBR in the queue and
that could return in a duplicate getting sent.
> Improve blockReportLeaseId acquisition to avoid repeated FBR
> ------------------------------------------------------------
>
> Key: HDFS-16023
> URL: https://issues.apache.org/jira/browse/HDFS-16023
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: JiangHua Zhu
> Assignee: JiangHua Zhu
> Priority: Minor
>
> When the NameNode receives the data (FBR) from the DataNode, it will put the
> data in the queue (BlockReportProcessingThread#queue), and there will be
> threads processing them thereafter.
> When the DataNode wants to send data (here, FBR) to the NameNode, it will
> first obtain a blockReportLeaseId from the NameNode. If the DataNode data
> already exists in the queue, there is no need to assign a blockReportLeaseId
> to the DataNode again.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
