[
https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14482692#comment-14482692
]
Yongjun Zhang commented on HDFS-7980:
-------------------------------------
Thanks [~walter.k.su].
Since an incremental block report is sent before a full block report, the
incremental block report will be processed by
{{processFirstBlockReport(storageInfo, newReport);}} and all the rest of the
block reports including the full block reports will be processed by
{{invalidatedBlocks = processReport(storageInfo, newReport);}}, would you
please explain how your change helps the performance issue? I guess I lack some
understanding here.
See [~szetszwo]'s comment
{quote}
As mentioned by Hui Zheng in the description, the problem seems that
blockReport() may send an incremental block report by
reportReceivedDeletedBlocks() before sending the full report. In that case, NN
treats the incremental block report as the first block report but not the full
report. The processing time for the full report is increased dramatically.
{quote}
Thanks.
> Incremental BlockReport will dramatically slow down the startup of a namenode
> ------------------------------------------------------------------------------
>
> Key: HDFS-7980
> URL: https://issues.apache.org/jira/browse/HDFS-7980
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Hui Zheng
> Assignee: Walter Su
> Attachments: HDFS-7980.001.patch
>
>
> In the current implementation the datanode will call the
> reportReceivedDeletedBlocks() method that is a IncrementalBlockReport before
> calling the bpNamenode.blockReport() method. So in a large(several thousands
> of datanodes) and busy cluster it will slow down(more than one hour) the
> startup of namenode.
> {code}
> List<DatanodeCommand> blockReport() throws IOException {
> // send block report if timer has expired.
> final long startTime = now();
> if (startTime - lastBlockReport <= dnConf.blockReportInterval) {
> return null;
> }
> final ArrayList<DatanodeCommand> cmds = new ArrayList<DatanodeCommand>();
> // Flush any block information that precedes the block report. Otherwise
> // we have a chance that we will miss the delHint information
> // or we will report an RBW replica after the BlockReport already reports
> // a FINALIZED one.
> reportReceivedDeletedBlocks();
> lastDeletedReport = startTime;
> .........
> // Send the reports to the NN.
> int numReportsSent = 0;
> int numRPCs = 0;
> boolean success = false;
> long brSendStartTime = now();
> try {
> if (totalBlockCount < dnConf.blockReportSplitThreshold) {
> // Below split threshold, send all reports in a single message.
> DatanodeCommand cmd = bpNamenode.blockReport(
> bpRegistration, bpos.getBlockPoolId(), reports);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
