[
https://issues.apache.org/jira/browse/HDFS-17446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17832782#comment-17832782
]
ASF GitHub Bot commented on HDFS-17446:
---------------------------------------
hiwangzhihui commented on code in PR #6685:
URL: https://github.com/apache/hadoop/pull/6685#discussion_r1546135795
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java:
##########
@@ -749,6 +749,8 @@ void addToReplicasMap(ReplicaMap volumeMap, File dir,
lazyWriteReplicaMap, isFinalized, exceptions, subTaskQueue);
subTask.fork();
subTaskQueue.add(subTask);
+ } else if (isFinalized && ++scanBlockCnt % 10000 == 0) {
+ LOG.info((scanBlockCnt - 1) + " blocks (finalize state) have been
scanned.");
Review Comment:
Please take care of concurrency issues that lead to inaccurate counting.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java:
##########
@@ -749,6 +749,8 @@ void addToReplicasMap(ReplicaMap volumeMap, File dir,
lazyWriteReplicaMap, isFinalized, exceptions, subTaskQueue);
subTask.fork();
subTaskQueue.add(subTask);
+ } else if (isFinalized && ++scanBlockCnt % 10000 == 0) {
+ LOG.info((scanBlockCnt - 1) + " blocks (finalize state) have been
scanned.");
Review Comment:
The log style is not easy to view.
> The DataNode adds a log to report its progress during addToReplicasMap
> execution.
> ---------------------------------------------------------------------------------
>
> Key: HDFS-17446
> URL: https://issues.apache.org/jira/browse/HDFS-17446
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
> Reporter: xiaojunxiang
> Priority: Critical
> Labels: pull-request-available
> Attachments: image-2024-03-28-23-53-29-057.png
>
>
> Datanodes do not print logs when addToReplicasMap is just started, so we will
> not be aware of its progress, or even wait up to 4 hours for a large cluster.
> This is very confusing.
> A cluster is as follows:
> !image-2024-03-28-23-53-29-057.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
