[
https://issues.apache.org/jira/browse/HDFS-17123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17747332#comment-17747332
]
ASF GitHub Bot commented on HDFS-17123:
---------------------------------------
hfutatzhanghb commented on PR #5889:
URL: https://github.com/apache/hadoop/pull/5889#issuecomment-1651135519
@Hexiaoqiao @zhangshuyan0 Sir, thanks for replying. Yes, I met some issues.
Details are below:
1. As mentioned in
[HDFS-17047](https://issues.apache.org/jira/browse/HDFS-17047), we found two ec
blocks of one same block group are in the different storages of one datanode.
This case can cause ec reconstruction infinitely.
2. To resolve this problem, We decide to use DirectoryScanner to detect
duplicated ec internal blocks and delete them. But delete them directly in any
order is not safe, because the datanode generates fbr using unordered storages
index. We can not infer which duplicated replica will be handled by
`BlockManager#addStoredBlock` method lastly, So I think we should sort datanode
storages when generating fbr to make sure we can use this order in
`FsDatasetImpl#checkAndUpdate` method to delete the replicas which handled by
namenode earlier.
3. And when deleteing duplicated replicas, We only delete those scanInfos
whose (now - time be scanned) > block report interval to make sure we do delete
safely.
Hope to receive your suggestions~ Thanks a lot.
> Sort datanodeStorages when generating StorageBlockReport[] in method
> BPServiceActor#blockReport for future convenience
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-17123
> URL: https://issues.apache.org/jira/browse/HDFS-17123
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Affects Versions: 3.4.0
> Reporter: farmmamba
> Assignee: farmmamba
> Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
