haiyang1987 commented on PR #6176: URL: https://github.com/apache/hadoop/pull/6176#issuecomment-1767833407
> > > Back to this PR. > > > IMO, adding a timeout mechanism may not add much pressure on NameNode. However, it seems that the implementation of that solution is more complex than the current patch and requires more comprehensive design and consideration. The good aspect is that the timeout mechanism can completely solve the problem of excess replica leakage, after all, the situation where datanodes fail to successfully delete replicas according to commands may not be limited to the scenario described in this JIRA. > > I totally support the solution @zhangshuyan0 mentioned here. https://issues.apache.org/jira/browse/HDFS-17218?focusedCommentId=17774766&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17774766 Thanks @Hexiaoqiao for your comment. Indeed this timeout mechanism solution can completely solve the problem of redundant copy leakage, however the implementation cost of this solution maybe relatively high. for a large cluster, such as balance and reduce replication etc occur frequently, and ExcessRedundancyMap stores a lot of information. During each iteration, DataNodes and their corresponding blocks are fully traversed for processing. this approach might potentially increase the load on holding the NameNode write lock, sure we can make good designs to avoid holding the write lock time as much as possible. The current pr to slove problem of ExcessRedundancyMap leakage on a case-by-case basis, and the implementation cost is relatively low. I think if there will be problems like ExcessRedundancyMap leakage in the future again, we should probably solve it case by case and find the root cause of the leakage to solve it. Of course, if we decide to adopt the timeout mechanism solution, I will submit a new PR. look forward to your feedback. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
