[ https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15489223#comment-15489223 ]
Arpit Agarwal commented on HDFS-10301: -------------------------------------- bq. I have modified the checkLease method in BlockReportLeaseManager to return true when node.leaseId == 0. Please let me know if you see any issues with this approach. [~redvine], IIUC this workaround bypasses the lease checks but the last report detection logic still remains broken. I am no longer sure zombie removal in heartbeats is safe and I was probably wrong to add it in HDFS-7596. Zombie removal is safe just after processing all storage reports from a full block report. So I think we should fix "last report detection". I believe the following two changes will fix this problem (same suggestion as my previous comment): # The DataNode sends a flag with the last RPC message that indicates all the previous reports have been successfully processed. This is guaranteed to be correct and removes the burden from the NN. # Eliminate single-RPC reports as Daryn suggested. Any thoughts on this? Thanks Konstantin and Vinitha for reporting this problem and your marathon efforts to fix it. It is a hard problem so I request we aim for consensus before committing a fix. > BlockReport retransmissions may lead to storages falsely being declared > zombie if storage report processing happens out of order > -------------------------------------------------------------------------------------------------------------------------------- > > Key: HDFS-10301 > URL: https://issues.apache.org/jira/browse/HDFS-10301 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode > Affects Versions: 2.6.1 > Reporter: Konstantin Shvachko > Assignee: Vinitha Reddy Gankidi > Priority: Critical > Fix For: 2.7.4 > > Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch, > HDFS-10301.004.patch, HDFS-10301.005.patch, HDFS-10301.006.patch, > HDFS-10301.007.patch, HDFS-10301.008.patch, HDFS-10301.009.patch, > HDFS-10301.01.patch, HDFS-10301.010.patch, HDFS-10301.011.patch, > HDFS-10301.012.patch, HDFS-10301.013.patch, HDFS-10301.014.patch, > HDFS-10301.branch-2.7.patch, HDFS-10301.branch-2.patch, > HDFS-10301.sample.patch, zombieStorageLogs.rtf > > > When NameNode is busy a DataNode can timeout sending a block report. Then it > sends the block report again. Then NameNode while process these two reports > at the same time can interleave processing storages from different reports. > This screws up the blockReportId field, which makes NameNode think that some > storages are zombie. Replicas from zombie storages are immediately removed, > causing missing blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org