[
https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15489223#comment-15489223
]
Arpit Agarwal commented on HDFS-10301:
--------------------------------------
bq. I have modified the checkLease method in BlockReportLeaseManager to return
true when node.leaseId == 0. Please let me know if you see any issues with this
approach.
[~redvine], IIUC this workaround bypasses the lease checks but the last report
detection logic still remains broken. I am no longer sure zombie removal in
heartbeats is safe and I was probably wrong to add it in HDFS-7596. Zombie
removal is safe just after processing all storage reports from a full block
report. So I think we should fix "last report detection".
I believe the following two changes will fix this problem (same suggestion as
my previous comment):
# The DataNode sends a flag with the last RPC message that indicates all the
previous reports have been successfully processed. This is guaranteed to be
correct and removes the burden from the NN.
# Eliminate single-RPC reports as Daryn suggested.
Any thoughts on this?
Thanks Konstantin and Vinitha for reporting this problem and your marathon
efforts to fix it. It is a hard problem so I request we aim for consensus
before committing a fix.
> BlockReport retransmissions may lead to storages falsely being declared
> zombie if storage report processing happens out of order
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-10301
> URL: https://issues.apache.org/jira/browse/HDFS-10301
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 2.6.1
> Reporter: Konstantin Shvachko
> Assignee: Vinitha Reddy Gankidi
> Priority: Critical
> Fix For: 2.7.4
>
> Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch,
> HDFS-10301.004.patch, HDFS-10301.005.patch, HDFS-10301.006.patch,
> HDFS-10301.007.patch, HDFS-10301.008.patch, HDFS-10301.009.patch,
> HDFS-10301.01.patch, HDFS-10301.010.patch, HDFS-10301.011.patch,
> HDFS-10301.012.patch, HDFS-10301.013.patch, HDFS-10301.014.patch,
> HDFS-10301.branch-2.7.patch, HDFS-10301.branch-2.patch,
> HDFS-10301.sample.patch, zombieStorageLogs.rtf
>
>
> When NameNode is busy a DataNode can timeout sending a block report. Then it
> sends the block report again. Then NameNode while process these two reports
> at the same time can interleave processing storages from different reports.
> This screws up the blockReportId field, which makes NameNode think that some
> storages are zombie. Replicas from zombie storages are immediately removed,
> causing missing blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
