[
https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15494623#comment-15494623
]
Arpit Agarwal commented on HDFS-10301:
--------------------------------------
bq. Balancer copies a replica from a source DN to a target DN and when finished
sends IBR with the target as a new replica location and a hint to remove old
replica from the source DN. If the source or the target storage fails during
this the transfer fails and Balancer moves on. If either of the storages fail
after the transfer it is the same as the regular failure, the block will become
under-replicated and recovered in due time.
We've seen IBRs are often delayed when the NN is overloaded so the NN's view of
the replica map can lag. But I agree leaving zombie removals to heartbeats only
fixes this bug and leaves us no worse than where we are today. The FBR vs
heartbeat discussion can be separate. If we go this way let's fix the detection
properly though. The last patch just no-ops the lease ID checks.
bq. For VolumeChoosingPolicy it is even more important to know early which
storages failed in order to avoid choosing them as targets.
By the way, the storage chosen by the NN is never used. The DN always uses the
result of running volume choosing policy locally.
> BlockReport retransmissions may lead to storages falsely being declared
> zombie if storage report processing happens out of order
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-10301
> URL: https://issues.apache.org/jira/browse/HDFS-10301
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 2.6.1
> Reporter: Konstantin Shvachko
> Assignee: Vinitha Reddy Gankidi
> Priority: Critical
> Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch,
> HDFS-10301.004.patch, HDFS-10301.005.patch, HDFS-10301.006.patch,
> HDFS-10301.007.patch, HDFS-10301.008.patch, HDFS-10301.009.patch,
> HDFS-10301.01.patch, HDFS-10301.010.patch, HDFS-10301.011.patch,
> HDFS-10301.012.patch, HDFS-10301.013.patch, HDFS-10301.014.patch,
> HDFS-10301.branch-2.7.patch, HDFS-10301.branch-2.patch,
> HDFS-10301.sample.patch, zombieStorageLogs.rtf
>
>
> When NameNode is busy a DataNode can timeout sending a block report. Then it
> sends the block report again. Then NameNode while process these two reports
> at the same time can interleave processing storages from different reports.
> This screws up the blockReportId field, which makes NameNode think that some
> storages are zombie. Replicas from zombie storages are immediately removed,
> causing missing blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
