[
https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15526429#comment-15526429
]
Daryn Sharp commented on HDFS-10301:
------------------------------------
Everyone appeared to agree on removing single-rpc FBRs. Are we going to remove
it?
Agree with Konstantin and Vinitha about concerns over "last" BR detection.
With no guaranteed order of processing, there is no "last" FBR but rather "all"
FBRs processed. We just need to safely know the NN is sufficiently in sync
with the DN for a specific point in time. The race to consider is apparently a
BR for a new volume is processed prior to receiving/processing a heartbeat
which includes a storage report for the new volume.
If we fix the detection, then agree on completely removing all the zombie
processing.
I like the existing behavior of flagging a storage as failed for deferred
removal by the heartbeat monitor. The default 10 minute lag is concerning. A
failure detected at runtime sends IBRs (wasn't aware DN did that, has it
always?) which alleviates the concern. However a failure detected at startup,
ex. rolling upgrade, will reduce durability for 10 minutes. Perhaps the DN
detects failed volumes quickly enough this shouldn't be a concern?
> BlockReport retransmissions may lead to storages falsely being declared
> zombie if storage report processing happens out of order
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-10301
> URL: https://issues.apache.org/jira/browse/HDFS-10301
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 2.6.1
> Reporter: Konstantin Shvachko
> Assignee: Vinitha Reddy Gankidi
> Priority: Critical
> Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch,
> HDFS-10301.004.patch, HDFS-10301.005.patch, HDFS-10301.006.patch,
> HDFS-10301.007.patch, HDFS-10301.008.patch, HDFS-10301.009.patch,
> HDFS-10301.01.patch, HDFS-10301.010.patch, HDFS-10301.011.patch,
> HDFS-10301.012.patch, HDFS-10301.013.patch, HDFS-10301.014.patch,
> HDFS-10301.branch-2.7.patch, HDFS-10301.branch-2.patch,
> HDFS-10301.sample.patch, zombieStorageLogs.rtf
>
>
> When NameNode is busy a DataNode can timeout sending a block report. Then it
> sends the block report again. Then NameNode while process these two reports
> at the same time can interleave processing storages from different reports.
> This screws up the blockReportId field, which makes NameNode think that some
> storages are zombie. Replicas from zombie storages are immediately removed,
> causing missing blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
