[
https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15536676#comment-15536676
]
Konstantin Shvachko commented on HDFS-10301:
--------------------------------------------
I think the solution for the {{checkLease()}} problem is
# to restore returning {{false}} in {{checkLease()}}. That way "bad" DNs will
not be able to send FBRs without obtaining a lease.
# Move release lease logic from per-storage {{BM.processReport()}} to
{{NameNodeRpcServer.blockReport()}} just after all storages in the report are
processed. That way we will not need to track the order of processing
per-storage reports by FutureTask threads, but just release the lease once all
storages are done, that is when {{curRpc + 1 == totalRpcs}} .
This will exactly match the current behavior of br-leases. It does not solve
the race between a timed out BR and the repeating BR in multi-RPC BR case. But
the race exists in current code as well, and I would prefer to address this in
another issue.
So before going to testing this on multiple versions and configurations I would
like to confirm with you guys that this is the only problem remaining, and we
have a consensus on the solution. Please confirm.
> BlockReport retransmissions may lead to storages falsely being declared
> zombie if storage report processing happens out of order
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-10301
> URL: https://issues.apache.org/jira/browse/HDFS-10301
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 2.6.1
> Reporter: Konstantin Shvachko
> Assignee: Vinitha Reddy Gankidi
> Priority: Critical
> Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch,
> HDFS-10301.004.patch, HDFS-10301.005.patch, HDFS-10301.006.patch,
> HDFS-10301.007.patch, HDFS-10301.008.patch, HDFS-10301.009.patch,
> HDFS-10301.01.patch, HDFS-10301.010.patch, HDFS-10301.011.patch,
> HDFS-10301.012.patch, HDFS-10301.013.patch, HDFS-10301.014.patch,
> HDFS-10301.branch-2.7.patch, HDFS-10301.branch-2.patch,
> HDFS-10301.sample.patch, zombieStorageLogs.rtf
>
>
> When NameNode is busy a DataNode can timeout sending a block report. Then it
> sends the block report again. Then NameNode while process these two reports
> at the same time can interleave processing storages from different reports.
> This screws up the blockReportId field, which makes NameNode think that some
> storages are zombie. Replicas from zombie storages are immediately removed,
> causing missing blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
