[
https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15298565#comment-15298565
]
Colin Patrick McCabe commented on HDFS-10301:
---------------------------------------------
Hi [~redvine],
Thanks for your interest in this. I wish I could get more people interested in
this JIRA-- it has been hard to raise interest, unfortunately.
Just to clarify, you don't need to assign a JIRA to yourself in order to post a
patch or suggest a solution. In general, when someone is actively working on a
patch, you should ask before reassigning their JIRAs to yourself.
A whole separate RPC just for reporting the storages which are present seems
excessive. It will add additional load to the namenode.
{code}
if (node.leaseId == 0) {
- LOG.warn("BR lease 0x{} is not valid for DN {}, because the DN " +
- "is not in the pending set.",
- Long.toHexString(id), dn.getDatanodeUuid());
- return false;
+ LOG.debug("DN {} is not in the pending set because BR with lease 0x{}
was processed out of order",
+ dn.getDatanodeUuid(), Long.toHexString(id));
+ return true;
{code}
The leaseId being 0 doesn't mean that the block report was processed out of
order. If you manually trigger a block report with the {{hdfs dfsadmin
\-triggerBlockReport}} command, it will also have lease id 0. Legacy block
reports will also have lease ID 0.
In general, your solution doesn't fix the problem during upgrade and is a much
bigger patch, which is why I think HDFS-10301.003.patch should be committed and
the RPC changes should be done in a follow-on JIRA. I do not see us
backporting RPC changes to all the stable branches.
> BlockReport retransmissions may lead to storages falsely being declared
> zombie if storage report processing happens out of order
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-10301
> URL: https://issues.apache.org/jira/browse/HDFS-10301
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 2.6.1
> Reporter: Konstantin Shvachko
> Assignee: Colin Patrick McCabe
> Priority: Critical
> Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch,
> HDFS-10301.004.patch, HDFS-10301.01.patch, HDFS-10301.sample.patch,
> zombieStorageLogs.rtf
>
>
> When NameNode is busy a DataNode can timeout sending a block report. Then it
> sends the block report again. Then NameNode while process these two reports
> at the same time can interleave processing storages from different reports.
> This screws up the blockReportId field, which makes NameNode think that some
> storages are zombie. Replicas from zombie storages are immediately removed,
> causing missing blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
