[
https://issues.apache.org/jira/browse/HDFS-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292994#comment-16292994
]
Kihwal Lee commented on HDFS-12070:
-----------------------------------
To complete the history lesson, I traced down when {{closeFile}} was added to
{{commitBlockSynchronization()}} and why no one is calling it with {{false}}
anymore.
It turns out, the {{closeFile}} argument has existed since the dawn of
{{commitBlockSynchronization()}}. It was added by HADOOP-3310 to 0.18 in 2008.
The old append dependeds on it. Even in this, the normal lease recovery would
always call it with {{closeFile == true}}. There was a new
{{ClientDatanodeProtocol}} method, {{recoverBlock()}}, which causes
{{commitBlockSynchronization()}} to be called with {{closeFile == false}}. I
guess this disappeard when {{recoverBlock()}} client command was removed from
datanode. Today, a {{recoverLease()}} call to namenode can be used instead. It
is really fortunate that the {{closeFile}} option was initially added and has
survived for 9 years in spite of lack use.
> Failed block recovery leaves files open indefinitely and at risk for data loss
> ------------------------------------------------------------------------------
>
> Key: HDFS-12070
> URL: https://issues.apache.org/jira/browse/HDFS-12070
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Daryn Sharp
>
> Files will remain open indefinitely if block recovery fails which creates a
> high risk of data loss. The replication monitor will not replicate these
> blocks.
> The NN provides the primary node a list of candidate nodes for recovery which
> involves a 2-stage process. The primary node removes any candidates that
> cannot init replica recovery (essentially alive and knows about the block) to
> create a sync list. Stage 2 issues updates to the sync list – _but fails if
> any node fails_ unlike the first stage. The NN should be informed of nodes
> that did succeed.
> Manual recovery will also fail until the problematic node is temporarily
> stopped so a connection refused will induce the bad node to be pruned from
> the candidates. Recovery succeeds, the lease is released, under replication
> is fixed, and block is invalidated from the bad node.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
