[
https://issues.apache.org/jira/browse/HDFS-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849577#comment-13849577
]
Suresh Srinivas commented on HDFS-5434:
---------------------------------------
Some comments:
bq. Server-side enforcement of pipeline size also seems somewhat more
complementary to where HDFS is headed with server-side tiered storage policies
(HDFS-4672).
I am not sure what you mean here. Can you add more details?
bq. sort of assumed when I read this JIRA that Eric was considering situations
where the admin "knows" that the storage is reliable despite having only one
replica. For example, if there is RAID and we trust it, or if the backend
storage system which the DataNode is using is doing replication underneath HDFS.
I think if HDFS claims support for this, RAID setup without spare disks to
rebuild the array could cause more data loss.
I feel that this looks more like a hack. Perhaps this configuration should be
for now hidden. Also I want to see a patch to make sure that this does not add
unnecessary complexity to the HDFS code, given this is not the use case that is
common for HDFS. I know that you have already commented that it is not ideal,
but doing this is as client only (hopefully this can be made pluggable)
functionality, where client on file closure reduces the replication count to 1
may be workable.
> Write resiliency for replica count 1
> ------------------------------------
>
> Key: HDFS-5434
> URL: https://issues.apache.org/jira/browse/HDFS-5434
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 2.2.0
> Reporter: Buddy
> Priority: Minor
>
> If a file has a replica count of one, the HDFS client is exposed to write
> failures if the data node fails during a write. With a pipeline of size of
> one, no recovery is possible if the sole data node dies.
> A simple fix is to force a minimum pipeline size of 2, while leaving the
> replication count as 1. The implementation for this is fairly non-invasive.
> Although the replica count is one, the block will be written to two data
> nodes instead of one. If one of the data nodes fails during the write, normal
> pipeline recovery will ensure that the write succeeds to the surviving data
> node.
> The existing code in the name node will prune the extra replica when it
> receives the block received reports for the finalized block from both data
> nodes. This results in the intended replica count of one for the block.
> This behavior should be controlled by a configuration option such as
> {{dfs.namenode.minPipelineSize}}.
> This behavior can be implemented in {{FSNameSystem.getAdditionalBlock()}} by
> ensuring that the pipeline size passed to
> {{BlockPlacementPolicy.chooseTarget()}} in the replication parameter is:
> {code}
> max(replication, ${dfs.namenode.minPipelineSize})
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
