[
https://issues.apache.org/jira/browse/HDFS-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789426#action_12789426
]
Hairong Kuang commented on HDFS-826:
------------------------------------
Would polling be good enough for your application? DFSOutputStream provides an
API to check if the number of active datanodes in the pipeline falls below the
replication factor.
The ultimate solution is not to decrease the length of the write pipeline on
errors. This is especially important for the block that's reopen for append. I
have been thinking about how to solve the problem and would like to put this to
be on the top of my work items. I will open a jira once I have any presentable
design.
> Allow a mechanism for an application to detect that datanode(s) have died in
> the write pipeline
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-826
> URL: https://issues.apache.org/jira/browse/HDFS-826
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
>
> HDFS does not replicate the last block of the file that is being currently
> written to by an application. Every datanode death in the write pipeline
> decreases the reliability of the last block of the currently-being-written
> block. This situation can be improved if the application can be notified of a
> datanode death in the write pipeline. Then, the application can decide what
> is the right course of action to be taken on this event.
> In our use-case, the application can close the file on the first datanode
> death, and start writing to a newly created file. This ensures that the
> reliability guarantee of a block is close to 3 at all time.
> One idea is to make DFSOutoutStream. write() throw an exception if the number
> of datanodes in the write pipeline fall below minimum.replication.factor that
> is set on the client (this is backward compatible).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
