[
https://issues.apache.org/jira/browse/HDFS-8383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737818#comment-14737818
]
Zhe Zhang commented on HDFS-8383:
---------------------------------
While we explore the ideas of refactoring {{DFSOutputStream}} and
{{BlockGroupDataStreamer}}, here are some thoughts:
# A simplified version of a streamer's {{run()}} loop (w.r.t error handling):
{code}
while (true) {
findError; // F
bumpStamp; // B
updatePipeline; // U
}
{code}
# I think the mission here is to handle all combinations of interleaved
{{F-B-U}} sequences from multiple streamers. For example:
{code}
F1-B1-F3-U1-B3-U3
F1-F3-B3-U3-B1-U1
{code}
# Regardless of the final solution I think we should write a test to emulate
the above interleaved combinations.
# This has been discussed above and in HDFS-8704: since we are never replacing
a DN, do we still need {{B}} and {{U}}? IIUC we still need them to identify
stale internal blocks (DN fails and comes back). But we can probably combine
them into a single RPC call. After getting a new {{genStamp}} we can probably
just set it on the block in NN?
# If we move {{B}} and {{U}} up (to {{OutputStream}} or {{BGDataStreamer}}),
the logic is much simpler. We just need to make sure to correctly handle the
arrival or new {{F}} messages while the upper level processes {{B+U}}. E.g.,
does it make sense the bump stamp multiple times.
> Tolerate multiple failures in DFSStripedOutputStream
> ----------------------------------------------------
>
> Key: HDFS-8383
> URL: https://issues.apache.org/jira/browse/HDFS-8383
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Tsz Wo Nicholas Sze
> Assignee: Walter Su
> Attachments: HDFS-8383.00.patch, HDFS-8383.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
