[
https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-9040:
----------------------------
Attachment: HDFS-9040-HDFS-7285.004.patch
Update the patch based on our discussion. The main changes compared with 003
version:
# Flush out all the enqueued data to DataNodes before handling failures and
bumping GS
# Instead of let each DataStreamer write their own last empty packet of the
block, we do it in the StripedOutputStream level so that we can still bump GS
for failure handling before some streamers close their internal blocks.
# During the test I found that some data streamer may take a long time to
close/create datanode connections. This may cause other streamers' connections
timeout. Thus the new patch adds an upper bound for the total waiting time of
creating datanode connections during failure handling.
A big missing part is test. We need to add a lot more tests to cover all
different scenarios. Maybe we can use HDFS-9098 to do it.
> Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests
> to Coordinator)
> -------------------------------------------------------------------------------------------
>
> Key: HDFS-9040
> URL: https://issues.apache.org/jira/browse/HDFS-9040
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Walter Su
> Assignee: Jing Zhao
> Attachments: HDFS-9040-HDFS-7285.002.patch,
> HDFS-9040-HDFS-7285.003.patch, HDFS-9040-HDFS-7285.004.patch,
> HDFS-9040.00.patch, HDFS-9040.001.wip.patch, HDFS-9040.02.bgstreamer.patch
>
>
> The general idea is to simplify error handling logic.
> Proposal 1:
> A BlockGroupDataStreamer to communicate with NN to allocate/update block, and
> StripedDataStreamer s only have to stream blocks to DNs.
> Proposal 2:
> See below the
> [comment|https://issues.apache.org/jira/browse/HDFS-9040?focusedCommentId=14741388&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14741388]
> from [~jingzhao].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
