[
https://issues.apache.org/jira/browse/HDFS-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-9079:
----------------------------
Attachment: HDFS-9079.01.patch
Rebased the patch on top of HDFS-9040 and made it more complete. It's still a
WIP. The main logic is:
# As described [above |
https://issues.apache.org/jira/browse/HDFS-9079?focusedCommentId=14905503&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14905503],
redesigned the coordinator to be an event processing daemon.
# Limiting the lifespan of {{StripedDataStreamer}} to a single block. This is
to simplify the logic.
# Preallocating GS by groups, so GS bumping can be processed locally by the
coordinator.
I also modified {{TestWriteStripedFileWithFailure}} to be a minimum error
handling test -- it writes a small file (< 1 block), and there's 1 failure
during the write. The patch passes the test and the sequence of events is as
expected. I'm now working on more complex tests including
{{TestDFSStripedOutputStreamWithFailure}}.
> Erasure coding: preallocate multiple generation stamps and serialize updates
> from data streamers
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-9079
> URL: https://issues.apache.org/jira/browse/HDFS-9079
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
> Affects Versions: HDFS-7285
> Reporter: Zhe Zhang
> Assignee: Zhe Zhang
> Attachments: HDFS-9079-HDFS-7285.00.patch, HDFS-9079.01.patch
>
>
> A non-striped DataStreamer goes through the following steps in error handling:
> {code}
> 1) Finds error => 2) Asks NN for new GS => 3) Gets new GS from NN => 4)
> Applies new GS to DN (createBlockOutputStream) => 5) Ack from DN => 6)
> Updates block on NN
> {code}
> To simplify the above we can preallocate GS when NN creates a new striped
> block group ({{FSN#createNewBlock}}). For each new striped block group we can
> reserve {{NUM_PARITY_BLOCKS}} GS's. Then steps 1~3 in the above sequence can
> be saved. If more than {{NUM_PARITY_BLOCKS}} errors have happened we
> shouldn't try to further recover anyway.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
