[
https://issues.apache.org/jira/browse/HDFS-7663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14908505#comment-14908505
]
Jing Zhao edited comment on HDFS-7663 at 9/25/15 6:53 PM:
----------------------------------------------------------
Thanks for sharing the thoughts, Walter! First I completely agree that we
should start from #3 you proposed. More specifically, quote from your doc:
{quote}
If lastBlock is complete, start a new blockGroup.
If lastBlock is UC, start a blockRecovery. After recovery, start a new
blockGroup. BlockRecovery prefers to save data from last paritial stripe as
much as possible.
pros: simple and clear code, with HDFS-3689.
cons: a) Has paritial blockGroup, waste NN memory. b) Could hurt read
performance.
{quote}
This is something we can start with.
Some high level thinking:
# Lease/Block recovery is actually a different topic and is a tricky problem.
As you suggested in HDFS-9040, we can create a separate jira for it, and the
second half of your doc can be our first proposal there.
# Looks like till now we do not have a correct solution for EC file lease
recovery (at least the protocol for DataNodes to do the recovery work)? We
should first let NameNode *NOT* send recovery commands to DataNodes for under
construction/recovery EC blocks.
# Appending to a partial block group is actually very similar to "keep writing
after calling hsync/hflush", since we need to overwrite the parity data of the
last stripe. We have an initial proposal in the latest design doc in HDFS-7285
about this part.
was (Author: jingzhao):
Thanks for sharing the thoughts, Walter! First I completely agree that we
should start from #3 you proposed. More specifically, quote from your doc:
{quote}
If lastBlock is complete, start a new blockGroup.
If lastBlock is UC, start a blockRecovery. After recovery, start a new
blockGroup. BlockRecovery prefers to save data from last paritial stripe as
much as possible.
pros: simple and clear code, with HDFS-3689.
cons: a) Has paritial blockGroup, waste NN memory. b) Could hurt read
performance.
{quote}
This is something we can easily support and also we should start with.
Some high level thinking:
# Lease/Block recovery is actually a different topic and is a tricky problem.
As you suggested in HDFS-9040, we can create a separate jira for it, and the
second half of your doc can be our first proposal there.
# Looks like till now we do not have a correct solution for EC file lease
recovery (at least the protocol for DataNodes to do the recovery work)? We
should first let NameNode *NOT* send recovery commands to DataNodes for under
construction/recovery EC blocks.
# Appending to a partial block group is actually very similar to "keep writing
after calling hsync/hflush", since we need to overwrite the parity data of the
last stripe. We have an initial proposal in the latest design doc in HDFS-7285
about this part.
> Erasure Coding: lease recovery / append on striped file
> -------------------------------------------------------
>
> Key: HDFS-7663
> URL: https://issues.apache.org/jira/browse/HDFS-7663
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Jing Zhao
> Assignee: Walter Su
> Attachments: HDFS-7663.00.txt
>
>
> Append should be easy if we have variable length block support from
> HDFS-3689, i.e., the new data will be appended to a new block. We need to
> revisit whether and how to support appending data to the original last block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
