[
https://issues.apache.org/jira/browse/HDFS-14978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17001041#comment-17001041
]
Aravindan Vijayan commented on HDFS-14978:
------------------------------------------
[~ayushtkn] Thanks for reviewing the design doc. Answers inline.
bq. Does the swapBlockList() Takes care of storage quota validations?
Not in this version. We plan to support it in the next version of this feature.
How is storage Policies handled here? do you retain it, There are some policies
not supported for EC, like ONE_SSD.
bq. The temporary file that is created will be created with a specific policy
for EC. And then the swap block list will also make sure that the replicated
file's policy is updated to the newly created EC one.
bq. What would be response, say you want to switch to 6-3 and you are able
fetch only 7 blocks, unable to write two parities, do we allow moving further,
despite moving into chances of making data more vulnerable? or do we fail?
We will fail fast.
bq. The Overhead of an extra tmp file still says?
The tmp file will be deleted at the end of the operation.
bq. I need to check this but, If a client is reading a file, the swapBlocks()
happens the block locations get changed, he shall go to refetch the locations
from namenode, that would be now EC Blocks, The client would be on
DFSInputStream not on DFSStripedInputStream, I might have messed up, but is
there a cover to this?
Yes this is a valid concern. We will have to try and repro this possible race
condition. The plan is to not fail the client's read operation while the file
is undergoing in place EC conversion.
> In-place Erasure Coding Conversion
> ----------------------------------
>
> Key: HDFS-14978
> URL: https://issues.apache.org/jira/browse/HDFS-14978
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: erasure-coding
> Affects Versions: 3.0.0
> Reporter: Wei-Chiu Chuang
> Assignee: Aravindan Vijayan
> Priority: Major
> Attachments: In-place Erasure Coding Conversion.pdf
>
>
> HDFS Erasure Coding is a new feature added in Apache Hadoop 3.0. It uses
> encoding algorithms to reduce disk space usage while retaining redundancy
> necessary for data recovery. It was a huge amount of work but it is just
> getting adopted after almost 2 years.
> One usability problem that’s blocking users from adopting HDFS Erasure Coding
> is that existing replicated files have to be copied to an EC-enabled
> directory explicitly. Renaming a file/directory to an EC-enabled directory
> does not automatically convert the blocks. Therefore users typically perform
> the following steps to erasure-code existing files:
> {noformat}
> Create $tmp directory, set EC policy at it
> Distcp $src to $tmp
> Delete $src (rm -rf $src)
> mv $tmp $src
> {noformat}
> There are several reasons why this is not popular:
> * Complex. The process involves several steps: distcp data to a temporary
> destination; delete source file; move destination to the source path.
> * Availability: there is a short period where nothing exists at the source
> path, and jobs may fail unexpectedly.
> * Overhead. During the copy phase, there is a point in time where all of
> source and destination files exist at the same time, exhausting disk space.
> * Not snapshot-friendly. If a snapshot is taken prior to performing the
> conversion, the source (replicated) files will be preserved in the cluster
> too. Therefore, the conversion actually increase storage space usage.
> * Not management-friendly. This approach changes file inode number,
> modification time and access time. Erasure coded files are supposed to store
> cold data, but this conversion makes data “hot” again.
> * Bulky. It’s either all or nothing. The directory may be partially erasure
> coded, but this approach simply erasure code everything again.
> To ease data management, we should offer a utility tool to convert replicated
> files to erasure coded files in-place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
