[
https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005454#comment-16005454
]
Kai Zheng commented on HDFS-7337:
---------------------------------
Thanks [~Sammi] for moving this on.
The PluggableErasureCodec-v4 design reflects all the latest discussions (mainly
with [~andrew.wang], [~Sammi] and [~jojochuang]) and related work. Important
ones are:
HADOOP-13200 Implement customizable and configurable erasure coders;
HDFS-11314 Enforce set of enabled EC policies on the NameNode;
HDFS-11605 Allow user to customize new erasure code policies.
To close the gap targeting for the upcoming {{3.0 ALPHA3}} release we're also
tightly working on these:
HDFS-11794 Add ec sub command -listCodec to show currently supported ec codecs
HDFS-11606 Add CLI cmd to remove an erasure code policy
HDFS-7859 Erasure Coding: Persist erasure coding policies in NameNode
As we're close and will make the changes into NameNode fsimage/editlog in
HDFS-7859, would it be great if more folks take a look at the latest design doc
and see if any concerns. Particularly, could I ping [~szetszwo], [~jingzhao],
[~zhz] and [~vinayrpet]. Thanks for your time!
> Configurable and pluggable Erasure Codec and schema
> ---------------------------------------------------
>
> Key: HDFS-7337
> URL: https://issues.apache.org/jira/browse/HDFS-7337
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: erasure-coding
> Reporter: Zhe Zhang
> Priority: Critical
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-7337-prototype-v1.patch,
> HDFS-7337-prototype-v2.zip, HDFS-7337-prototype-v3.zip,
> PluggableErasureCodec.pdf, PluggableErasureCodec-v2.pdf,
> PluggableErasureCodec-v3.pdf, PluggableErasureCodec v4.pdf
>
>
> According to HDFS-7285 and the design, this considers to support multiple
> Erasure Codecs via pluggable approach. It allows to define and configure
> multiple codec schemas with different coding algorithms and parameters. The
> resultant codec schemas can be utilized and specified via command tool for
> different file folders. While design and implement such pluggable framework,
> it’s also to implement a concrete codec by default (Reed Solomon) to prove
> the framework is useful and workable. Separate JIRA could be opened for the
> RS codec implementation.
> Note HDFS-7353 will focus on the very low level codec API and implementation
> to make concrete vendor libraries transparent to the upper layer. This JIRA
> focuses on high level stuffs that interact with configuration, schema and etc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
