[
https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200140#comment-14200140
]
Kai Zheng commented on HDFS-7337:
---------------------------------
The points made in the prototype patch are:
* Multiple erasure codecs can be configured and referenced by their names;
* Multiple erasure codec instances or schemas can be defined with kinds of
options in schema file, and can be specified via their distinguished names;
* ErasureCodec takes care of two aspects, ECSchema for NameNode/ECManager, and
ErasureCoder for DataNode/ECWorker;
* ECSchema is loaded from configuration and can also be persisted in compact
form to be passed to DataNode if desired;
* ErasureCodec is also responsible fro calculating BlockGroup given required
original data blocks and to be computed parity blocks;
* ErasureCoder can be initialized with options from schema and performs
basically encoding/decoding of ECChunks;
* ErasureCoder can be implemented using Jerasure library or Intel ISA library.
The concrete coder should only be created in DataNode side, thus corresponding
libraries are only required in DataNodes. NameNode doesn't need to create
coders;
* RS codec and LRC codec with corresponding coders are to be supported, as
they're typical cases for such API definition;
* RS and LRC coder implementations will be provided by default using Intel ISA
library.
> Configurable and pluggable Erasure Codec and schema
> ---------------------------------------------------
>
> Key: HDFS-7337
> URL: https://issues.apache.org/jira/browse/HDFS-7337
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Zhe Zhang
> Assignee: Kai Zheng
> Attachments: HDFS-7337-prototype-v1.patch
>
>
> According to HDFS-7285 and the design, this considers to support multiple
> Erasure Codecs via pluggable approach. It allows to define and configure
> multiple codec schemas with different coding algorithms and parameters. The
> resultant codec schemas can be utilized and specified via command tool for
> different file folders. While design and implement such pluggable framework,
> it’s also to implement a concrete codec by default (Reed Solomon) to prove
> the framework is useful and workable. Separate JIRA could be opened for the
> RS codec implementation.
> Note HDFS-7353 will focus on the very low level codec API and implementation
> to make concrete vendor libraries transparent to the upper layer. This JIRA
> focuses on high level stuffs that interact with configuration, schema and etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
