[
https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020209#comment-16020209
]
Lei (Eddy) Xu commented on HDFS-7337:
-------------------------------------
Hi, [~drankye] and [~Sammi]
Thanks a lot for the reply. The explanation helps a lot.
bq. There are several system wide codecs to use, including RS codec, RS legacy
codec and XOR codec.
Is there a way to choose a system-wide *_default_* codec? So that after the
cluster being initialized, users and admins can just specify a zone /
directory to be "erasure coded", instead of choosing from several different
codes, and each one has its own trade-offs, which require user / admin to
understand?
bq. while it supports to add / remove policies using CLI, it dose not support
to enable / use the policy via CLI?
My concern is that, even if the admin is able to add policy via the API
*dynamically*, it still requires the admin to reboot NN, or ssh into NN /
change conf files and reload NN confs, to enable the policy? It makes the
workflow complicated. I think using API / CLI and ssh NN / changing conf files
should be two different sets of operations. If possible, it is more consistent
to do the EC policy management in either one, or both. The current design is
doing half of the management in each approach.
Thanks.
> Configurable and pluggable Erasure Codec and schema
> ---------------------------------------------------
>
> Key: HDFS-7337
> URL: https://issues.apache.org/jira/browse/HDFS-7337
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: erasure-coding
> Reporter: Zhe Zhang
> Priority: Critical
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-7337-prototype-v1.patch,
> HDFS-7337-prototype-v2.zip, HDFS-7337-prototype-v3.zip,
> PluggableErasureCodec.pdf, PluggableErasureCodec-v2.pdf,
> PluggableErasureCodec-v3.pdf, PluggableErasureCodec v4.pdf
>
>
> According to HDFS-7285 and the design, this considers to support multiple
> Erasure Codecs via pluggable approach. It allows to define and configure
> multiple codec schemas with different coding algorithms and parameters. The
> resultant codec schemas can be utilized and specified via command tool for
> different file folders. While design and implement such pluggable framework,
> it’s also to implement a concrete codec by default (Reed Solomon) to prove
> the framework is useful and workable. Separate JIRA could be opened for the
> RS codec implementation.
> Note HDFS-7353 will focus on the very low level codec API and implementation
> to make concrete vendor libraries transparent to the upper layer. This JIRA
> focuses on high level stuffs that interact with configuration, schema and etc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
