[
https://issues.apache.org/jira/browse/HADOOP-13665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15932055#comment-15932055
]
Kai Zheng commented on HADOOP-13665:
------------------------------------
Thanks all for the discussion and I took a quick look.
{code}
+ conf.set(CodecUtil.IO_ERASURECODE_CODEC_CODERS,
+ ErasureCodeConstants.RS_CODEC_NAME + "," +
+ ErasureCodeConstants.XOR_CODEC_NAME);
{code}
It doesn't make sense to mix and configure something for RS codec and XOR codec
together. Instead, it should be to allow configuring more coders for the same
codec in order, IIUC.
For example, for the RS codec, we have the pure Java coder and also the ISA-L
based native coder. As Chuang proposed, what we want to do is to allow
configuring the two coders using the property like
{{io.erasurecode.codec.rs.rawcoders}} in addition to what we have already as
below
{code}
/** Raw coder factory for the RS codec. */
public static final String IO_ERASURECODE_CODEC_RS_RAWCODER_KEY =
"io.erasurecode.codec.rs.rawcoder";
public static final String IO_ERASURECODE_CODEC_RS_RAWCODER_DEFAULT =
RSRawErasureCoderFactory.class.getCanonicalName();
/** Raw coder factory for the RS legacy codec. */
public static final String IO_ERASURECODE_CODEC_RS_LEGACY_RAWCODER_KEY =
"io.erasurecode.codec.rs-legacy.rawcoder";
public static final String IO_ERASURECODE_CODEC_RS_LEGACY_RAWCODER_DEFAULT =
RSRawErasureCoderFactoryLegacy.class.getCanonicalName();
/** Raw coder factory for the XOR codec. */
public static final String IO_ERASURECODE_CODEC_XOR_RAWCODER_KEY =
"io.erasurecode.codec.xor.rawcoder";
public static final String IO_ERASURECODE_CODEC_XOR_RAWCODER_DEFAULT =
XORRawErasureCoderFactory.class.getCanonicalName();
{code}
> Erasure Coding codec should support fallback coder
> --------------------------------------------------
>
> Key: HADOOP-13665
> URL: https://issues.apache.org/jira/browse/HADOOP-13665
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: io
> Reporter: Wei-Chiu Chuang
> Assignee: Kai Sasaki
> Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13665.01.patch, HADOOP-13665.02.patch,
> HADOOP-13665.03.patch, HADOOP-13665.04.patch, HADOOP-13665.05.patch
>
>
> The current EC codec supports a single coder only (by default pure Java
> implementation). If the native coder is specified but is unavailable, it
> should fallback to pure Java implementation.
> One possible solution is to follow the convention of existing Hadoop native
> codec, such as transport encryption (see {{CryptoCodec.java}}). It supports
> fallback by specifying two or multiple coders as the value of property, and
> loads coders in order.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
