[
https://issues.apache.org/jira/browse/HADOOP-13665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944991#comment-15944991
]
Kai Zheng commented on HADOOP-13665:
------------------------------------
Thanks Kai for the update!
{code}
+ private static String getRawCoders(Configuration conf, String codec) {
+ switch (codec) {
+ case ErasureCodeConstants.RS_CODEC_NAME:
+ case ErasureCodeConstants.RS_LEGACY_CODEC_NAME:
+ return conf.get(
+ IO_ERASURECODE_CODEC_RS_RAWCODERS_KEY,
+ IO_ERASURECODE_CODEC_RS_RAWCODERS_DEFAULT);
{code}
1. RS_CODEC and RS_LEGACY_CODEC should be different codecs. They both need to
be configured which raw coders to use.
2. The {{switch}} could be avoided here. To be simple, I think we can reduce
the property key name according to the codec name. For example, if the codec
name is {{xcodec}}, then the property key to be used to get the raw coders can
be: {{"io.erasurecode.codec." + xcodec + ".rawcoders"}};
{code}
+ private static String getRawCoderFactNameWithFallback(Configuration conf,
+ String codecs) {
+ for (String codec : Splitter.on(",").split(codecs)) {
+ String factName = getRawCoderFactNameFromCodec(conf, codec);
+ if (factName != null) {
+ return factName;
+ }
+ // In case of not found raw coder, fallback to next
+ }
+ throw new IllegalArgumentException("Raw coder factory not configured " +
+ "for custom codecs: " + codecs);
+ }
{code}
Just calling {{getRawCoderFactNameFromCodec}} may be not enough, the failure
can happen during coder instance creating, for example, if native library
loading failed.
An elegant way could be to come up a composite RawErasureCoderFactory that
wraps and favors the configured raw coders list one by one during
{{createEncoder/Decoder}} call.
> Erasure Coding codec should support fallback coder
> --------------------------------------------------
>
> Key: HADOOP-13665
> URL: https://issues.apache.org/jira/browse/HADOOP-13665
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: io
> Reporter: Wei-Chiu Chuang
> Assignee: Kai Sasaki
> Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13665.01.patch, HADOOP-13665.02.patch,
> HADOOP-13665.03.patch, HADOOP-13665.04.patch, HADOOP-13665.05.patch,
> HADOOP-13665.06.patch, HADOOP-13665.07.patch
>
>
> The current EC codec supports a single coder only (by default pure Java
> implementation). If the native coder is specified but is unavailable, it
> should fallback to pure Java implementation.
> One possible solution is to follow the convention of existing Hadoop native
> codec, such as transport encryption (see {{CryptoCodec.java}}). It supports
> fallback by specifying two or multiple coders as the value of property, and
> loads coders in order.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
