[jira] [Comment Edited] (HADOOP-13200) Seeking a better approach allowing to customize and configure erasure coders
[ https://issues.apache.org/jira/browse/HADOOP-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985957#comment-15985957 ] Tim Yao edited comment on HADOOP-13200 at 4/27/17 3:51 AM: --- In the new patch HADOOP-13200.11.patch, the above configuration errors are all fixed. Only simple modifications are done by replacing NativeRSRawErasureCoderFactory.class.getCanonicalName() with NativeRSRawErasureCoderFactory.CODER_NAME, which shall not cause any additional problems. was (Author: timmyyao): In the new patch HADOOP-13200.11.patch, the above configuration errors are all fixed. Only simple modifications are done by replace NativeRSRawErasureCoderFactory.class.getCanonicalName() with NativeRSRawErasureCoderFactory.CODER_NAME, which shall not cause any additional problems. > Seeking a better approach allowing to customize and configure erasure coders > > > Key: HADOOP-13200 > URL: https://issues.apache.org/jira/browse/HADOOP-13200 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Tim Yao >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HADOOP-13200.02.patch, HADOOP-13200.03.patch, > HADOOP-13200.04.patch, HADOOP-13200.05.patch, HADOOP-13200.06.patch, > HADOOP-13200.07.patch, HADOOP-13200.08.patch, HADOOP-13200.09.patch, > HADOOP-13200.10.patch, HADOOP-13200.11.patch > > > This is a follow-on task for HADOOP-13010 as discussed over there. There may > be some better approach allowing to customize and configure erasure coders > than the current having raw coder factory, as [~cmccabe] suggested. Will copy > the relevant comments here to continue the discussion. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13200) Seeking a better approach allowing to customize and configure erasure coders
[
https://issues.apache.org/jira/browse/HADOOP-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985929#comment-15985929
]
Tim Yao edited comment on HADOOP-13200 at 4/27/17 3:39 AM:
---
I have double checked the code and don't think it's related to this patch
either, as CodecUtil is never used in this class.
By the way, I examined some other bugs when checking all the usages of
CodecUtil. As the configuration of erasure code has been changed, some
configurations should be changed accordingly in unit tests. All those bugs lie
in the codes like the following:
{quote}
if (ErasureCodeNative.isNativeCodeLoaded()) {
conf.set(
CodecUtil.IO_ERASURECODE_CODEC_RS_RAWCODERS_KEY,
NativeRSRawErasureCoderFactory.class.getCanonicalName());
}
{quote}
The configuration is done when native code is loaded, so currently there is no
error displayed. I will correct them immediately.
was (Author: timmyyao):
I have double checked the code and don't think it's related to this patch
either, as CodecUtil is never used in this class.
By the way, I examined some other bugs when checking all the usages of
CodecUtil. As the configuration of erasure code has been changed, some
configurations should be changed accordingly in unit tests. The bugs lie in the
codes like the following:
{quote}
if (ErasureCodeNative.isNativeCodeLoaded()) {
conf.set(
CodecUtil.IO_ERASURECODE_CODEC_RS_RAWCODERS_KEY,
NativeRSRawErasureCoderFactory.class.getCanonicalName());
}
{quote}
The configuration is done when native code is loaded, so currently there is no
error displayed. I will correct them immediately.
> Seeking a better approach allowing to customize and configure erasure coders
>
>
> Key: HADOOP-13200
> URL: https://issues.apache.org/jira/browse/HADOOP-13200
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Tim Yao
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13200.02.patch, HADOOP-13200.03.patch,
> HADOOP-13200.04.patch, HADOOP-13200.05.patch, HADOOP-13200.06.patch,
> HADOOP-13200.07.patch, HADOOP-13200.08.patch, HADOOP-13200.09.patch,
> HADOOP-13200.10.patch
>
>
> This is a follow-on task for HADOOP-13010 as discussed over there. There may
> be some better approach allowing to customize and configure erasure coders
> than the current having raw coder factory, as [~cmccabe] suggested. Will copy
> the relevant comments here to continue the discussion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13200) Seeking a better approach allowing to customize and configure erasure coders
[
https://issues.apache.org/jira/browse/HADOOP-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985929#comment-15985929
]
Tim Yao edited comment on HADOOP-13200 at 4/27/17 3:37 AM:
---
I have double checked the code and don't think it's related to this patch
either, as CodecUtil is never used in this class.
By the way, I examined some other bugs when checking all the usages of
CodecUtil. As the configuration of erasure code has been changed, some
configurations should be changed accordingly in unit tests. The bugs lie in the
codes like the following:
{quote}
if (ErasureCodeNative.isNativeCodeLoaded()) {
conf.set(
CodecUtil.IO_ERASURECODE_CODEC_RS_RAWCODERS_KEY,
NativeRSRawErasureCoderFactory.class.getCanonicalName());
}
{quote}
The configuration is done when native code is loaded, so currently there is no
error displayed. I will correct them immediately.
was (Author: timmyyao):
I have double checked the code and don't think it's related to this patch
either, as CodecUtil is never used in this class.
By the way, I examined some other bugs when checking all the usages of
CodecUtil. As the configuration of erasure code has been changed, some
configurations should be changed accordingly. The bugs lie in the codes like
the following:
{quote}
if (ErasureCodeNative.isNativeCodeLoaded()) {
conf.set(
CodecUtil.IO_ERASURECODE_CODEC_RS_RAWCODERS_KEY,
NativeRSRawErasureCoderFactory.class.getCanonicalName());
}
{quote}
The configuration is done when native code is loaded, so currently there is no
error displayed. I will correct them immediately.
> Seeking a better approach allowing to customize and configure erasure coders
>
>
> Key: HADOOP-13200
> URL: https://issues.apache.org/jira/browse/HADOOP-13200
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Tim Yao
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13200.02.patch, HADOOP-13200.03.patch,
> HADOOP-13200.04.patch, HADOOP-13200.05.patch, HADOOP-13200.06.patch,
> HADOOP-13200.07.patch, HADOOP-13200.08.patch, HADOOP-13200.09.patch,
> HADOOP-13200.10.patch
>
>
> This is a follow-on task for HADOOP-13010 as discussed over there. There may
> be some better approach allowing to customize and configure erasure coders
> than the current having raw coder factory, as [~cmccabe] suggested. Will copy
> the relevant comments here to continue the discussion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13200) Seeking a better approach allowing to customize and configure erasure coders
[
https://issues.apache.org/jira/browse/HADOOP-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985929#comment-15985929
]
Tim Yao edited comment on HADOOP-13200 at 4/27/17 3:36 AM:
---
I have double checked the code and don't think it's related to this patch
either, as CodecUtil is never used in this class.
By the way, I examined some other bugs when checking all the usages of
CodecUtil. As the configuration of erasure code has been changed, some
configurations should be changed accordingly. The bugs lie in the codes like
the following:
{quote}
if (ErasureCodeNative.isNativeCodeLoaded()) {
conf.set(
CodecUtil.IO_ERASURECODE_CODEC_RS_RAWCODERS_KEY,
NativeRSRawErasureCoderFactory.class.getCanonicalName());
}
{quote}
The configuration is done when native code is loaded, so currently there is no
error displayed. I will correct them immediately.
was (Author: timmyyao):
I have double checked the code and don't think it's related to this patch
either, as CodecUtil is never used in this class.
By the way, I examined some other bugs when checking all the usages of
CodecUtil. As the configuration of erasure code has been changed, some
configurations should be changed accordingly. The bugs lie in the codes like
the following:
{quote}
if (ErasureCodeNative.isNativeCodeLoaded()) {
conf.set(
CodecUtil.IO_ERASURECODE_CODEC_RS_RAWCODERS_KEY,
NativeRSRawErasureCoderFactory.class.getCanonicalName());
}
{quote}
The configuration is done when native code is loaded, so currently there is no
error displayed. I will correct them immediately.
> Seeking a better approach allowing to customize and configure erasure coders
>
>
> Key: HADOOP-13200
> URL: https://issues.apache.org/jira/browse/HADOOP-13200
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Tim Yao
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13200.02.patch, HADOOP-13200.03.patch,
> HADOOP-13200.04.patch, HADOOP-13200.05.patch, HADOOP-13200.06.patch,
> HADOOP-13200.07.patch, HADOOP-13200.08.patch, HADOOP-13200.09.patch,
> HADOOP-13200.10.patch
>
>
> This is a follow-on task for HADOOP-13010 as discussed over there. There may
> be some better approach allowing to customize and configure erasure coders
> than the current having raw coder factory, as [~cmccabe] suggested. Will copy
> the relevant comments here to continue the discussion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13200) Seeking a better approach allowing to customize and configure erasure coders
[
https://issues.apache.org/jira/browse/HADOOP-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985929#comment-15985929
]
Tim Yao edited comment on HADOOP-13200 at 4/27/17 3:32 AM:
---
I have double checked the code and don't think it's related to this patch
either, as CodecUtil is never used in this class.
By the way, I examined some other bugs when checking all the usages of
CodecUtil. As the configuration of erasure code has been changed, some
configurations should be changed accordingly. The bugs lie in the codes like
the following:
{quote}
if (ErasureCodeNative.isNativeCodeLoaded()) {
conf.set(
CodecUtil.IO_ERASURECODE_CODEC_RS_RAWCODERS_KEY,
NativeRSRawErasureCoderFactory.class.getCanonicalName());
}
{quote}
The configuration is done when native code is loaded, so currently there is no
error displayed. I will correct them immediately.
was (Author: timmyyao):
I have double checked the code and don't think it's related to this patch
either, as CodecUtil is never used in this class.
By the way, I examined some other bugs when checking all the usages of
CodecUtil. As the configuration of erasure code has been changed, some
configurations should be changed accordingly. The bugs lie in the codes like
the following:
{quote}
if (ErasureCodeNative.isNativeCodeLoaded()) {
conf.set(
CodecUtil.IO_ERASURECODE_CODEC_RS_RAWCODERS_KEY,
NativeRSRawErasureCoderFactory.class.getCanonicalName());
}
{quote}
The configuration is done when native code is loaded, so currently there is no
error displayed. I will correct them immediately.
> Seeking a better approach allowing to customize and configure erasure coders
>
>
> Key: HADOOP-13200
> URL: https://issues.apache.org/jira/browse/HADOOP-13200
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Tim Yao
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13200.02.patch, HADOOP-13200.03.patch,
> HADOOP-13200.04.patch, HADOOP-13200.05.patch, HADOOP-13200.06.patch,
> HADOOP-13200.07.patch, HADOOP-13200.08.patch, HADOOP-13200.09.patch,
> HADOOP-13200.10.patch
>
>
> This is a follow-on task for HADOOP-13010 as discussed over there. There may
> be some better approach allowing to customize and configure erasure coders
> than the current having raw coder factory, as [~cmccabe] suggested. Will copy
> the relevant comments here to continue the discussion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13200) Seeking a better approach allowing to customize and configure erasure coders
[
https://issues.apache.org/jira/browse/HADOOP-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985929#comment-15985929
]
Tim Yao edited comment on HADOOP-13200 at 4/27/17 3:32 AM:
---
I have double checked the code and don't think it's related to this patch
either, as CodecUtil is never used in this class.
By the way, I examined some other bugs when checking all the usages of
CodecUtil. As the configuration of erasure code has been changed, some
configurations should be changed accordingly. The bugs lie in the codes like
the following:
{quote}
if (ErasureCodeNative.isNativeCodeLoaded()) {
conf.set(
CodecUtil.IO_ERASURECODE_CODEC_RS_RAWCODERS_KEY,
NativeRSRawErasureCoderFactory.class.getCanonicalName());
}
{quote}
The configuration is done when native code is loaded, so currently there is no
error displayed. I will correct them immediately.
was (Author: timmyyao):
I have double checked the code and don't think it's related to this patch
either, as CodecUtil is never used in this class.
By the way, I examined some other bugs when checking all the usages of
CodecUtil. As the configuration of erasure code has been changed, some
configurations should be changed accordingly. The bugs lie in the codes like
the following:
{quote}
if (ErasureCodeNative.isNativeCodeLoaded()) {
conf.set(
CodecUtil.IO_ERASURECODE_CODEC_RS_RAWCODERS_KEY,
NativeRSRawErasureCoderFactory.class.getCanonicalName());
}
{quote}
The configuration is done when native code is loaded, so currently there is no
error displayed. I will correct them immediately.
> Seeking a better approach allowing to customize and configure erasure coders
>
>
> Key: HADOOP-13200
> URL: https://issues.apache.org/jira/browse/HADOOP-13200
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Tim Yao
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13200.02.patch, HADOOP-13200.03.patch,
> HADOOP-13200.04.patch, HADOOP-13200.05.patch, HADOOP-13200.06.patch,
> HADOOP-13200.07.patch, HADOOP-13200.08.patch, HADOOP-13200.09.patch,
> HADOOP-13200.10.patch
>
>
> This is a follow-on task for HADOOP-13010 as discussed over there. There may
> be some better approach allowing to customize and configure erasure coders
> than the current having raw coder factory, as [~cmccabe] suggested. Will copy
> the relevant comments here to continue the discussion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13200) Seeking a better approach allowing to customize and configure erasure coders
[
https://issues.apache.org/jira/browse/HADOOP-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978968#comment-15978968
]
Wei-Chiu Chuang edited comment on HADOOP-13200 at 4/21/17 4:01 PM:
---
I saw a few nits in the 06 patch:
* Could you switch to slf4j? That way you can avoid {{LOG.isDebugEnabled()}}
and can use curl based parameterization.
* Would it be possible to add a test for the case where two coders register the
same name?
* The following code
{code}
" cannot be registered because its coder name " + coderFactory
{code}
shouldn't it be coderFactory.getCoderName()?
* Still in CodecRegistry constructor. I think you want to continue instead of
break if a coder has conflict. Otherwise you would throw an exception instead
of just log an error message.
was (Author: jojochuang):
I saw a few nits in the 06 patch:
* Could you switch to slf4j? That way you can avoid {{LOG.isDebugEnabled()}}
and can use curl based parameterization.
* Could you add a test for the case where two coders register the same name?
* The following code
{code}
" cannot be registered because its coder name " + coderFactory
{code}
shouldn't it be coderFactory.getCoderName()?
> Seeking a better approach allowing to customize and configure erasure coders
>
>
> Key: HADOOP-13200
> URL: https://issues.apache.org/jira/browse/HADOOP-13200
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Tim Yao
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13200.02.patch, HADOOP-13200.03.patch,
> HADOOP-13200.04.patch, HADOOP-13200.05.patch, HADOOP-13200.06.patch
>
>
> This is a follow-on task for HADOOP-13010 as discussed over there. There may
> be some better approach allowing to customize and configure erasure coders
> than the current having raw coder factory, as [~cmccabe] suggested. Will copy
> the relevant comments here to continue the discussion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
