[jira] [Updated] (HDFS-12046) Hadoop CRC implementation using Intel ISA-L library
[ https://issues.apache.org/jira/browse/HDFS-12046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-12046: - Status: Patch Available (was: Open) > Hadoop CRC implementation using Intel ISA-L library > --- > > Key: HDFS-12046 > URL: https://issues.apache.org/jira/browse/HDFS-12046 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: luhuichun >Assignee: luhuichun > Attachments: HDFS-12046-001.patch, ISA-L CRC Performance Report using > intel ISA-L.pdf > > > Intel ISA-L open source library provides set of highly optimized functions > for RAID, erasure code, CRC, cryptographic hash, encryption, and compression. > Ref. https://github.com/01org/isa-l. HDFS-EC has already integrated ISA-L and > added the necessary building options support for Hadoop. For Hadoop CRC, we > recently explored more, developing a Hadoop CRC using Intel ISA-L, performing > a test on Broadwell and Skylake servers, comparing the performance against > Hadoop native CRC. On Broadwell/Skylake, ISA-L CRC has about 8%~ performance > gain over Hadoop native CRC. We suggest adding a new Hadoop native CRC using > the ISA-L library, the extra advantage is it’s already optimized when we > upgrade to new servers and Hadoop developers don’t have to maintain their own > bunch of ASM codes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11975) Provide a system-default EC policy
[
https://issues.apache.org/jira/browse/HDFS-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
luhuichun updated HDFS-11975:
-
Attachment: HDFS-11975-009.patch
[~Sammi] Hi Sammi, updated according to your comments
> Provide a system-default EC policy
> --
>
> Key: HDFS-11975
> URL: https://issues.apache.org/jira/browse/HDFS-11975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: luhuichun
> Attachments: HDFS-11975-001.patch, HDFS-11975-002.patch,
> HDFS-11975-003.patch, HDFS-11975-004.patch, HDFS-11975-005.patch,
> HDFS-11975-006.patch, HDFS-11975-007.patch, HDFS-11975-008.patch,
> HDFS-11975-009.patch
>
>
> From the usability point of view, it'd be nice to be able to specify a
> system-wide EC policy, i.e., in {{hdfs-site.xml}}. For most of users / admin
> / downstream projects, it is not necessary to know the tradeoffs of the EC
> policy, considering that it requires the knowledge of EC, the actual physical
> topology of the clusters, and many other factors (i.e., network, cluster size
> and etc).
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11975) Provide a system-default EC policy
[
https://issues.apache.org/jira/browse/HDFS-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
luhuichun updated HDFS-11975:
-
Attachment: HDFS-11975-008.patch
update with small change
> Provide a system-default EC policy
> --
>
> Key: HDFS-11975
> URL: https://issues.apache.org/jira/browse/HDFS-11975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: luhuichun
> Attachments: HDFS-11975-001.patch, HDFS-11975-002.patch,
> HDFS-11975-003.patch, HDFS-11975-004.patch, HDFS-11975-005.patch,
> HDFS-11975-006.patch, HDFS-11975-007.patch, HDFS-11975-008.patch
>
>
> From the usability point of view, it'd be nice to be able to specify a
> system-wide EC policy, i.e., in {{hdfs-site.xml}}. For most of users / admin
> / downstream projects, it is not necessary to know the tradeoffs of the EC
> policy, considering that it requires the knowledge of EC, the actual physical
> topology of the clusters, and many other factors (i.e., network, cluster size
> and etc).
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12046) Hadoop CRC implementation using Intel ISA-L library
[ https://issues.apache.org/jira/browse/HDFS-12046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-12046: - Summary: Hadoop CRC implementation using Intel ISA-L library (was: Hadoop CRC implementation using Intel ISAL library) > Hadoop CRC implementation using Intel ISA-L library > --- > > Key: HDFS-12046 > URL: https://issues.apache.org/jira/browse/HDFS-12046 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: luhuichun >Assignee: luhuichun > Attachments: ISA-L CRC Performance Report using intel ISA-L.pdf > > > Intel ISA-L open source library provides set of highly optimized functions > for RAID, erasure code, CRC, cryptographic hash, encryption, and compression. > Ref. https://github.com/01org/isa-l. HDFS-EC has already integrated ISA-L and > added the necessary building options support for Hadoop. For Hadoop CRC, we > recently explored more, developing a Hadoop CRC using Intel ISA-L, performing > a test on Broadwell and Skylake servers, comparing the performance against > Hadoop native CRC. On Broadwell/Skylake, ISA-L CRC has about 8%~ performance > gain over Hadoop native CRC. We suggest adding a new Hadoop native CRC using > the ISA-L library, the extra advantage is it’s already optimized when we > upgrade to new servers and Hadoop developers don’t have to maintain their own > bunch of ASM codes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12046) Hadoop CRC implementation using Intel ISA-L library
[ https://issues.apache.org/jira/browse/HDFS-12046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-12046: - Attachment: ISA-L CRC Performance Report using intel ISA-L.pdf > Hadoop CRC implementation using Intel ISA-L library > --- > > Key: HDFS-12046 > URL: https://issues.apache.org/jira/browse/HDFS-12046 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: luhuichun >Assignee: luhuichun > Attachments: ISA-L CRC Performance Report using intel ISA-L.pdf > > > Intel ISA-L open source library provides set of highly optimized functions > for RAID, erasure code, CRC, cryptographic hash, encryption, and compression. > Ref. https://github.com/01org/isa-l. HDFS-EC has already integrated ISA-L and > added the necessary building options support for Hadoop. For Hadoop CRC, we > recently explored more, developing a Hadoop CRC using Intel ISA-L, performing > a test on Broadwell and Skylake servers, comparing the performance against > Hadoop native CRC. On Broadwell/Skylake, ISA-L CRC has about 8%~ performance > gain over Hadoop native CRC. We suggest adding a new Hadoop native CRC using > the ISA-L library, the extra advantage is it’s already optimized when we > upgrade to new servers and Hadoop developers don’t have to maintain their own > bunch of ASM codes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12046) Hadoop CRC implementation using Intel ISA-L library
[ https://issues.apache.org/jira/browse/HDFS-12046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-12046: - Attachment: (was: ISA-L CRC Performance Report using intel ISA-L.pdf) > Hadoop CRC implementation using Intel ISA-L library > --- > > Key: HDFS-12046 > URL: https://issues.apache.org/jira/browse/HDFS-12046 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: luhuichun >Assignee: luhuichun > > Intel ISA-L open source library provides set of highly optimized functions > for RAID, erasure code, CRC, cryptographic hash, encryption, and compression. > Ref. https://github.com/01org/isa-l. HDFS-EC has already integrated ISA-L and > added the necessary building options support for Hadoop. For Hadoop CRC, we > recently explored more, developing a Hadoop CRC using Intel ISA-L, performing > a test on Broadwell and Skylake servers, comparing the performance against > Hadoop native CRC. On Broadwell/Skylake, ISA-L CRC has about 8%~ performance > gain over Hadoop native CRC. We suggest adding a new Hadoop native CRC using > the ISA-L library, the extra advantage is it’s already optimized when we > upgrade to new servers and Hadoop developers don’t have to maintain their own > bunch of ASM codes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11975) Provide a system-default EC policy
[
https://issues.apache.org/jira/browse/HDFS-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
luhuichun updated HDFS-11975:
-
Attachment: HDFS-11975-002.patch
> Provide a system-default EC policy
> --
>
> Key: HDFS-11975
> URL: https://issues.apache.org/jira/browse/HDFS-11975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: luhuichun
> Attachments: HDFS-11975-001.patch, HDFS-11975-002.patch
>
>
> From the usability point of view, it'd be nice to be able to specify a
> system-wide EC policy, i.e., in {{hdfs-site.xml}}. For most of users / admin
> / downstream projects, it is not necessary to know the tradeoffs of the EC
> policy, considering that it requires the knowledge of EC, the actual physical
> topology of the clusters, and many other factors (i.e., network, cluster size
> and etc).
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11975) Provide a system-default EC policy
[
https://issues.apache.org/jira/browse/HDFS-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
luhuichun updated HDFS-11975:
-
Attachment: HDFS-11975-004.patch
> Provide a system-default EC policy
> --
>
> Key: HDFS-11975
> URL: https://issues.apache.org/jira/browse/HDFS-11975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: luhuichun
> Attachments: HDFS-11975-001.patch, HDFS-11975-002.patch,
> HDFS-11975-003.patch, HDFS-11975-004.patch
>
>
> From the usability point of view, it'd be nice to be able to specify a
> system-wide EC policy, i.e., in {{hdfs-site.xml}}. For most of users / admin
> / downstream projects, it is not necessary to know the tradeoffs of the EC
> policy, considering that it requires the knowledge of EC, the actual physical
> topology of the clusters, and many other factors (i.e., network, cluster size
> and etc).
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11975) Provide a system-default EC policy
[
https://issues.apache.org/jira/browse/HDFS-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16071874#comment-16071874
]
luhuichun edited comment on HDFS-11975 at 7/4/17 1:27 PM:
--
[~eddyxu]，[~Sammi]，[~drankye] Update the patch, the failed unit test is solved.
For the issue of system-default EC policy, I add two const in
DFSConfigKeys.java. One for user setting, the other is hard code default value
"RS-6-3-64k". The NameNodeRpcServer do the judgement according to the
configuration and user CLI. Thanks for reviewing.
was (Author: luhuichun):
[~eddyxu][~Sammi][~drankye] Update the patch, the failed unit test is solved.
For the issue of system-default EC policy, I set a two configuration in
DFSConfigKeys.java. One for user setting, the other is hard code default value
"RS-6-3-64k". The NameNodeRpcServer do the judgement according to the
configuration and user CLI. Thanks for reviewing.
> Provide a system-default EC policy
> --
>
> Key: HDFS-11975
> URL: https://issues.apache.org/jira/browse/HDFS-11975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: luhuichun
> Attachments: HDFS-11975-001.patch, HDFS-11975-002.patch,
> HDFS-11975-003.patch, HDFS-11975-004.patch
>
>
> From the usability point of view, it'd be nice to be able to specify a
> system-wide EC policy, i.e., in {{hdfs-site.xml}}. For most of users / admin
> / downstream projects, it is not necessary to know the tradeoffs of the EC
> policy, considering that it requires the knowledge of EC, the actual physical
> topology of the clusters, and many other factors (i.e., network, cluster size
> and etc).
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11975) Provide a system-default EC policy
[
https://issues.apache.org/jira/browse/HDFS-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
luhuichun updated HDFS-11975:
-
Attachment: HDFS-11975-003.patch
> Provide a system-default EC policy
> --
>
> Key: HDFS-11975
> URL: https://issues.apache.org/jira/browse/HDFS-11975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: luhuichun
> Attachments: HDFS-11975-001.patch, HDFS-11975-002.patch,
> HDFS-11975-003.patch
>
>
> From the usability point of view, it'd be nice to be able to specify a
> system-wide EC policy, i.e., in {{hdfs-site.xml}}. For most of users / admin
> / downstream projects, it is not necessary to know the tradeoffs of the EC
> policy, considering that it requires the knowledge of EC, the actual physical
> topology of the clusters, and many other factors (i.e., network, cluster size
> and etc).
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11975) Provide a system-default EC policy
[
https://issues.apache.org/jira/browse/HDFS-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16071874#comment-16071874
]
luhuichun commented on HDFS-11975:
--
[~eddyxu][~Sammi][~drankye] Update the patch, the failed unit test is solved.
For the issue of system-default EC policy, I set a two configuration in
DFSConfigKeys.java. One for user setting, the other is hard code default value
"RS-6-3-64k". The NameNodeRpcServer do the judgement according to the
configuration and user CLI. Thanks for reviewing.
> Provide a system-default EC policy
> --
>
> Key: HDFS-11975
> URL: https://issues.apache.org/jira/browse/HDFS-11975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: luhuichun
> Attachments: HDFS-11975-001.patch, HDFS-11975-002.patch,
> HDFS-11975-003.patch
>
>
> From the usability point of view, it'd be nice to be able to specify a
> system-wide EC policy, i.e., in {{hdfs-site.xml}}. For most of users / admin
> / downstream projects, it is not necessary to know the tradeoffs of the EC
> policy, considering that it requires the knowledge of EC, the actual physical
> topology of the clusters, and many other factors (i.e., network, cluster size
> and etc).
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11975) Provide a system-default EC policy
[
https://issues.apache.org/jira/browse/HDFS-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
luhuichun updated HDFS-11975:
-
Attachment: HDFS-11975-005.patch
update with some minor change
> Provide a system-default EC policy
> --
>
> Key: HDFS-11975
> URL: https://issues.apache.org/jira/browse/HDFS-11975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: luhuichun
> Attachments: HDFS-11975-001.patch, HDFS-11975-002.patch,
> HDFS-11975-003.patch, HDFS-11975-004.patch, HDFS-11975-005.patch
>
>
> From the usability point of view, it'd be nice to be able to specify a
> system-wide EC policy, i.e., in {{hdfs-site.xml}}. For most of users / admin
> / downstream projects, it is not necessary to know the tradeoffs of the EC
> policy, considering that it requires the knowledge of EC, the actual physical
> topology of the clusters, and many other factors (i.e., network, cluster size
> and etc).
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11975) Provide a system-default EC policy
[
https://issues.apache.org/jira/browse/HDFS-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
luhuichun updated HDFS-11975:
-
Attachment: HDFS-11975-006.patch
we find we can set optional ecPolicyName in erasurecoding.proto and remove
"Null check" in ClientNamenodeProtocolTranslatorPB and
ClientNamenodeProtocolServerSideTranslatorPB, this can be avoid of passing the
const string which is a not better solution.
> Provide a system-default EC policy
> --
>
> Key: HDFS-11975
> URL: https://issues.apache.org/jira/browse/HDFS-11975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: luhuichun
> Attachments: HDFS-11975-001.patch, HDFS-11975-002.patch,
> HDFS-11975-003.patch, HDFS-11975-004.patch, HDFS-11975-005.patch,
> HDFS-11975-006.patch
>
>
> From the usability point of view, it'd be nice to be able to specify a
> system-wide EC policy, i.e., in {{hdfs-site.xml}}. For most of users / admin
> / downstream projects, it is not necessary to know the tradeoffs of the EC
> policy, considering that it requires the knowledge of EC, the actual physical
> topology of the clusters, and many other factors (i.e., network, cluster size
> and etc).
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11975) Provide a system-default EC policy
[
https://issues.apache.org/jira/browse/HDFS-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
luhuichun updated HDFS-11975:
-
Attachment: HDFS-11975-007.patch
update with some small changes
> Provide a system-default EC policy
> --
>
> Key: HDFS-11975
> URL: https://issues.apache.org/jira/browse/HDFS-11975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: luhuichun
> Attachments: HDFS-11975-001.patch, HDFS-11975-002.patch,
> HDFS-11975-003.patch, HDFS-11975-004.patch, HDFS-11975-005.patch,
> HDFS-11975-006.patch, HDFS-11975-007.patch
>
>
> From the usability point of view, it'd be nice to be able to specify a
> system-wide EC policy, i.e., in {{hdfs-site.xml}}. For most of users / admin
> / downstream projects, it is not necessary to know the tradeoffs of the EC
> policy, considering that it requires the knowledge of EC, the actual physical
> topology of the clusters, and many other factors (i.e., network, cluster size
> and etc).
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12046) Hadoop CRC implementation using Intel ISA-L library
[ https://issues.apache.org/jira/browse/HDFS-12046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-12046: - Attachment: HDFS-12046-001.patch > Hadoop CRC implementation using Intel ISA-L library > --- > > Key: HDFS-12046 > URL: https://issues.apache.org/jira/browse/HDFS-12046 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: luhuichun >Assignee: luhuichun > Attachments: HDFS-12046-001.patch > > > Intel ISA-L open source library provides set of highly optimized functions > for RAID, erasure code, CRC, cryptographic hash, encryption, and compression. > Ref. https://github.com/01org/isa-l. HDFS-EC has already integrated ISA-L and > added the necessary building options support for Hadoop. For Hadoop CRC, we > recently explored more, developing a Hadoop CRC using Intel ISA-L, performing > a test on Broadwell and Skylake servers, comparing the performance against > Hadoop native CRC. On Broadwell/Skylake, ISA-L CRC has about 8%~ performance > gain over Hadoop native CRC. We suggest adding a new Hadoop native CRC using > the ISA-L library, the extra advantage is it’s already optimized when we > upgrade to new servers and Hadoop developers don’t have to maintain their own > bunch of ASM codes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11975) Provide a system-default EC policy
[
https://issues.apache.org/jira/browse/HDFS-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
luhuichun updated HDFS-11975:
-
Attachment: HDFS-11975-010.patch
thx [~Sammi] for review comments, updated
> Provide a system-default EC policy
> --
>
> Key: HDFS-11975
> URL: https://issues.apache.org/jira/browse/HDFS-11975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: luhuichun
> Attachments: HDFS-11975-001.patch, HDFS-11975-002.patch,
> HDFS-11975-003.patch, HDFS-11975-004.patch, HDFS-11975-005.patch,
> HDFS-11975-006.patch, HDFS-11975-007.patch, HDFS-11975-008.patch,
> HDFS-11975-009.patch, HDFS-11975-010.patch
>
>
> From the usability point of view, it'd be nice to be able to specify a
> system-wide EC policy, i.e., in {{hdfs-site.xml}}. For most of users / admin
> / downstream projects, it is not necessary to know the tradeoffs of the EC
> policy, considering that it requires the knowledge of EC, the actual physical
> topology of the clusters, and many other factors (i.e., network, cluster size
> and etc).
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12046) Hadoop CRC implementation using Intel ISA-L library
[ https://issues.apache.org/jira/browse/HDFS-12046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-12046: - Attachment: ISA-L CRC Performance Report using intel ISA-L.pdf > Hadoop CRC implementation using Intel ISA-L library > --- > > Key: HDFS-12046 > URL: https://issues.apache.org/jira/browse/HDFS-12046 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: luhuichun >Assignee: luhuichun > Attachments: HDFS-12046-001.patch, ISA-L CRC Performance Report using > intel ISA-L.pdf > > > Intel ISA-L open source library provides set of highly optimized functions > for RAID, erasure code, CRC, cryptographic hash, encryption, and compression. > Ref. https://github.com/01org/isa-l. HDFS-EC has already integrated ISA-L and > added the necessary building options support for Hadoop. For Hadoop CRC, we > recently explored more, developing a Hadoop CRC using Intel ISA-L, performing > a test on Broadwell and Skylake servers, comparing the performance against > Hadoop native CRC. On Broadwell/Skylake, ISA-L CRC has about 8%~ performance > gain over Hadoop native CRC. We suggest adding a new Hadoop native CRC using > the ISA-L library, the extra advantage is it’s already optimized when we > upgrade to new servers and Hadoop developers don’t have to maintain their own > bunch of ASM codes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11646) Add -E option in 'ls' to list erasure coding policy of each file and directory if applicable
[ https://issues.apache.org/jira/browse/HDFS-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11646: - Attachment: (was: HADOOP-11646.patch) > Add -E option in 'ls' to list erasure coding policy of each file and > directory if applicable > > > Key: HDFS-11646 > URL: https://issues.apache.org/jira/browse/HDFS-11646 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11646-001.patch, HDFS-11646-002.patch > > > Add -E option in "ls" to show erasure coding policy of file and directory, > leverage the "number_of_replicas " column. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11646) Add -E option in 'ls' to list erasure coding policy of each file and directory if applicable
[ https://issues.apache.org/jira/browse/HDFS-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11646: - Attachment: HDFS-11646-001.patch > Add -E option in 'ls' to list erasure coding policy of each file and > directory if applicable > > > Key: HDFS-11646 > URL: https://issues.apache.org/jira/browse/HDFS-11646 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HADOOP-11646.patch, HDFS-11646-001.patch > > > Add -E option in "ls" to show erasure coding policy of file and directory, > leverage the "number_of_replicas " column. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11647: - Attachment: HDFS-11647-001.patch > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HADOOP-11647.patch, HDFS-11647-001.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11647: - Attachment: (was: HADOOP-11647.patch) > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11647-001.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11646) Add -E option in 'ls' to list erasure coding policy of each file and directory if applicable
[ https://issues.apache.org/jira/browse/HDFS-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11646: - Attachment: HDFS-11646-002.patch > Add -E option in 'ls' to list erasure coding policy of each file and > directory if applicable > > > Key: HDFS-11646 > URL: https://issues.apache.org/jira/browse/HDFS-11646 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HADOOP-11646.patch, HDFS-11646-001.patch, > HDFS-11646-002.patch > > > Add -E option in "ls" to show erasure coding policy of file and directory, > leverage the "number_of_replicas " column. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11647: - Attachment: HDFS-11647-002.patch > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11647-001.patch, HDFS-11647-002.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11647: - Attachment: HDFS-11647-003.patch > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11647-001.patch, HDFS-11647-002.patch, > HDFS-11647-003.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11647: - Attachment: HDFS-11647-007.patch > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11647-001.patch, HDFS-11647-002.patch, > HDFS-11647-003.patch, HDFS-11647-004.patch, HDFS-11647-005.patch, > HDFS-11647-006.patch, HDFS-11647-007.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055064#comment-16055064 ] luhuichun commented on HDFS-11647: -- @Lei (Eddy) Xu thx Eddy, updated > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11647-001.patch, HDFS-11647-002.patch, > HDFS-11647-003.patch, HDFS-11647-004.patch, HDFS-11647-005.patch, > HDFS-11647-006.patch, HDFS-11647-007.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11975) Provide a system-default EC policy
[
https://issues.apache.org/jira/browse/HDFS-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
luhuichun updated HDFS-11975:
-
Status: Patch Available (was: Open)
> Provide a system-default EC policy
> --
>
> Key: HDFS-11975
> URL: https://issues.apache.org/jira/browse/HDFS-11975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: luhuichun
> Attachments: HDFS-11975-001.patch
>
>
> From the usability point of view, it'd be nice to be able to specify a
> system-wide EC policy, i.e., in {{hdfs-site.xml}}. For most of users / admin
> / downstream projects, it is not necessary to know the tradeoffs of the EC
> policy, considering that it requires the knowledge of EC, the actual physical
> topology of the clusters, and many other factors (i.e., network, cluster size
> and etc).
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11975) Provide a system-default EC policy
[
https://issues.apache.org/jira/browse/HDFS-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
luhuichun updated HDFS-11975:
-
Attachment: HDFS-11975-001.patch
> Provide a system-default EC policy
> --
>
> Key: HDFS-11975
> URL: https://issues.apache.org/jira/browse/HDFS-11975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: luhuichun
> Attachments: HDFS-11975-001.patch
>
>
> From the usability point of view, it'd be nice to be able to specify a
> system-wide EC policy, i.e., in {{hdfs-site.xml}}. For most of users / admin
> / downstream projects, it is not necessary to know the tradeoffs of the EC
> policy, considering that it requires the knowledge of EC, the actual physical
> topology of the clusters, and many other factors (i.e., network, cluster size
> and etc).
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11646) Add -E option in 'ls' to list erasure coding policy of each file and directory if applicable
[ https://issues.apache.org/jira/browse/HDFS-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11646: - Attachment: HDFS-11646-005.patch update patch > Add -E option in 'ls' to list erasure coding policy of each file and > directory if applicable > > > Key: HDFS-11646 > URL: https://issues.apache.org/jira/browse/HDFS-11646 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11646-001.patch, HDFS-11646-002.patch, > HDFS-11646-003.patch, HDFS-11646-004.patch, HDFS-11646-005.patch > > > Add -E option in "ls" to show erasure coding policy of file and directory, > leverage the "number_of_replicas " column. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12046) Hadoop CRC implementation using Intel ISAL-CRC library
luhuichun created HDFS-12046: Summary: Hadoop CRC implementation using Intel ISAL-CRC library Key: HDFS-12046 URL: https://issues.apache.org/jira/browse/HDFS-12046 Project: Hadoop HDFS Issue Type: Improvement Reporter: luhuichun Assignee: luhuichun Intel ISA-L open source library provides set of highly optimized functions for RAID, erasure code, CRC, cryptographic hash, encryption, and compression. Ref. https://github.com/01org/isa-l. HDFS-EC has already integrated ISA-L and added the necessary building options support for Hadoop. For Hadoop CRC, we recently explored more, developing a Hadoop CRC using Intel ISA-L, performing a test on Broadwell and Skylake servers, comparing the performance against Hadoop native CRC. On Broadwell/Skylake, ISA-L CRC has about 8%~ performance gain over Hadoop native CRC. We suggest adding a new Hadoop native CRC using the ISA-L library, the extra advantage is it’s already optimized when we upgrade to new servers and Hadoop developers don’t have to maintain their own bunch of ASM codes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11646) Add -E option in 'ls' to list erasure coding policy of each file and directory if applicable
[ https://issues.apache.org/jira/browse/HDFS-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11646: - Attachment: HADOOP-11646.patch > Add -E option in 'ls' to list erasure coding policy of each file and > directory if applicable > > > Key: HDFS-11646 > URL: https://issues.apache.org/jira/browse/HDFS-11646 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HADOOP-11646.patch > > > Add -E option in "ls" to show erasure coding policy of file and directory, > leverage the "number_of_replicas " column. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11647: - Attachment: HADOOP-11647.patch > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HADOOP-11647.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11646) Add -E option in 'ls' to list erasure coding policy of each file and directory if applicable
[ https://issues.apache.org/jira/browse/HDFS-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11646: - Status: Patch Available (was: Open) > Add -E option in 'ls' to list erasure coding policy of each file and > directory if applicable > > > Key: HDFS-11646 > URL: https://issues.apache.org/jira/browse/HDFS-11646 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HADOOP-11646.patch > > > Add -E option in "ls" to show erasure coding policy of file and directory, > leverage the "number_of_replicas " column. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11646) Add -E option in 'ls' to list erasure coding policy of each file and directory if applicable
[ https://issues.apache.org/jira/browse/HDFS-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11646: - Attachment: HADOOP-11646.patch > Add -E option in 'ls' to list erasure coding policy of each file and > directory if applicable > > > Key: HDFS-11646 > URL: https://issues.apache.org/jira/browse/HDFS-11646 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HADOOP-11646.patch, HDFS-11646-001.patch, > HDFS-11646-002.patch > > > Add -E option in "ls" to show erasure coding policy of file and directory, > leverage the "number_of_replicas " column. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11647: - Attachment: HDFS-11647-004.patch > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11647-001.patch, HDFS-11647-002.patch, > HDFS-11647-003.patch, HDFS-11647-004.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11646) Add -E option in 'ls' to list erasure coding policy of each file and directory if applicable
[ https://issues.apache.org/jira/browse/HDFS-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11646: - Attachment: HDFS-11646-003.patch > Add -E option in 'ls' to list erasure coding policy of each file and > directory if applicable > > > Key: HDFS-11646 > URL: https://issues.apache.org/jira/browse/HDFS-11646 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11646-001.patch, HDFS-11646-002.patch, > HDFS-11646-003.patch > > > Add -E option in "ls" to show erasure coding policy of file and directory, > leverage the "number_of_replicas " column. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11646) Add -E option in 'ls' to list erasure coding policy of each file and directory if applicable
[ https://issues.apache.org/jira/browse/HDFS-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11646: - Attachment: (was: HADOOP-11646.patch) > Add -E option in 'ls' to list erasure coding policy of each file and > directory if applicable > > > Key: HDFS-11646 > URL: https://issues.apache.org/jira/browse/HDFS-11646 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11646-001.patch, HDFS-11646-002.patch, > HDFS-11646-003.patch > > > Add -E option in "ls" to show erasure coding policy of file and directory, > leverage the "number_of_replicas " column. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11646) Add -E option in 'ls' to list erasure coding policy of each file and directory if applicable
[ https://issues.apache.org/jira/browse/HDFS-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050010#comment-16050010 ] luhuichun commented on HDFS-11646: -- [~eddyxu][~andrew.wang] thx for comments, updated the patch > Add -E option in 'ls' to list erasure coding policy of each file and > directory if applicable > > > Key: HDFS-11646 > URL: https://issues.apache.org/jira/browse/HDFS-11646 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11646-001.patch, HDFS-11646-002.patch, > HDFS-11646-003.patch, HDFS-11646-004.patch > > > Add -E option in "ls" to show erasure coding policy of file and directory, > leverage the "number_of_replicas " column. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050012#comment-16050012 ] luhuichun commented on HDFS-11647: -- [~eddyxu] Hi Eddy, updated according to your comments, thx for review > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11647-001.patch, HDFS-11647-002.patch, > HDFS-11647-003.patch, HDFS-11647-004.patch, HDFS-11647-005.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11647: - Attachment: HDFS-11647-005.patch > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11647-001.patch, HDFS-11647-002.patch, > HDFS-11647-003.patch, HDFS-11647-004.patch, HDFS-11647-005.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11646) Add -E option in 'ls' to list erasure coding policy of each file and directory if applicable
[ https://issues.apache.org/jira/browse/HDFS-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11646: - Attachment: HDFS-11646-004.patch > Add -E option in 'ls' to list erasure coding policy of each file and > directory if applicable > > > Key: HDFS-11646 > URL: https://issues.apache.org/jira/browse/HDFS-11646 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11646-001.patch, HDFS-11646-002.patch, > HDFS-11646-003.patch, HDFS-11646-004.patch > > > Add -E option in "ls" to show erasure coding policy of file and directory, > leverage the "number_of_replicas " column. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11646) Add -E option in 'ls' to list erasure coding policy of each file and directory if applicable
[ https://issues.apache.org/jira/browse/HDFS-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051276#comment-16051276 ] luhuichun commented on HDFS-11646: -- [~eddyxu] ok, will update today > Add -E option in 'ls' to list erasure coding policy of each file and > directory if applicable > > > Key: HDFS-11646 > URL: https://issues.apache.org/jira/browse/HDFS-11646 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11646-001.patch, HDFS-11646-002.patch, > HDFS-11646-003.patch, HDFS-11646-004.patch > > > Add -E option in "ls" to show erasure coding policy of file and directory, > leverage the "number_of_replicas " column. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11647: - Attachment: HDFS-11647-006.patch updated with new version > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11647-001.patch, HDFS-11647-002.patch, > HDFS-11647-003.patch, HDFS-11647-004.patch, HDFS-11647-005.patch, > HDFS-11647-006.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053507#comment-16053507 ] luhuichun commented on HDFS-11647: -- [~eddyxu] Hi Eddy maybe we can use getErasureCodingPolicyName(). > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11647-001.patch, HDFS-11647-002.patch, > HDFS-11647-003.patch, HDFS-11647-004.patch, HDFS-11647-005.patch, > HDFS-11647-006.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053530#comment-16053530 ] luhuichun commented on HDFS-11647: -- [~eddyxu] in ContentSummaryComputationContext.java， we follow the other functions' name practice in the same file,So I think getErasureCodingPolicyName() is better and clear. ecPolicy() is named in other file by patches committed before, so I think we can open a refactor JIRA to fix this issue. > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11647-001.patch, HDFS-11647-002.patch, > HDFS-11647-003.patch, HDFS-11647-004.patch, HDFS-11647-005.patch, > HDFS-11647-006.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luhuichun updated HDFS-11647: - Comment: was deleted (was: [~eddyxu] Hi Eddy maybe we can use getErasureCodingPolicyName().) > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11647-001.patch, HDFS-11647-002.patch, > HDFS-11647-003.patch, HDFS-11647-004.patch, HDFS-11647-005.patch, > HDFS-11647-006.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053530#comment-16053530 ] luhuichun edited comment on HDFS-11647 at 6/19/17 6:57 AM: --- [~eddyxu] in ContentSummaryComputationContext.java， we follow the other functions' name practice in the same file,So I think getErasureCodingPolicyName() is better and clear. I did not find ecPolicy() in code base. was (Author: luhuichun): [~eddyxu] in ContentSummaryComputationContext.java， we follow the other functions' name practice in the same file,So I think getErasureCodingPolicyName() is better and clear. ecPolicy() is named in other file by patches committed before, so I think we can open a refactor JIRA to fix this issue. > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11647-001.patch, HDFS-11647-002.patch, > HDFS-11647-003.patch, HDFS-11647-004.patch, HDFS-11647-005.patch, > HDFS-11647-006.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
