[jira] [Commented] (HDFS-6709) Implement off-heap data structures for NameNode and other HDFS memory optimization

2014-07-22 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14070021#comment-14070021 ] Kai Zheng commented on HDFS-6709: - Hi Colin, I learned the patch. The Slab stuffs look

[jira] [Commented] (HDFS-6658) Namenode memory optimization - Block replicas list

2014-07-22 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14070028#comment-14070028 ] Kai Zheng commented on HDFS-6658: - So would it be better to put the BlockList stuffs off

[jira] [Commented] (HDFS-6709) Implement off-heap data structures for NameNode and other HDFS memory optimization

2014-07-22 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071222#comment-14071222 ] Kai Zheng commented on HDFS-6709: - bq. The {{ByteBuffer}} interface should be slower, since

[jira] [Commented] (HDFS-6709) Implement off-heap data structures for NameNode and other HDFS memory optimization

2014-07-24 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14074027#comment-14074027 ] Kai Zheng commented on HDFS-6709: - bq. Sadly, while investigating off heap performance

[jira] [Commented] (HDFS-6709) Implement off-heap data structures for NameNode and other HDFS memory optimization

2014-07-25 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14074189#comment-14074189 ] Kai Zheng commented on HDFS-6709: - I repeated the test in the post and sadly found it's

[jira] [Created] (HDFS-5087) Allowing specific JAVA heap max setting for HDFS related services

2013-08-11 Thread Kai Zheng (JIRA)
Kai Zheng created HDFS-5087: --- Summary: Allowing specific JAVA heap max setting for HDFS related services Key: HDFS-5087 URL: https://issues.apache.org/jira/browse/HDFS-5087 Project: Hadoop HDFS

[jira] [Updated] (HDFS-5087) Allowing specific JAVA heap max setting for HDFS related services

2013-08-11 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-5087: Attachment: HDFS-5087.patch Modified hdfs script to check and respect any specific JAVA heap max setting

[jira] [Updated] (HDFS-5087) Allowing specific JAVA heap max setting for HDFS related services

2013-08-11 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-5087: Status: Patch Available (was: Open) Allowing specific JAVA heap max setting for HDFS related services

[jira] [Commented] (HDFS-5087) Allowing specific JAVA heap max setting for HDFS related services

2013-08-13 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738942#comment-13738942 ] Kai Zheng commented on HDFS-5087: - It only changed a script and was tested manually.

[jira] [Commented] (HDFS-5087) Allowing specific JAVA heap max setting for HDFS related services

2013-08-14 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740103#comment-13740103 ] Kai Zheng commented on HDFS-5087: - In hdfs script, with the following line {code} exec

[jira] [Commented] (HDFS-5087) Allowing specific JAVA heap max setting for HDFS related services

2013-08-14 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740414#comment-13740414 ] Kai Zheng commented on HDFS-5087: - bq.just process the command line parameters and strip

[jira] [Created] (HDFS-5129) JournalNode should also respect DFS_WEB_AUTHENTICATION_KERBEROS_KEYTAB_KEY as NN and SNN

2013-08-24 Thread Kai Zheng (JIRA)
Kai Zheng created HDFS-5129: --- Summary: JournalNode should also respect DFS_WEB_AUTHENTICATION_KERBEROS_KEYTAB_KEY as NN and SNN Key: HDFS-5129 URL: https://issues.apache.org/jira/browse/HDFS-5129 Project:

[jira] [Resolved] (HDFS-5129) JournalNode should also respect DFS_WEB_AUTHENTICATION_KERBEROS_KEYTAB_KEY as NN and SNN

2013-08-24 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng resolved HDFS-5129. - Resolution: Duplicate Duplicate of HDFS-5091. Support for spnego keytab separate from the JournalNode

[jira] [Created] (HDFS-5152) Avoiding redundant Kerberos login for Zookeeper client in ActiveStandbyElector

2013-08-30 Thread Kai Zheng (JIRA)
Kai Zheng created HDFS-5152: --- Summary: Avoiding redundant Kerberos login for Zookeeper client in ActiveStandbyElector Key: HDFS-5152 URL: https://issues.apache.org/jira/browse/HDFS-5152 Project: Hadoop

[jira] [Updated] (HDFS-5152) Avoiding redundant Kerberos login for Zookeeper client in ActiveStandbyElector

2013-09-02 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-5152: Attachment: HDFS-5152.patch Attached a patch. Pending to be submitted when required patches get in.

[jira] [Commented] (HDFS-5152) Avoiding redundant Kerberos login for Zookeeper client in ActiveStandbyElector

2013-09-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13757068#comment-13757068 ] Kai Zheng commented on HDFS-5152: - bq.Is that correct? Yes, exactly. bq.Will this change

[jira] [Commented] (HDFS-5152) Avoiding redundant Kerberos login for Zookeeper client in ActiveStandbyElector

2013-09-04 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13758693#comment-13758693 ] Kai Zheng commented on HDFS-5152: - In non-secure mode login is null and the behavior stays

[jira] [Commented] (HDFS-5248) Checkpointing fails in uploading the Image to the Namenode if the merge requires more than 10 hours in secure cluster

2013-09-23 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13774813#comment-13774813 ] Kai Zheng commented on HDFS-5248: - I guess there might be a place calling ugi.doAs or like

[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS

2014-10-29 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14188623#comment-14188623 ] Kai Zheng commented on HDFS-7285: - Thanks Zhe for hosting the session. bq.After soliciting

[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS

2014-11-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14195736#comment-14195736 ] Kai Zheng commented on HDFS-7285: - Thanks everyone for having the branch. We're currently

[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS

2014-11-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14195748#comment-14195748 ] Kai Zheng commented on HDFS-7285: - Hi [~apurtell], I think you have good points. As the

[jira] [Created] (HDFS-7343) A comprehensive storage policy engine

2014-11-03 Thread Kai Zheng (JIRA)
Kai Zheng created HDFS-7343: --- Summary: A comprehensive storage policy engine Key: HDFS-7343 URL: https://issues.apache.org/jira/browse/HDFS-7343 Project: Hadoop HDFS Issue Type: Improvement

[jira] [Updated] (HDFS-7343) A comprehensive and flexible storage policy engine

2014-11-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7343: Summary: A comprehensive and flexible storage policy engine (was: A comprehensive storage policy engine)

[jira] [Commented] (HDFS-7343) A comprehensive and flexible storage policy engine

2014-11-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14195760#comment-14195760 ] Kai Zheng commented on HDFS-7343: - Abstracted from [~Andrew.wang]'s [meeting

[jira] [Commented] (HDFS-7343) A comprehensive and flexible storage policy engine

2014-11-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14195766#comment-14195766 ] Kai Zheng commented on HDFS-7343: - For [~apurtell]'s

[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS

2014-11-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14195770#comment-14195770 ] Kai Zheng commented on HDFS-7285: - Hi [~andrew.wang], to address the policy engine aspects

[jira] [Created] (HDFS-7344) Erasure Coding worker and support in DataNode

2014-11-03 Thread Kai Zheng (JIRA)
Kai Zheng created HDFS-7344: --- Summary: Erasure Coding worker and support in DataNode Key: HDFS-7344 URL: https://issues.apache.org/jira/browse/HDFS-7344 Project: Hadoop HDFS Issue Type: Sub-task

[jira] [Created] (HDFS-7345) Local Reconstruction Codes (LRC)

2014-11-03 Thread Kai Zheng (JIRA)
Kai Zheng created HDFS-7345: --- Summary: Local Reconstruction Codes (LRC) Key: HDFS-7345 URL: https://issues.apache.org/jira/browse/HDFS-7345 Project: Hadoop HDFS Issue Type: Sub-task Affects

[jira] [Updated] (HDFS-7344) Erasure Coding worker and support in DataNode

2014-11-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7344: Assignee: Li Bo (was: Kai Zheng) Erasure Coding worker and support in DataNode

[jira] [Assigned] (HDFS-7338) Reed-Solomon codec library support

2014-11-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng reassigned HDFS-7338: --- Assignee: Li Bo Reed-Solomon codec library support --

[jira] [Updated] (HDFS-7337) Configurable and pluggable Erasure Codec and schema

2014-11-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7337: Description: According to HDFS-7285 and the design, this considers to support multiple Erasure Codecs

[jira] [Commented] (HDFS-7337) Configurable and pluggable Erasure Codec and schema

2014-11-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14195815#comment-14195815 ] Kai Zheng commented on HDFS-7337: - To better position this issue and avoid the possible

[jira] [Created] (HDFS-7346) Process erasure encoding work

2014-11-03 Thread Kai Zheng (JIRA)
Kai Zheng created HDFS-7346: --- Summary: Process erasure encoding work Key: HDFS-7346 URL: https://issues.apache.org/jira/browse/HDFS-7346 Project: Hadoop HDFS Issue Type: Task Components:

[jira] [Commented] (HDFS-7337) Configurable and pluggable Erasure Codec and schema

2014-11-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14195821#comment-14195821 ] Kai Zheng commented on HDFS-7337: - Yes, agree. Please go ahead. Thanks. Configurable and

[jira] [Created] (HDFS-7348) Process erasure decoding work

2014-11-03 Thread Kai Zheng (JIRA)
Kai Zheng created HDFS-7348: --- Summary: Process erasure decoding work Key: HDFS-7348 URL: https://issues.apache.org/jira/browse/HDFS-7348 Project: Hadoop HDFS Issue Type: Task Reporter:

[jira] [Assigned] (HDFS-7338) Reed-Solomon codec library support

2014-11-04 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng reassigned HDFS-7338: --- Assignee: Kai Zheng (was: Li Bo) Reed-Solomon codec library support

[jira] [Updated] (HDFS-7285) Erasure Coding Support inside HDFS

2014-11-04 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7285: Fix Version/s: HDFS-EC Erasure Coding Support inside HDFS --

[jira] [Created] (HDFS-7353) Common Erasure Codec API and plugin support

2014-11-04 Thread Kai Zheng (JIRA)
Kai Zheng created HDFS-7353: --- Summary: Common Erasure Codec API and plugin support Key: HDFS-7353 URL: https://issues.apache.org/jira/browse/HDFS-7353 Project: Hadoop HDFS Issue Type: Sub-task

[jira] [Assigned] (HDFS-7353) Common Erasure Codec API and plugin support

2014-11-04 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng reassigned HDFS-7353: --- Assignee: Kai Zheng Common Erasure Codec API and plugin support

[jira] [Commented] (HDFS-7338) Reed-Solomon codec library support

2014-11-04 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14196142#comment-14196142 ] Kai Zheng commented on HDFS-7338: - This will follow the common codec API and plugin support

[jira] [Updated] (HDFS-7338) Reed-Solomon codec library support

2014-11-04 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7338: Description: This is to provide RS codec implementation for encoding and decoding. Reed-Solomon codec

[jira] [Updated] (HDFS-7337) Configurable and pluggable Erasure Codec and schema

2014-11-04 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7337: Description: According to HDFS-7285 and the design, this considers to support multiple Erasure Codecs via

[jira] [Commented] (HDFS-7337) Configurable and pluggable Erasure Codec and schema

2014-11-04 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14196154#comment-14196154 ] Kai Zheng commented on HDFS-7337: - Also opened HDFS-7353 to focus on the very low level

[jira] [Commented] (HDFS-7337) Configurable and pluggable Erasure Codec and schema

2014-11-04 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14196163#comment-14196163 ] Kai Zheng commented on HDFS-7337: - Zhe, let me consider these issues together and think

[jira] [Assigned] (HDFS-7337) Configurable and pluggable Erasure Codec and schema

2014-11-04 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng reassigned HDFS-7337: --- Assignee: Kai Zheng Configurable and pluggable Erasure Codec and schema

[jira] [Updated] (HDFS-7353) Common Erasure Codec API and plugin support

2014-11-04 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7353: Fix Version/s: HDFS-EC Common Erasure Codec API and plugin support

[jira] [Updated] (HDFS-7337) Configurable and pluggable Erasure Codec and schema

2014-11-06 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7337: Attachment: HDFS-7337-prototype-v1.patch Initial prototype patch for quick ideas. Configurable and

[jira] [Commented] (HDFS-7337) Configurable and pluggable Erasure Codec and schema

2014-11-06 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14200034#comment-14200034 ] Kai Zheng commented on HDFS-7337: - [~zhz], would you take a look at the quick prototype

[jira] [Created] (HDFS-7371) Loading EC schemas from configuration

2014-11-06 Thread Kai Zheng (JIRA)
Kai Zheng created HDFS-7371: --- Summary: Loading EC schemas from configuration Key: HDFS-7371 URL: https://issues.apache.org/jira/browse/HDFS-7371 Project: Hadoop HDFS Issue Type: Task

[jira] [Commented] (HDFS-7371) Loading EC schemas from configuration

2014-11-06 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14200049#comment-14200049 ] Kai Zheng commented on HDFS-7371: - Configuring multiple erasure codecs (*JerasureRS*,

[jira] [Commented] (HDFS-7371) Loading EC schemas from configuration

2014-11-06 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14200051#comment-14200051 ] Kai Zheng commented on HDFS-7371: - In the above defined schemas, the options/parameters

[jira] [Commented] (HDFS-7337) Configurable and pluggable Erasure Codec and schema

2014-11-06 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14200140#comment-14200140 ] Kai Zheng commented on HDFS-7337: - The points made in the prototype patch are: * Multiple

[jira] [Assigned] (HDFS-7343) A comprehensive and flexible storage policy engine

2014-11-06 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng reassigned HDFS-7343: --- Assignee: Kai Zheng A comprehensive and flexible storage policy engine

[jira] [Commented] (HDFS-7343) A comprehensive and flexible storage policy engine

2014-11-06 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201258#comment-14201258 ] Kai Zheng commented on HDFS-7343: - Yes Andrew. I linked the mentioned issue and will take

[jira] [Commented] (HDFS-7343) A comprehensive and flexible storage policy engine

2014-11-06 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201284#comment-14201284 ] Kai Zheng commented on HDFS-7343: - As far as I understand for now, the policy engine

[jira] [Updated] (HDFS-7343) A comprehensive and flexible storage policy engine

2014-11-06 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7343: Target Version/s: (was: HDFS-EC) A comprehensive and flexible storage policy engine

[jira] [Commented] (HDFS-7363) Pluggable algorithms to form block groups in erasure coding

2014-11-09 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14204273#comment-14204273 ] Kai Zheng commented on HDFS-7363: - The logic was intended to be covered in erasure codec by

[jira] [Resolved] (HDFS-7363) Pluggable algorithms to form block groups in erasure coding

2014-11-09 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng resolved HDFS-7363. - Resolution: Duplicate Pluggable algorithms to form block groups in erasure coding

[jira] [Commented] (HDFS-7345) Local Reconstruction Codes (LRC)

2014-11-10 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205962#comment-14205962 ] Kai Zheng commented on HDFS-7345: - From [Facebook’s advanced erasure

[jira] [Commented] (HDFS-7345) Local Reconstruction Codes (LRC)

2014-11-11 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14207572#comment-14207572 ] Kai Zheng commented on HDFS-7345: - Thanks [~andrew.wang] for the escalating. Local

[jira] [Commented] (HDFS-7345) Local Reconstruction Codes (LRC)

2014-11-11 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14207583#comment-14207583 ] Kai Zheng commented on HDFS-7345: - Anyway how it goes for the IP issue, I would continue

[jira] [Updated] (HDFS-7337) Configurable and pluggable Erasure Codec and schema

2014-11-12 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7337: Attachment: PluggableErasureCodec.pdf For time saving, I composed this doc to illustrate the erasure codec

[jira] [Reopened] (HDFS-7363) Pluggable algorithms to form block groups in erasure coding

2014-11-12 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng reopened HDFS-7363: - Assignee: Kai Zheng Let me reuse this item as a sub task of HDFS-7337, where BlockGrouper is defined

[jira] [Updated] (HDFS-7353) Common Erasure Coder API

2014-11-12 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7353: Description: This is to abstract and define common coder API across different codec algorithms like RS, XOR

[jira] [Updated] (HDFS-7353) Common Erasure Coder API

2014-11-12 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7353: Summary: Common Erasure Coder API (was: Common Erasure Codec API and plugin support) Common Erasure Coder

[jira] [Updated] (HDFS-7345) Local Reconstruction Codes (LRC)

2014-11-12 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7345: Description: HDFS-7285 proposes to support Erasure Coding inside HDFS, supports multiple Erasure Coding

[jira] [Updated] (HDFS-7345) Local Repairable Codes (LRC)

2014-11-13 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7345: Summary: Local Repairable Codes (LRC) (was: Local Reconstruction Codes (LRC)) Local Repairable Codes

[jira] [Updated] (HDFS-7345) Local Repairable Codes (LRC)

2014-11-13 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7345: Description: HDFS-7285 proposes to support Erasure Coding inside HDFS, supports multiple Erasure Coding

[jira] [Commented] (HDFS-7345) Local Repairable Codes (LRC)

2014-11-13 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14209516#comment-14209516 ] Kai Zheng commented on HDFS-7345: - To possibly avoid the IP concern, I changed the title

[jira] [Updated] (HDFS-7345) Local Repairable Codes (LRC)

2014-11-13 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7345: Attachment: LRC-compare.pdf Added a simple slides to illustrate the differences among all the so-called LRC

[jira] [Updated] (HDFS-7371) Loading EC schemas from configuration

2014-11-20 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7371: Issue Type: Sub-task (was: Task) Parent: HDFS-7285 Loading EC schemas from configuration

[jira] [Updated] (HDFS-7346) Process erasure encoding work

2014-11-20 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7346: Issue Type: Sub-task (was: Task) Parent: HDFS-7285 Process erasure encoding work

[jira] [Updated] (HDFS-7348) Process erasure decoding work

2014-11-20 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7348: Issue Type: Sub-task (was: Task) Parent: HDFS-7285 Process erasure decoding work

[jira] [Updated] (HDFS-7338) Reed-Solomon codes

2014-11-20 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7338: Summary: Reed-Solomon codes (was: Reed-Solomon codec library support) Reed-Solomon codes

[jira] [Created] (HDFS-7417) XOR codes

2014-11-20 Thread Kai Zheng (JIRA)
Kai Zheng created HDFS-7417: --- Summary: XOR codes Key: HDFS-7417 URL: https://issues.apache.org/jira/browse/HDFS-7417 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng

[jira] [Created] (HDFS-7418) RS coder in pure Java

2014-11-20 Thread Kai Zheng (JIRA)
Kai Zheng created HDFS-7418: --- Summary: RS coder in pure Java Key: HDFS-7418 URL: https://issues.apache.org/jira/browse/HDFS-7418 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai

[jira] [Updated] (HDFS-7353) Erasure Coder for concrete encoding and decoding

2014-11-20 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7353: Summary: Erasure Coder for concrete encoding and decoding (was: Common Erasure Coder API) Erasure Coder

[jira] [Updated] (HDFS-7353) Erasure Coder for concrete encoding and decoding

2014-11-20 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7353: Description: This is to abstract and define erasure coder API across different codes algorithms like RS, XOR

[jira] [Updated] (HDFS-7418) RS coder in pure Java

2014-11-20 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7418: Description: This will implement RS coder by porting existing codes in HDFS-RAID in the new codec and coder

[jira] [Commented] (HDFS-7349) Support DFS command for the EC encoding

2014-12-01 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231023#comment-14231023 ] Kai Zheng commented on HDFS-7349: - Hi Vinay, The patch looks great even in the initial

[jira] [Commented] (HDFS-7349) Support DFS command for the EC encoding

2014-12-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14232819#comment-14232819 ] Kai Zheng commented on HDFS-7349: - Thanks Vinay for the detailed clarifying. About

[jira] [Updated] (HDFS-7337) Configurable and pluggable Erasure Codec and schema

2014-12-04 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7337: Attachment: HDFS-7337-prototype-v3.zip Updated the prototype codes: 1. Refined the APIs, ErasureCoder

[jira] [Commented] (HDFS-6133) Make Balancer support exclude specified path

2014-12-20 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14255032#comment-14255032 ] Kai Zheng commented on HDFS-6133: - Hi Yunjiong, The patch looks great. Minor comments: 1.

[jira] [Commented] (HDFS-7337) Configurable and pluggable Erasure Codec and schema

2014-12-30 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261964#comment-14261964 ] Kai Zheng commented on HDFS-7337: - We're discussing offline thru the design and codes. When

[jira] [Updated] (HDFS-7371) Loading predefined EC schemas from configuration

2015-02-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7371: Description: System administrator can configure multiple EC codecs in hdfs-site.xml file, and codec

[jira] [Updated] (HDFS-7371) Loading predefined EC schemas from configuration

2015-02-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7371: Description: System administrator can configure multiple EC codecs in hdfs-site.xml file, and codec

[jira] [Commented] (HDFS-7349) Support DFS command for the EC encoding

2015-02-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304733#comment-14304733 ] Kai Zheng commented on HDFS-7349: - bq.It's a design question whether we should ask users to

[jira] [Updated] (HDFS-7371) Loading predefined EC schemas from configuration

2015-02-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7371: Summary: Loading predefined EC schemas from configuration (was: Loading EC schemas from configuration)

[jira] [Commented] (HDFS-7731) Can not start HA namenode with security enabled

2015-02-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304746#comment-14304746 ] Kai Zheng commented on HDFS-7731: - bq.The key version is same as the version in the keytab

[jira] [Commented] (HDFS-7371) Loading predefined EC schemas from configuration

2015-02-03 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304730#comment-14304730 ] Kai Zheng commented on HDFS-7371: - [~zhz] has good idea to store schema parameters as

[jira] [Commented] (HDFS-7349) Support DFS command for the EC encoding

2015-02-04 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306525#comment-14306525 ] Kai Zheng commented on HDFS-7349: - OK no problem. I'm working on the way from low level

[jira] [Commented] (HDFS-7371) Loading predefined EC schemas from configuration

2015-02-04 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306507#comment-14306507 ] Kai Zheng commented on HDFS-7371: - Thanks Zhe. Will look at the design and think about it.

[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS

2015-02-02 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302086#comment-14302086 ] Kai Zheng commented on HDFS-7285: - bq.I think we need to allow dynamic stripe cell size

[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS

2015-02-02 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302602#comment-14302602 ] Kai Zheng commented on HDFS-7285: - I'm not sure storage policy can cover all the cases and

[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS

2015-02-02 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302583#comment-14302583 ] Kai Zheng commented on HDFS-7285: - Yes we have EC zones, each zone actually represents a

[jira] [Commented] (HDFS-7349) Support DFS command for the EC encoding

2015-02-02 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302584#comment-14302584 ] Kai Zheng commented on HDFS-7349: - Sorry I missed this. Will look at it today. Support

[jira] [Updated] (HDFS-7417) Raw XOR coder

2015-02-02 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7417: Fix Version/s: HDFS-EC Raw XOR coder - Key: HDFS-7417 URL:

[jira] [Commented] (HDFS-7545) Data striping support in HDFS client

2015-02-02 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301229#comment-14301229 ] Kai Zheng commented on HDFS-7545: - Hi Bo, Can you open a sub-task JIRA for your first

[jira] [Updated] (HDFS-7417) Raw XOR coder

2015-02-02 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7417: Attachment: HDFS-7417-v2.patch Updated the patch merging latest changes on the branch. Raw XOR coder

[jira] [Updated] (HDFS-7417) Raw XOR coder

2015-02-02 Thread Kai Zheng (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7417: Status: Patch Available (was: Open) Raw XOR coder - Key: HDFS-7417

  1   2   3   4   5   6   7   8   9   10   >