[jira] [Commented] (HADOOP-11646) Erasure Coder API for encoding and decoding of block group
[ https://issues.apache.org/jira/browse/HADOOP-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938013#comment-14938013 ] Hudson commented on HADOOP-11646: - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HADOOP-11646. Erasure Coder API for encoding and decoding of block group (zhezhang: rev 8f89d7489df0d5c8236a1929c93e3f5ab7149031) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureEncodingStep.java * hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureEncoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureDecoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureDecodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureEncoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureCodingStep.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestErasureCoderBase.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/XorErasureDecoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureDecoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECBlock.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/XorErasureEncoder.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestXorCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECBlockGroup.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureCodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureCoder.java > Erasure Coder API for encoding and decoding of block group > -- > > Key: HADOOP-11646 > URL: https://issues.apache.org/jira/browse/HADOOP-11646 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: HDFS-7285 > > Attachments: HADOOP-11646-v4.patch, HADOOP-11646-v5.patch, > HDFS-7662-v1.patch, HDFS-7662-v2.patch, HDFS-7662-v3.patch > > > This is to define ErasureCoder API for encoding and decoding of BlockGroup. > Given a BlockGroup, ErasureCoder extracts data chunks from the blocks and > leverages RawErasureCoder defined in HADOOP-11514 to perform concrete > encoding or decoding. Note this mainly focuses on the basic fundamental > aspects, and solves encoding, data blocks recovering and etc. Regarding > parity blocks recovering, as it involves multiple steps, HADOOP-11550 will > handle it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11646) Erasure Coder API for encoding and decoding of block group
[ https://issues.apache.org/jira/browse/HADOOP-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937794#comment-14937794 ] Hudson commented on HADOOP-11646: - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HADOOP-11646. Erasure Coder API for encoding and decoding of block group (zhezhang: rev 8f89d7489df0d5c8236a1929c93e3f5ab7149031) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/XorErasureEncoder.java * hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureDecodingStep.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestXorCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureDecoder.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestErasureCoderBase.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureEncoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureCodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/XorErasureDecoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureEncoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureEncodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECBlock.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureCodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECBlockGroup.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureDecoder.java > Erasure Coder API for encoding and decoding of block group > -- > > Key: HADOOP-11646 > URL: https://issues.apache.org/jira/browse/HADOOP-11646 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: HDFS-7285 > > Attachments: HADOOP-11646-v4.patch, HADOOP-11646-v5.patch, > HDFS-7662-v1.patch, HDFS-7662-v2.patch, HDFS-7662-v3.patch > > > This is to define ErasureCoder API for encoding and decoding of BlockGroup. > Given a BlockGroup, ErasureCoder extracts data chunks from the blocks and > leverages RawErasureCoder defined in HADOOP-11514 to perform concrete > encoding or decoding. Note this mainly focuses on the basic fundamental > aspects, and solves encoding, data blocks recovering and etc. Regarding > parity blocks recovering, as it involves multiple steps, HADOOP-11550 will > handle it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11646) Erasure Coder API for encoding and decoding of block group
[ https://issues.apache.org/jira/browse/HADOOP-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938583#comment-14938583 ] Hudson commented on HADOOP-11646: - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #465 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/465/]) HADOOP-11646. Erasure Coder API for encoding and decoding of block group (zhezhang: rev 8f89d7489df0d5c8236a1929c93e3f5ab7149031) * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestErasureCoderBase.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/XorErasureEncoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureCodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureDecoder.java * hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureEncoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureDecoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECBlock.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureEncodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/XorErasureDecoder.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestXorCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECBlockGroup.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureCodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureEncoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureDecodingStep.java > Erasure Coder API for encoding and decoding of block group > -- > > Key: HADOOP-11646 > URL: https://issues.apache.org/jira/browse/HADOOP-11646 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: HDFS-7285 > > Attachments: HADOOP-11646-v4.patch, HADOOP-11646-v5.patch, > HDFS-7662-v1.patch, HDFS-7662-v2.patch, HDFS-7662-v3.patch > > > This is to define ErasureCoder API for encoding and decoding of BlockGroup. > Given a BlockGroup, ErasureCoder extracts data chunks from the blocks and > leverages RawErasureCoder defined in HADOOP-11514 to perform concrete > encoding or decoding. Note this mainly focuses on the basic fundamental > aspects, and solves encoding, data blocks recovering and etc. Regarding > parity blocks recovering, as it involves multiple steps, HADOOP-11550 will > handle it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11646) Erasure Coder API for encoding and decoding of block group
[ https://issues.apache.org/jira/browse/HADOOP-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938389#comment-14938389 ] Hudson commented on HADOOP-11646: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2408 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2408/]) HADOOP-11646. Erasure Coder API for encoding and decoding of block group (zhezhang: rev 8f89d7489df0d5c8236a1929c93e3f5ab7149031) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureDecoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureDecoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureEncodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureDecodingStep.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestXorCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/XorErasureEncoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureEncoder.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestErasureCoderBase.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/XorErasureDecoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureEncoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureCodingStep.java * hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECBlock.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureCodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECBlockGroup.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureCoder.java > Erasure Coder API for encoding and decoding of block group > -- > > Key: HADOOP-11646 > URL: https://issues.apache.org/jira/browse/HADOOP-11646 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: HDFS-7285 > > Attachments: HADOOP-11646-v4.patch, HADOOP-11646-v5.patch, > HDFS-7662-v1.patch, HDFS-7662-v2.patch, HDFS-7662-v3.patch > > > This is to define ErasureCoder API for encoding and decoding of BlockGroup. > Given a BlockGroup, ErasureCoder extracts data chunks from the blocks and > leverages RawErasureCoder defined in HADOOP-11514 to perform concrete > encoding or decoding. Note this mainly focuses on the basic fundamental > aspects, and solves encoding, data blocks recovering and etc. Regarding > parity blocks recovering, as it involves multiple steps, HADOOP-11550 will > handle it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11646) Erasure Coder API for encoding and decoding of block group
[ https://issues.apache.org/jira/browse/HADOOP-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937537#comment-14937537 ] Hudson commented on HADOOP-11646: - FAILURE: Integrated in Hadoop-Yarn-trunk #1203 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1203/]) HADOOP-11646. Erasure Coder API for encoding and decoding of block group (zhezhang: rev 8f89d7489df0d5c8236a1929c93e3f5ab7149031) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureDecoder.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestErasureCoderBase.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureCodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureDecoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureEncoder.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestXorCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureDecodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/XorErasureEncoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECBlock.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureEncoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/XorErasureDecoder.java * hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureEncodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECBlockGroup.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureCodingStep.java > Erasure Coder API for encoding and decoding of block group > -- > > Key: HADOOP-11646 > URL: https://issues.apache.org/jira/browse/HADOOP-11646 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: HDFS-7285 > > Attachments: HADOOP-11646-v4.patch, HADOOP-11646-v5.patch, > HDFS-7662-v1.patch, HDFS-7662-v2.patch, HDFS-7662-v3.patch > > > This is to define ErasureCoder API for encoding and decoding of BlockGroup. > Given a BlockGroup, ErasureCoder extracts data chunks from the blocks and > leverages RawErasureCoder defined in HADOOP-11514 to perform concrete > encoding or decoding. Note this mainly focuses on the basic fundamental > aspects, and solves encoding, data blocks recovering and etc. Regarding > parity blocks recovering, as it involves multiple steps, HADOOP-11550 will > handle it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11646) Erasure Coder API for encoding and decoding of block group
[ https://issues.apache.org/jira/browse/HADOOP-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937287#comment-14937287 ] Hudson commented on HADOOP-11646: - FAILURE: Integrated in Hadoop-trunk-Commit #8548 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8548/]) HADOOP-11646. Erasure Coder API for encoding and decoding of block group (zhezhang: rev 8f89d7489df0d5c8236a1929c93e3f5ab7149031) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/XorErasureEncoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureEncoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureEncodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECBlock.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureCodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureEncoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureDecoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureDecodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECBlockGroup.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestXorCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureDecoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureCoder.java * hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestErasureCoderBase.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/XorErasureDecoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureCodingStep.java > Erasure Coder API for encoding and decoding of block group > -- > > Key: HADOOP-11646 > URL: https://issues.apache.org/jira/browse/HADOOP-11646 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: HDFS-7285 > > Attachments: HADOOP-11646-v4.patch, HADOOP-11646-v5.patch, > HDFS-7662-v1.patch, HDFS-7662-v2.patch, HDFS-7662-v3.patch > > > This is to define ErasureCoder API for encoding and decoding of BlockGroup. > Given a BlockGroup, ErasureCoder extracts data chunks from the blocks and > leverages RawErasureCoder defined in HADOOP-11514 to perform concrete > encoding or decoding. Note this mainly focuses on the basic fundamental > aspects, and solves encoding, data blocks recovering and etc. Regarding > parity blocks recovering, as it involves multiple steps, HADOOP-11550 will > handle it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11646) Erasure Coder API for encoding and decoding of block group
[ https://issues.apache.org/jira/browse/HADOOP-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937092#comment-14937092 ] Hudson commented on HADOOP-11646: - FAILURE: Integrated in Hadoop-Hdfs-trunk #2379 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2379/]) HADOOP-11646. Erasure Coder API for encoding and decoding of block group (zhezhang: rev 8f89d7489df0d5c8236a1929c93e3f5ab7149031) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureCodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureDecoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureEncoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECBlockGroup.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureCodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/XorErasureDecoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/XorErasureEncoder.java * hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestErasureCoderBase.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECBlock.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureDecodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureEncoder.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestXorCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureEncodingStep.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureDecoder.java > Erasure Coder API for encoding and decoding of block group > -- > > Key: HADOOP-11646 > URL: https://issues.apache.org/jira/browse/HADOOP-11646 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: HDFS-7285 > > Attachments: HADOOP-11646-v4.patch, HADOOP-11646-v5.patch, > HDFS-7662-v1.patch, HDFS-7662-v2.patch, HDFS-7662-v3.patch > > > This is to define ErasureCoder API for encoding and decoding of BlockGroup. > Given a BlockGroup, ErasureCoder extracts data chunks from the blocks and > leverages RawErasureCoder defined in HADOOP-11514 to perform concrete > encoding or decoding. Note this mainly focuses on the basic fundamental > aspects, and solves encoding, data blocks recovering and etc. Regarding > parity blocks recovering, as it involves multiple steps, HADOOP-11550 will > handle it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11646) Erasure Coder API for encoding and decoding of block group
[ https://issues.apache.org/jira/browse/HADOOP-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352615#comment-14352615 ] Vinayakumar B commented on HADOOP-11646: +1, for the latest patch. Erasure Coder API for encoding and decoding of block group -- Key: HADOOP-11646 URL: https://issues.apache.org/jira/browse/HADOOP-11646 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-7285 Attachments: HADOOP-11646-v4.patch, HADOOP-11646-v5.patch, HDFS-7662-v1.patch, HDFS-7662-v2.patch, HDFS-7662-v3.patch This is to define ErasureCoder API for encoding and decoding of BlockGroup. Given a BlockGroup, ErasureCoder extracts data chunks from the blocks and leverages RawErasureCoder defined in HADOOP-11514 to perform concrete encoding or decoding. Note this mainly focuses on the basic fundamental aspects, and solves encoding, data blocks recovering and etc. Regarding parity blocks recovering, as it involves multiple steps, HADOOP-11550 will handle it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11646) Erasure Coder API for encoding and decoding of block group
[ https://issues.apache.org/jira/browse/HADOOP-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351143#comment-14351143 ] Kai Zheng commented on HADOOP-11646: Thanks [~vinayrpet] for the codes ! Erasure Coder API for encoding and decoding of block group -- Key: HADOOP-11646 URL: https://issues.apache.org/jira/browse/HADOOP-11646 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-7285 Attachments: HADOOP-11646-v4.patch, HDFS-7662-v1.patch, HDFS-7662-v2.patch, HDFS-7662-v3.patch This is to define ErasureCoder API for encoding and decoding of BlockGroup. Given a BlockGroup, ErasureCoder extracts data chunks from the blocks and leverages RawErasureCoder defined in HADOOP-11514 to perform concrete encoding or decoding. Note this mainly focuses on the basic fundamental aspects, and solves encoding, data blocks recovering and etc. Regarding parity blocks recovering, as it involves multiple steps, HADOOP-11550 will handle it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11646) Erasure Coder API for encoding and decoding of block group
[
https://issues.apache.org/jira/browse/HADOOP-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14350230#comment-14350230
]
Vinayakumar B commented on HADOOP-11646:
One nit for the latest patch.
For the below part, {code}+ECBlock[] erasedBlocks = new ECBlock[numErased];
+for (int i = 0; i inputBlocks.length i erasedBlocks.length; i++) {
+ if (inputBlocks[i].isErased()) {
+erasedBlocks[i] = inputBlocks[i];
+ }
+}{code}
I expect as below,
{code}
+ECBlock[] erasedBlocks = new ECBlock[numErased];
+int j=0;
+for (int i = 0; i inputBlocks.length j erasedBlocks.length; i++) {
+ if (inputBlocks[i].isErased()) {
+erasedBlocks[j++] = inputBlocks[i];
+ }
+}{code}
Erasure Coder API for encoding and decoding of block group
--
Key: HADOOP-11646
URL: https://issues.apache.org/jira/browse/HADOOP-11646
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Fix For: HDFS-7285
Attachments: HADOOP-11646-v4.patch, HDFS-7662-v1.patch,
HDFS-7662-v2.patch, HDFS-7662-v3.patch
This is to define ErasureCoder API for encoding and decoding of BlockGroup.
Given a BlockGroup, ErasureCoder extracts data chunks from the blocks and
leverages RawErasureCoder defined in HADOOP-11514 to perform concrete
encoding or decoding. Note this mainly focuses on the basic fundamental
aspects, and solves encoding, data blocks recovering and etc. Regarding
parity blocks recovering, as it involves multiple steps, HADOOP-11550 will
handle it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11646) Erasure Coder API for encoding and decoding of block group
[
https://issues.apache.org/jira/browse/HADOOP-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348510#comment-14348510
]
Vinayakumar B commented on HADOOP-11646:
Hi kai,
Here are some of the comments of first look.
1. {code}
+ /**
+ *
+ * @return true if it's missing, otherwise false
+ */
+ public boolean isErased() {
+return isErased;
+ }{code}
{{isErased}}, Do you mean is block is missing and needs to re-construct?
if Yes, I think better name would be {{isMissed}} itself.
2.
{code}+ @Override
+ public ErasureCodingStep decode(ECBlockGroup blockGroup) {
+return performDecoding(blockGroup);
+ }
+
+ /**
+ * Perform decoding against a block blockGroup.
+ * @param blockGroup
+ * @return decoding step for caller to do the real work
+ */
+ protected abstract ErasureCodingStep performDecoding(ECBlockGroup
blockGroup);{code}
Why there are two separate methods with same params. Why can't just directly
IMPLs can implement {{decode(..)}} instead of implementing
{{performDecoding(..)}} ?
Same Q in AbstractErasureEncoder also
3.
For copying from one array to another System.arrayCopy(..) would be better.
{code}
+int idx = 0;
+for (int i = 0; i getNumParityUnits(); i++) {
+ inputBlocks[idx ++] = blockGroup.getParityBlocks()[i];
+}
+for (int i = 0; i getNumDataUnits(); i++) {
+ inputBlocks[idx ++] = blockGroup.getDataBlocks()[i];
+}{code}
could be done as
{code}System.arraycopy(blockGroup.getParityBlocks(), 0, inputBlocks, 0,
getNumParityUnits());
System.arraycopy(blockGroup.getDataBlocks(), 0, inputBlocks,
getNumParityUnits(), getNumDataUnits());{code}
Correct?
4. I think {{protected ECBlock[] getOutputBlocks(ECBlockGroup blockGroup) {}}
could be re-written as below for simplicity.
{code} protected int getNumErasedBlocks(ECBlockGroup blockGroup) {
int num = getNumErasedBlocks(blockGroup.getParityBlocks());
num += getNumErasedBlocks(blockGroup.getDataBlocks());
return num;
}{code}
5. Need to track the index of the erasedBlocks separately. Otherwise will throw
AIOBException
{code}+ECBlock[] erasedBlocks = new ECBlock[numErased];
+for (int i = 0; i inputBlocks.length; i++) {
+ if (inputBlocks[i].isErased()) {
+erasedBlocks[i] = inputBlocks[i];
+ }
+}{code}
6. Rename the patch with HADOOP prefix
Erasure Coder API for encoding and decoding of block group
--
Key: HADOOP-11646
URL: https://issues.apache.org/jira/browse/HADOOP-11646
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Fix For: HDFS-7285
Attachments: HDFS-7662-v1.patch, HDFS-7662-v2.patch,
HDFS-7662-v3.patch
This is to define ErasureCoder API for encoding and decoding of BlockGroup.
Given a BlockGroup, ErasureCoder extracts data chunks from the blocks and
leverages RawErasureCoder defined in HDFS-7353 to perform concrete encoding
or decoding.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11646) Erasure Coder API for encoding and decoding of block group
[ https://issues.apache.org/jira/browse/HADOOP-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349856#comment-14349856 ] Kai Zheng commented on HADOOP-11646: Thanks [~vinayrpet] for the great and detailed review ! I will resolve the comments and update patch today. Erasure Coder API for encoding and decoding of block group -- Key: HADOOP-11646 URL: https://issues.apache.org/jira/browse/HADOOP-11646 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-7285 Attachments: HDFS-7662-v1.patch, HDFS-7662-v2.patch, HDFS-7662-v3.patch This is to define ErasureCoder API for encoding and decoding of BlockGroup. Given a BlockGroup, ErasureCoder extracts data chunks from the blocks and leverages RawErasureCoder defined in HDFS-7353 to perform concrete encoding or decoding. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11646) Erasure Coder API for encoding and decoding of block group
[
https://issues.apache.org/jira/browse/HADOOP-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14350041#comment-14350041
]
Kai Zheng commented on HADOOP-11646:
bq.{{isErased}}, Do you mean is block is missing and needs to re-construct?
Yes right. We have two cases, one is lost at all, the other is corrupt. I guess
both can be thought of as erased or bad due to erasure, and should consider
reconstruction. I updated the comment here.
bq.Why there are two separate methods with same params.
I'm thinking that there can be some common pre-checking and error handling
logic for all coder implementations in {{decode}} method before the call
{{performDecoding}}, though I don't have the codes for now.
For #3, #4, #5, #6, I followed and updated the codes accordingly. Thanks for
the great suggestions.
Erasure Coder API for encoding and decoding of block group
--
Key: HADOOP-11646
URL: https://issues.apache.org/jira/browse/HADOOP-11646
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Fix For: HDFS-7285
Attachments: HDFS-7662-v1.patch, HDFS-7662-v2.patch,
HDFS-7662-v3.patch
This is to define ErasureCoder API for encoding and decoding of BlockGroup.
Given a BlockGroup, ErasureCoder extracts data chunks from the blocks and
leverages RawErasureCoder defined in HDFS-7353 to perform concrete encoding
or decoding.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11646) Erasure Coder API for encoding and decoding of block group
[
https://issues.apache.org/jira/browse/HADOOP-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14350048#comment-14350048
]
Kai Zheng commented on HADOOP-11646:
I uploaded a new patch. In addition to above changes, I also reverted back the
{{coder}} package from {{blockcoder}}. When I have finally almost all the codes
in hand, the original package looks more comfortable to me. In
{{org.apache.hadoop.io.erasurecode}} we will have {{coder}}, {{codec}}, and
{{rawcoder}} sub-packages.
Erasure Coder API for encoding and decoding of block group
--
Key: HADOOP-11646
URL: https://issues.apache.org/jira/browse/HADOOP-11646
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Fix For: HDFS-7285
Attachments: HADOOP-11646-v4.patch, HDFS-7662-v1.patch,
HDFS-7662-v2.patch, HDFS-7662-v3.patch
This is to define ErasureCoder API for encoding and decoding of BlockGroup.
Given a BlockGroup, ErasureCoder extracts data chunks from the blocks and
leverages RawErasureCoder defined in HDFS-7353 to perform concrete encoding
or decoding.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11646) Erasure Coder API for encoding and decoding of block group
[ https://issues.apache.org/jira/browse/HADOOP-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14350060#comment-14350060 ] Kai Zheng commented on HADOOP-11646: Hi [~vinayrpet] and [~zhz], I updated the JIRA description to limit its scope a bit. It said: bq.Note this mainly focuses on the basic fundamental aspects, and solves encoding, data blocks recovering and etc. Regarding parity blocks recovering, as it involves multiple steps, HADOOP-11550 will handle it. Hope this works. Would you help review more ? Thanks. Erasure Coder API for encoding and decoding of block group -- Key: HADOOP-11646 URL: https://issues.apache.org/jira/browse/HADOOP-11646 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-7285 Attachments: HADOOP-11646-v4.patch, HDFS-7662-v1.patch, HDFS-7662-v2.patch, HDFS-7662-v3.patch This is to define ErasureCoder API for encoding and decoding of BlockGroup. Given a BlockGroup, ErasureCoder extracts data chunks from the blocks and leverages RawErasureCoder defined in HADOOP-11514 to perform concrete encoding or decoding. Note this mainly focuses on the basic fundamental aspects, and solves encoding, data blocks recovering and etc. Regarding parity blocks recovering, as it involves multiple steps, HADOOP-11550 will handle it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11646) Erasure Coder API for encoding and decoding of block group
[
https://issues.apache.org/jira/browse/HADOOP-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14350042#comment-14350042
]
Kai Zheng commented on HADOOP-11646:
bq.{{isErased}}, Do you mean is block is missing and needs to re-construct?
Yes right. We have two cases, one is lost at all, the other is corrupt. I guess
both can be thought of as erased or bad due to erasure, and should consider
reconstruction. I updated the comment here.
bq.Why there are two separate methods with same params.
I'm thinking that there can be some common pre-checking and error handling
logic for all coder implementations in {{decode}} method before the call
{{performDecoding}}, though I don't have the codes for now.
For #3, #4, #5, #6, I followed and updated the codes accordingly. Thanks for
the great suggestions.
Erasure Coder API for encoding and decoding of block group
--
Key: HADOOP-11646
URL: https://issues.apache.org/jira/browse/HADOOP-11646
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Fix For: HDFS-7285
Attachments: HDFS-7662-v1.patch, HDFS-7662-v2.patch,
HDFS-7662-v3.patch
This is to define ErasureCoder API for encoding and decoding of BlockGroup.
Given a BlockGroup, ErasureCoder extracts data chunks from the blocks and
leverages RawErasureCoder defined in HDFS-7353 to perform concrete encoding
or decoding.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11646) Erasure Coder API for encoding and decoding of block group
[ https://issues.apache.org/jira/browse/HADOOP-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14350043#comment-14350043 ] Kai Zheng commented on HADOOP-11646: My mouse is bad and hit two times for the post. I will update the patch according to the above discussion. Erasure Coder API for encoding and decoding of block group -- Key: HADOOP-11646 URL: https://issues.apache.org/jira/browse/HADOOP-11646 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-7285 Attachments: HADOOP-11646-v4.patch, HDFS-7662-v1.patch, HDFS-7662-v2.patch, HDFS-7662-v3.patch This is to define ErasureCoder API for encoding and decoding of BlockGroup. Given a BlockGroup, ErasureCoder extracts data chunks from the blocks and leverages RawErasureCoder defined in HDFS-7353 to perform concrete encoding or decoding. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
