[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937932#comment-14937932 ] Hudson commented on HDFS-8156: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8156. Add/implement necessary APIs even we just have the system (zhezhang: rev e8df2581c3293d0b6c43862edbf034f9e7851dbf) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECSchema.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ECSchemaManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingZoneManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingZones.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestSchemaLoader.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestECSchemas.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestECSchema.java > Add/implement necessary APIs even we just have the system default schema > > > Key: HDFS-8156 > URL: https://issues.apache.org/jira/browse/HDFS-8156 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch, > HDFS-8156-v3.patch, HDFS-8156-v4.patch, HDFS-8156-v5.patch, > HDFS-8156-v6.patch, HDFS-8156-v7.patch, HDFS-8156-v8.patch, HDFS-8156-v9.patch > > > According to the discussion here, this issue was repurposed and modified. > This is to add and implement some necessary APIs even we just have the system > default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as > they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938152#comment-14938152 ] Hudson commented on HDFS-8156: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-8156. Add/implement necessary APIs even we just have the system (zhezhang: rev e8df2581c3293d0b6c43862edbf034f9e7851dbf) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingZones.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestECSchema.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingZoneManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ECSchemaManager.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestSchemaLoader.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestECSchemas.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECSchema.java > Add/implement necessary APIs even we just have the system default schema > > > Key: HDFS-8156 > URL: https://issues.apache.org/jira/browse/HDFS-8156 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch, > HDFS-8156-v3.patch, HDFS-8156-v4.patch, HDFS-8156-v5.patch, > HDFS-8156-v6.patch, HDFS-8156-v7.patch, HDFS-8156-v8.patch, HDFS-8156-v9.patch > > > According to the discussion here, this issue was repurposed and modified. > This is to add and implement some necessary APIs even we just have the system > default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as > they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938721#comment-14938721 ] Hudson commented on HDFS-8156: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #465 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/465/]) HDFS-8156. Add/implement necessary APIs even we just have the system (zhezhang: rev e8df2581c3293d0b6c43862edbf034f9e7851dbf) * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestECSchema.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestECSchemas.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ECSchemaManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingZones.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingZoneManager.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECSchema.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestSchemaLoader.java > Add/implement necessary APIs even we just have the system default schema > > > Key: HDFS-8156 > URL: https://issues.apache.org/jira/browse/HDFS-8156 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch, > HDFS-8156-v3.patch, HDFS-8156-v4.patch, HDFS-8156-v5.patch, > HDFS-8156-v6.patch, HDFS-8156-v7.patch, HDFS-8156-v8.patch, HDFS-8156-v9.patch > > > According to the discussion here, this issue was repurposed and modified. > This is to add and implement some necessary APIs even we just have the system > default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as > they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938529#comment-14938529 ] Hudson commented on HDFS-8156: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2408 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2408/]) HDFS-8156. Add/implement necessary APIs even we just have the system (zhezhang: rev e8df2581c3293d0b6c43862edbf034f9e7851dbf) * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestSchemaLoader.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECSchema.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingZoneManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingZones.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestECSchema.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ECSchemaManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestECSchemas.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > Add/implement necessary APIs even we just have the system default schema > > > Key: HDFS-8156 > URL: https://issues.apache.org/jira/browse/HDFS-8156 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch, > HDFS-8156-v3.patch, HDFS-8156-v4.patch, HDFS-8156-v5.patch, > HDFS-8156-v6.patch, HDFS-8156-v7.patch, HDFS-8156-v8.patch, HDFS-8156-v9.patch > > > According to the discussion here, this issue was repurposed and modified. > This is to add and implement some necessary APIs even we just have the system > default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as > they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937679#comment-14937679 ] Hudson commented on HDFS-8156: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1203 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1203/]) HDFS-8156. Add/implement necessary APIs even we just have the system (zhezhang: rev e8df2581c3293d0b6c43862edbf034f9e7851dbf) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestECSchemas.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestSchemaLoader.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingZones.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ECSchemaManager.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestECSchema.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingZoneManager.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECSchema.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java > Add/implement necessary APIs even we just have the system default schema > > > Key: HDFS-8156 > URL: https://issues.apache.org/jira/browse/HDFS-8156 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch, > HDFS-8156-v3.patch, HDFS-8156-v4.patch, HDFS-8156-v5.patch, > HDFS-8156-v6.patch, HDFS-8156-v7.patch, HDFS-8156-v8.patch, HDFS-8156-v9.patch > > > According to the discussion here, this issue was repurposed and modified. > This is to add and implement some necessary APIs even we just have the system > default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as > they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937233#comment-14937233 ] Hudson commented on HDFS-8156: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2379 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2379/]) HDFS-8156. Add/implement necessary APIs even we just have the system (zhezhang: rev e8df2581c3293d0b6c43862edbf034f9e7851dbf) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ECSchemaManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestSchemaLoader.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECSchema.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestECSchemas.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingZoneManager.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestECSchema.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingZones.java > Add/implement necessary APIs even we just have the system default schema > > > Key: HDFS-8156 > URL: https://issues.apache.org/jira/browse/HDFS-8156 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch, > HDFS-8156-v3.patch, HDFS-8156-v4.patch, HDFS-8156-v5.patch, > HDFS-8156-v6.patch, HDFS-8156-v7.patch, HDFS-8156-v8.patch, HDFS-8156-v9.patch > > > According to the discussion here, this issue was repurposed and modified. > This is to add and implement some necessary APIs even we just have the system > default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as > they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937429#comment-14937429 ] Hudson commented on HDFS-8156: -- FAILURE: Integrated in Hadoop-trunk-Commit #8548 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8548/]) HDFS-8156. Add/implement necessary APIs even we just have the system (zhezhang: rev e8df2581c3293d0b6c43862edbf034f9e7851dbf) * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestECSchema.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingZoneManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestSchemaLoader.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingZones.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECSchema.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestECSchemas.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ECSchemaManager.java > Add/implement necessary APIs even we just have the system default schema > > > Key: HDFS-8156 > URL: https://issues.apache.org/jira/browse/HDFS-8156 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch, > HDFS-8156-v3.patch, HDFS-8156-v4.patch, HDFS-8156-v5.patch, > HDFS-8156-v6.patch, HDFS-8156-v7.patch, HDFS-8156-v8.patch, HDFS-8156-v9.patch > > > According to the discussion here, this issue was repurposed and modified. > This is to add and implement some necessary APIs even we just have the system > default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as > they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[
https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14507885#comment-14507885
]
Tsz Wo Nicholas Sze commented on HDFS-8156:
---
+1 the latest patch looks great. Thanks, Kai.
{quote}
Could two schemas having the same name but different set of logical
parameters?
We should avoid such schemas in a system. Will update the document reflecting
this thought.
Should two schemas having the same set of logical parameters but different
names be considered the same?
We should also avoid such schemas. Will update the document reflecting this
thought.
{quote}
Relying on documentation may not be sufficient since users could still abuse
the system and get unexpected results. The code should enforce it. Let do it
later.
Add/implement necessary APIs even we just have the system default schema
Key: HDFS-8156
URL: https://issues.apache.org/jira/browse/HDFS-8156
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch,
HDFS-8156-v3.patch, HDFS-8156-v4.patch, HDFS-8156-v5.patch,
HDFS-8156-v6.patch, HDFS-8156-v7.patch, HDFS-8156-v8.patch, HDFS-8156-v9.patch
According to the discussion here, this issue was repurposed and modified.
This is to add and implement some necessary APIs even we just have the system
default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as
they're still subject to further discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508012#comment-14508012 ] Kai Zheng commented on HDFS-8156: - Yes I have to thanks Nicholas for sorting this out! Thanks Xinwei for more review and Zhe for committing of this. Add/implement necessary APIs even we just have the system default schema Key: HDFS-8156 URL: https://issues.apache.org/jira/browse/HDFS-8156 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch, HDFS-8156-v3.patch, HDFS-8156-v4.patch, HDFS-8156-v5.patch, HDFS-8156-v6.patch, HDFS-8156-v7.patch, HDFS-8156-v8.patch, HDFS-8156-v9.patch According to the discussion here, this issue was repurposed and modified. This is to add and implement some necessary APIs even we just have the system default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14507900#comment-14507900 ] Kai Zheng commented on HDFS-8156: - bq.The code should enforce it. Let do it later. That's also what I meant, in addition to adding the thoughts and related discussion in the document in HDFS-7337. I thought we'll follow on this in HADOOP-11840. Thanks. Add/implement necessary APIs even we just have the system default schema Key: HDFS-8156 URL: https://issues.apache.org/jira/browse/HDFS-8156 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch, HDFS-8156-v3.patch, HDFS-8156-v4.patch, HDFS-8156-v5.patch, HDFS-8156-v6.patch, HDFS-8156-v7.patch, HDFS-8156-v8.patch, HDFS-8156-v9.patch According to the discussion here, this issue was repurposed and modified. This is to add and implement some necessary APIs even we just have the system default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505663#comment-14505663 ] Tsz Wo Nicholas Sze commented on HDFS-8156: --- - Please do not add extractChunkSize(). Similar to initWith(..), it is unnecessary. - The other fields numDataUnits, numParityUnits and chunkSize should also be final. - The javadoc is still incorrect. Add/implement necessary APIs even we just have the system default schema Key: HDFS-8156 URL: https://issues.apache.org/jira/browse/HDFS-8156 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch, HDFS-8156-v3.patch, HDFS-8156-v4.patch, HDFS-8156-v5.patch, HDFS-8156-v6.patch According to the discussion here, this issue was repurposed and modified. This is to add and implement some necessary APIs even we just have the system default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[
https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506221#comment-14506221
]
Xinwei Qin commented on HDFS-8156:
---
A minor comment:
The {{toString()}} method will output string similar to the following format:
{code}
ECSchema=[Name=xxx,option1=xxx,option2=xxx,option3=xxx,]
{code}
The last option will have a comma. Remove the last comma and add a whitespace
between every two options will be better.
Add/implement necessary APIs even we just have the system default schema
Key: HDFS-8156
URL: https://issues.apache.org/jira/browse/HDFS-8156
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch,
HDFS-8156-v3.patch, HDFS-8156-v4.patch, HDFS-8156-v5.patch,
HDFS-8156-v6.patch, HDFS-8156-v7.patch, HDFS-8156-v8.patch
According to the discussion here, this issue was repurposed and modified.
This is to add and implement some necessary APIs even we just have the system
default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as
they're still subject to further discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[
https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506104#comment-14506104
]
Tsz Wo Nicholas Sze commented on HDFS-8156:
---
This one is much better. Thanks. Some minor comments:
- Should the DEFAULT_CODEC_NAME be RS?
{code}
private static final String DEFAULT_CODEC_NAME = rs;
- private static final String DEFAULT_SCHEMA_NAME = SYS-DEFAULT-RS-6-3;
+ private static final String DEFAULT_SCHEMA_NAME = RS-6-3;
{code}
- Please remove the comment. Otherwise, the generated javadoc won't look good.
{code}
+ /**
+ * Most common and general parameters found in typical erasure codes:
+ * numDataUnits, numParityUnits, chunkSize
+ */
{code}
- Please refactor try-catch(NumberFormatException) statements to a method so
that numDataUnits, numParityUnits and chunkSize can all use it.
Add/implement necessary APIs even we just have the system default schema
Key: HDFS-8156
URL: https://issues.apache.org/jira/browse/HDFS-8156
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch,
HDFS-8156-v3.patch, HDFS-8156-v4.patch, HDFS-8156-v5.patch,
HDFS-8156-v6.patch, HDFS-8156-v7.patch
According to the discussion here, this issue was repurposed and modified.
This is to add and implement some necessary APIs even we just have the system
default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as
they're still subject to further discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504023#comment-14504023 ] Kai Zheng commented on HDFS-8156: - bq.It seems not hard to do it. Just need to move the code around. Doing so we have to duplicate codes in initWith in the first constructor. I thought something like initWith is a typical pratice to share codes among constructors. Would you double check, thanks! Add/implement necessary APIs even we just have the system default schema Key: HDFS-8156 URL: https://issues.apache.org/jira/browse/HDFS-8156 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch, HDFS-8156-v3.patch, HDFS-8156-v4.patch, HDFS-8156-v5.patch According to the discussion here, this issue was repurposed and modified. This is to add and implement some necessary APIs even we just have the system default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504037#comment-14504037 ] Tsz Wo Nicholas Sze commented on HDFS-8156: --- Doing so we have to duplicate codes in initWith in the first constructor. I thought something like initWith is a typical pratice to share codes among constructors. ... Why the code have to be duplicated? No duplication if the code is well organized. Adding another method (initWith) is unnecessary. Add/implement necessary APIs even we just have the system default schema Key: HDFS-8156 URL: https://issues.apache.org/jira/browse/HDFS-8156 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch, HDFS-8156-v3.patch, HDFS-8156-v4.patch, HDFS-8156-v5.patch According to the discussion here, this issue was repurposed and modified. This is to add and implement some necessary APIs even we just have the system default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504056#comment-14504056 ] Kai Zheng commented on HDFS-8156: - I would appreciate if you could illustrate the idea with codes doing it avoiding the duplication. Thanks. Add/implement necessary APIs even we just have the system default schema Key: HDFS-8156 URL: https://issues.apache.org/jira/browse/HDFS-8156 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch, HDFS-8156-v3.patch, HDFS-8156-v4.patch, HDFS-8156-v5.patch According to the discussion here, this issue was repurposed and modified. This is to add and implement some necessary APIs even we just have the system default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[
https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502619#comment-14502619
]
Kai Zheng commented on HDFS-8156:
-
Hi [~szetszwo],
bq.Why do we need initWith? Could initWith be changed to a constructor and
change all the fields to final, i.e. making ECSchema immutable?
{{initWith}} is needed because the codes in it couldn't be in a constructor as
it needs to be called in the end of another constructor. It's already immutable
now even the fields are not final because no setting is allowed once
constructed.
bq.Could two schemas having the same name but different set of logical
parameters?
We should avoid such schemas in a system. Will update the document reflecting
this thought.
bq.Should two schemas having the same set of logical parameters but different
names be considered the same?
We should also avoid such schemas. Will update the document reflecting this
thought.
Maybe we can enhance {{ECSchema}} to add such comparision logics when we
support multiple schemas. For now I didn't change the codes.
Patch was updated for your other comments. Please help review one more time,
thanks a lot!
Add/implement necessary APIs even we just have the system default schema
Key: HDFS-8156
URL: https://issues.apache.org/jira/browse/HDFS-8156
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch,
HDFS-8156-v3.patch, HDFS-8156-v4.patch, HDFS-8156-v5.patch
According to the discussion here, this issue was repurposed and modified.
This is to add and implement some necessary APIs even we just have the system
default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as
they're still subject to further discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503329#comment-14503329 ] Tsz Wo Nicholas Sze commented on HDFS-8156: --- initWith is needed because the codes in it couldn't be in a constructor as it needs to be called in the end of another constructor. ... It seems not hard to do it. Just need to move the code around. Add/implement necessary APIs even we just have the system default schema Key: HDFS-8156 URL: https://issues.apache.org/jira/browse/HDFS-8156 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch, HDFS-8156-v3.patch, HDFS-8156-v4.patch, HDFS-8156-v5.patch According to the discussion here, this issue was repurposed and modified. This is to add and implement some necessary APIs even we just have the system default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499585#comment-14499585 ] Kai Zheng commented on HDFS-8156: - I had an offline discussion with [~umamaheswararao], [~vinayrpet], and [~hitliuyi], we thought it's OK for our side to have only ONE schema in the phase 1 since [~szetszwo] insists on that. I repurposed and modified the JIRA accordingly. I will upload the corresponding patch soon. Please review, thanks. Add/implement necessary APIs even we just have the system default schema Key: HDFS-8156 URL: https://issues.apache.org/jira/browse/HDFS-8156 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch, HDFS-8156-v3.patch According to the discussion here, this issue was repurposed and modified. This is to add and implement some necessary APIs even we just have the system default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500420#comment-14500420 ] Tsz Wo Nicholas Sze commented on HDFS-8156: --- ... Regarding the test concern, please note RS-10-4 was already of unit tests when coming up the coder. We would add more what's left. Please don't add anything for RS-10-4 for the moment. Testing the entire erasure code feature is different from having a few unit test for RS-10-4. We even do not have a test plan posted in HDFS-7285 yet. Add/implement necessary APIs even we just have the system default schema Key: HDFS-8156 URL: https://issues.apache.org/jira/browse/HDFS-8156 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch, HDFS-8156-v3.patch, HDFS-8156-v4.patch According to the discussion here, this issue was repurposed and modified. This is to add and implement some necessary APIs even we just have the system default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[
https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500447#comment-14500447
]
Tsz Wo Nicholas Sze commented on HDFS-8156:
---
- Should chunkSize be renamed to cellSize?
- The javadoc below is wrong. A javadoc block does not work for multiple
fields. We need to have a javadoc block for each field.
{code}
+ /**
+ * The most common and general parameters found in typical erasure codes.
+ */
private int numDataUnits;
private int numParityUnits;
private int chunkSize;
{code}
- Why do we need initWith? Could initWith be changed to a constructor and
change all the fields to final, i.e. making ECSchema immutable?
- A schemas have a name and a set of logical parameters (i.e. numDataUnits,
numParityUnits, chunkSize, extraOptions).
-* Could two schemas having the same name but different set of logical
parameters?
-* Should two schemas having the same set of logical parameters but different
names be considered the same?
Add/implement necessary APIs even we just have the system default schema
Key: HDFS-8156
URL: https://issues.apache.org/jira/browse/HDFS-8156
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch,
HDFS-8156-v3.patch, HDFS-8156-v4.patch
According to the discussion here, this issue was repurposed and modified.
This is to add and implement some necessary APIs even we just have the system
default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as
they're still subject to further discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500626#comment-14500626 ] Tsz Wo Nicholas Sze commented on HDFS-8156: --- I had an offline discussion with Uma Maheswara Rao G, Vinayakumar B, and Yi Liu, we thought it's OK for our side to have only ONE schema in the phase 1 since Tsz Wo Nicholas Sze insists on that. ... That's great. Thanks a lot! Add/implement necessary APIs even we just have the system default schema Key: HDFS-8156 URL: https://issues.apache.org/jira/browse/HDFS-8156 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch, HDFS-8156-v3.patch, HDFS-8156-v4.patch According to the discussion here, this issue was repurposed and modified. This is to add and implement some necessary APIs even we just have the system default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[
https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500659#comment-14500659
]
Tsz Wo Nicholas Sze commented on HDFS-8156:
---
- Why ECSchemaManager needs to store the conf? It is not used.
{code}
+ private final Configuration conf;
{code}
- There are unused imports.
Add/implement necessary APIs even we just have the system default schema
Key: HDFS-8156
URL: https://issues.apache.org/jira/browse/HDFS-8156
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch,
HDFS-8156-v3.patch, HDFS-8156-v4.patch
According to the discussion here, this issue was repurposed and modified.
This is to add and implement some necessary APIs even we just have the system
default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as
they're still subject to further discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500882#comment-14500882 ] Kai Zheng commented on HDFS-8156: - Thanks [~szetszwo] for the good review. bq.Should chunkSize be renamed to cellSize? In my understanding, chunk could be used in erasure coding context, as we would perform coding chunk by chunk for blocks. cell is more used in stripping context. ECSchema is defined as part of the erasure codec framework so chunkSize is used. It's consistent in the framework with other parts. Other points are all good and I will address them next week. Thanks. Add/implement necessary APIs even we just have the system default schema Key: HDFS-8156 URL: https://issues.apache.org/jira/browse/HDFS-8156 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch, HDFS-8156-v3.patch, HDFS-8156-v4.patch According to the discussion here, this issue was repurposed and modified. This is to add and implement some necessary APIs even we just have the system default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
