[jira] [Commented] (HDFS-8181) createErasureCodingZone sets retryCache state as false always
[ https://issues.apache.org/jira/browse/HDFS-8181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937941#comment-14937941 ] Hudson commented on HDFS-8181: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8181. createErasureCodingZone sets retryCache state as false always (zhezhang: rev dfba46ab577ef76b681d1848ded2be269079602a) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java > createErasureCodingZone sets retryCache state as false always > - > > Key: HDFS-8181 > URL: https://issues.apache.org/jira/browse/HDFS-8181 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: HDFS-7285 > > Attachments: HDFS-8181-0.patch > > > Currently createErasureCodingZone sets the RetryCache state as false always. > Once the operation success it should pass true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7936) Erasure coding: resolving conflicts in the branch when merging trunk changes.
[ https://issues.apache.org/jira/browse/HDFS-7936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937916#comment-14937916 ] Hudson commented on HDFS-7936: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-7936. Erasure coding: resolving conflicts when merging with (zhezhang: rev 11585883a9eb30ba080b9aa49dba42cb0a797d75) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiffList.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java HDFS-7936. Erasure coding: resolving conflicts when merging with (zhezhang: rev 544f75d6512fefd0e36f24a35e6b7472ca7bf301) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java HDFS-7936. Erasure coding: resolving conflicts when merging with (zhezhang: rev ea2e60fbcc79c65ec571224bd3f57c262a5d9114) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNameNodePrunesMissingStorages.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java HDFS-7936. Erasure coding: resolving conflicts in the branch when (zhezhang: rev 97378e4cd0902fcfe4d591c92b47fc292c032438) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java HDFS-7936. Erasure coding: resolving conflicts in the branch when (zhezhang: rev 578019d6a27bd3128fae1bdf7caadd8767b91e8c) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java HDFS-7936. Erasure coding: resolving conflicts in the branch when (zhezhang: rev c243319eab89928bc06e0357a50f9422cbe54f7d) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java HDFS-7936. Erasure coding: resolving conflicts in the branch when (zhezhang: rev 35797b08899195170715835568f96ab51bef0d98) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java HDFS-7936. Erasure coding: resolving conflicts in the branch when (zhezhang: rev 6bacaa9a5233cbad7f311ccd9d8f8dc9375c732d) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestTruncateQuotaUpdate.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStripedINodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java > Erasure coding: resolving conflicts in the branch when merging trunk changes. > -- > > Key: HDFS-7936 > URL: https://issues.apache.org/jira/browse/HDFS-7936 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: HDFS-7285 > > Attachments: HDFS-7936-001.patch, HDFS-7936-002.patch, > HDFS-7936-003.patch, HDFS-7936-004.patch, HDFS-7936-005.patch > > > This will be used to track and resolve conflicts when merging trunk changes. > Below is a list of trunk changes that have caused conflicts (updated weekly): > # HDFS-7903 > # HDFS-7435 > # HDFS-7930 > # HDFS-7960 > # HDFS-7742 > # HDFS-8035 > # HDFS-8169 > # HDFS-8327 > # HDFS-8357 -- This message was sent by
[jira] [Commented] (HDFS-8760) Erasure Coding: reuse BlockReader when reading the same block in pread
[ https://issues.apache.org/jira/browse/HDFS-8760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937937#comment-14937937 ] Hudson commented on HDFS-8760: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8760. Erasure Coding: reuse BlockReader when reading the same block (jing9: rev f8f7a923b76abcd1d0242c15a536b20af1c1695e) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteReadStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/StripedFileTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > Erasure Coding: reuse BlockReader when reading the same block in pread > -- > > Key: HDFS-8760 > URL: https://issues.apache.org/jira/browse/HDFS-8760 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: HDFS-7285 > > Attachments: HDFS-8760-HDFS-7285.001.patch, HDFS-8760.000.patch > > > Currently in pread, we create a new block reader for each aligned stripe even > though these stripes belong to the same block. It's better to reuse them to > avoid unnecessary block reader creation overhead. This can also avoid reading > from the same bad DataNode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block
[ https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937912#comment-14937912 ] Hudson commented on HDFS-8543: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8543. Erasure Coding: processOverReplicatedBlock() handles striped (jing9: rev 448cb7df676d3c0f5fdc52fbbe736f3b54e519a3) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddOverReplicatedStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java > Erasure Coding: processOverReplicatedBlock() handles striped block > -- > > Key: HDFS-8543 > URL: https://issues.apache.org/jira/browse/HDFS-8543 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Walter Su >Assignee: Walter Su > Fix For: HDFS-7285 > > Attachments: HDFS-8543-HDFS-7285.01.patch, > HDFS-8543-HDFS-7285.02.patch, HDFS-8543-HDFS-7285.03.patch, > HDFS-8543-HDFS-7285.04.patch, HDFS-8543-HDFS-7285.05.patch, > HDFS-8543-HDFS-7285.06.patch, HDFS-8543-HDFS-7285.07.patch > > > striped block group could be over replicated when: 1.dead DN comes back. 2. > Balancer/Mover copies block before deletes it. > This jira add logic for processOverReplicatedBlock() handling striped block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8619) Erasure Coding: revisit replica counting for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937919#comment-14937919
]
Hudson commented on HDFS-8619:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8619. Erasure Coding: revisit replica counting for striped blocks. (yliu:
rev f32d9a175837c5b6c3d008089e46475d27a0935c)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecoding.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java
> Erasure Coding: revisit replica counting for striped blocks
> ---
>
> Key: HDFS-8619
> URL: https://issues.apache.org/jira/browse/HDFS-8619
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-8619-HDFS-7285.001.patch, HDFS-8619.000.patch
>
>
> Currently we use the same {{BlockManager#countNodes}} method for striped
> blocks, which simply treat each internal block as a replica. However, for a
> striped block, we may have more complicated scenario, e.g., we have multiple
> replicas of the first internal block while we miss some other internal
> blocks. Using the current {{countNodes}} methods can lead to wrong decision
> in these scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8005) Erasure Coding: simplify striped block recovery work computation and add tests
[
https://issues.apache.org/jira/browse/HDFS-8005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937918#comment-14937918
]
Hudson commented on HDFS-8005:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8005. Erasure Coding: simplify striped block recovery work (zhezhang: rev
cd655ee817a3307bc1a1a119eb4266978ecd7fb2)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestRecoverStripedBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestRecoverStripedBlocks.java
> Erasure Coding: simplify striped block recovery work computation and add tests
> --
>
> Key: HDFS-8005
> URL: https://issues.apache.org/jira/browse/HDFS-8005
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Fix For: HDFS-7285
>
> Attachments: HDFS-8005.000.patch, HDFS-8005.000.patch,
> HDFS-8005.001.patch
>
>
> HDFS-7369 adds the functionality to distribute recovery work of striped
> blocks to datanodes. There are still some pending issues:
> # In {{BlockManager#chooseSourceNode}}, a node is added into
> {{healthyIndices}} without checking if its block is live and healthy
> # The test {{TestRecoverStripedBlcoks#testMissingStripedBlock}} has not
> tested striped blocks because the file is created before setting the storage
> policy
> # In {{computeRecoveryWorkForBlocks}}, instead of using
> {{BlockCollection#isStriped}}, we'd better use {{BlockInfo#isStriped}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8408) Revisit and refactor ErasureCodingInfo
[
https://issues.apache.org/jira/browse/HDFS-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937914#comment-14937914
]
Hudson commented on HDFS-8408:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8408. Revisit and refactor ErasureCodingInfo (Contributed by (zhezhang:
rev 9a18598e2da8e699ed852ffa30fd7f503902190c)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ErasureCodingInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/erasurecoding.proto
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/erasurecode/ECCommand.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingZoneManager.java
* hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testErasureCodingConf.xml
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingZones.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ErasureCodingZoneInfo.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java
HDFS-8408. Revisit and refactor ErasureCodingInfo (Contributed by (zhezhang:
rev 80c56c2d3167f5195c72802071ea6dfb52abff1a)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ErasureCodingZone.java
> Revisit and refactor ErasureCodingInfo
> --
>
> Key: HDFS-8408
> URL: https://issues.apache.org/jira/browse/HDFS-8408
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
>Priority: Minor
> Fix For: HDFS-7285
>
> Attachments: HDFS-8408-HDFS-7285-01.patch,
> HDFS-8408-HDFS-7285-02.patch
>
>
> As mentioned in HDFS-8375
> [here|https://issues.apache.org/jira/browse/HDFS-8375?focusedCommentId=14544618=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14544618]
>
> {{ErasureCodingInfo}} needs a revisit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8023) Erasure Coding: retrieve eraure coding schema for a file from NameNode
[ https://issues.apache.org/jira/browse/HDFS-8023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937911#comment-14937911 ] Hudson commented on HDFS-8023: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8023. Erasure Coding: retrieve eraure coding schema for a file from (zhezhang: rev 16484f00308c8cdee1f4b26639587cfb171ea341) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingZones.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java HDFS-8023. Erasure Coding: retrieve eraure coding schema for a file from (zhezhang: rev 2ca3e38feb4191ef17fd58b8b4719fd5297a5e21) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ECInfo.java > Erasure Coding: retrieve eraure coding schema for a file from NameNode > -- > > Key: HDFS-8023 > URL: https://issues.apache.org/jira/browse/HDFS-8023 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Vinayakumar B > Fix For: HDFS-7285 > > Attachments: HDFS-8023-01.patch, HDFS-8023-02.patch > > > NameNode needs to provide RPC call for client and tool to retrieve eraure > coding schema for a file from NameNode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8183) Erasure Coding: Improve DFSStripedOutputStream closing of datastreamer threads
[ https://issues.apache.org/jira/browse/HDFS-8183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937925#comment-14937925 ] Hudson commented on HDFS-8183: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8183. Erasure Coding: Improve DFSStripedOutputStream closing of (zhezhang: rev 1a31f1c303e8f541f0b22eccf85f3b789a220f21) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > Erasure Coding: Improve DFSStripedOutputStream closing of datastreamer threads > -- > > Key: HDFS-8183 > URL: https://issues.apache.org/jira/browse/HDFS-8183 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rakesh R >Assignee: Rakesh R > Fix For: HDFS-7285 > > Attachments: HDFS-8183-001.patch, HDFS-8183-002.patch > > > The idea of this task is to improve closing of all the streamers. Presently > if any of the streamer throws an exception, it will returning immediately. > This leaves all the other streamer threads running. Instead its good to > handle the exceptions of each streamer independently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7839) Erasure coding: implement facilities in NameNode to create and manage EC zones
[ https://issues.apache.org/jira/browse/HDFS-7839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937902#comment-14937902 ] Hudson commented on HDFS-7839: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-7839. Erasure coding: implement facilities in NameNode to create (zhezhang: rev 1af8c148626effe1b41fc536019fd3349f485d59) * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInitialEncoding.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingZoneManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingZones.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestRecoverStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java > Erasure coding: implement facilities in NameNode to create and manage EC zones > -- > > Key: HDFS-7839 > URL: https://issues.apache.org/jira/browse/HDFS-7839 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: HDFS-7285 > > Attachments: HDFS-7839-000.patch, HDFS-7839-001.patch, > HDFS-7839-002.patch, HDFS-7839-003.patch > > > As a quick first step to facilitate initial development and testing, > HDFS-7347 added EC configuration in file header as one storage policy. We > have discussed and [concluded > |https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14296210=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14296210] > that EC configurations should be part of XAttr. This JIRA aims to add the > basic EC XAttr structure. HDFS-7337 will add configurable and pluggable > schema info. > This JIRA will follow the [plan | > https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14370307=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14370307] > we made under HDFS-7285. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7837) Erasure Coding: allocate and persist striped blocks in NameNode
[ https://issues.apache.org/jira/browse/HDFS-7837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937907#comment-14937907 ] Hudson commented on HDFS-7837: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-7837. Erasure Coding: allocate and persist striped blocks in (zhezhang: rev 1e1e93040748231dc913190aec1e031c379d8271) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/fsimage.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeLayoutVersion.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockgroup.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java > Erasure Coding: allocate and persist striped blocks in NameNode > --- > > Key: HDFS-7837 > URL: https://issues.apache.org/jira/browse/HDFS-7837 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: HDFS-7285 > > Attachments: HDFS-7837.000.patch, HDFS-7837.001.patch > > > Try to finish the remaining work from HDFS-7339 (except the > ClientProtocol/DFSClient part): > # Allow FSNamesystem#getAdditionalBlock to create striped blocks and persist > striped blocks to editlog > # Update FSImage for max allocated striped block ID > # Update the block commit/complete logic in BlockManager -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8186) Erasure coding: Make block placement policy for EC file configurable
[ https://issues.apache.org/jira/browse/HDFS-8186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937927#comment-14937927 ] Hudson commented on HDFS-8186: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8186. Erasure coding: Make block placement policy for EC file (zhezhang: rev e53fa769c97416af69ea567aecd44f67e896688b) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicies.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java > Erasure coding: Make block placement policy for EC file configurable > > > Key: HDFS-8186 > URL: https://issues.apache.org/jira/browse/HDFS-8186 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Walter Su >Assignee: Walter Su > Fix For: HDFS-7285 > > Attachments: HDFS-8186-HDFS-7285.002.txt, > HDFS-8186-HDFS-7285.003.patch, HDFS-8186.001.txt > > > This includes: > 1. User can config block placement policy for EC file in xml configuration > file. > 2. EC policy works for EC file, replication policy works for non-EC file. > They are coexistent. > Not includes: > 1. Details of block placement policy for EC. Discussion and implementation > goes to HDFS-7613. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8978) Erasure coding: fix 2 failed tests of DFSStripedOutputStream
[ https://issues.apache.org/jira/browse/HDFS-8978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937923#comment-14937923 ] Hudson commented on HDFS-8978: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8978. Erasure coding: fix 2 failed tests of DFSStripedOutputStream. (jing9: rev 60bd765ac1b7d21189efc47058c00ff956a2dc86) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteReadStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecoding.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/StripedFileTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithMissingBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java > Erasure coding: fix 2 failed tests of DFSStripedOutputStream > > > Key: HDFS-8978 > URL: https://issues.apache.org/jira/browse/HDFS-8978 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Reporter: Walter Su >Assignee: Walter Su >Priority: Minor > Fix For: HDFS-7285 > > Attachments: HDFS-8978-HDFS-7285.01.patch, > HDFS-8978-HDFS-7285.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8882) Erasure Coding: Use datablocks, parityblocks and cell size from ErasureCodingPolicy
[ https://issues.apache.org/jira/browse/HDFS-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937935#comment-14937935 ] Hudson commented on HDFS-8882: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8882. Erasure Coding: Use datablocks, parityblocks and cell size (zhezhang: rev c09dc258a8f64fab852bf6f26187163480dbee3c) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithMissingBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestSequentialBlockGroupId.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFSStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeModeWithStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteReadStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlocksWithLocations.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestRecoverStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStripedINodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/StripedFileTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddOverReplicatedStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecoding.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewerWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteStripedFileWithFailure.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestStripedBlockUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java > Erasure Coding: Use datablocks, parityblocks and cell size from > ErasureCodingPolicy > --- > > Key: HDFS-8882 > URL: https://issues.apache.org/jira/browse/HDFS-8882 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Vinayakumar B >Assignee: Vinayakumar B >
[jira] [Commented] (HDFS-8366) Erasure Coding: Make the timeout parameter of polling blocking queue configurable in DFSStripedOutputStream
[
https://issues.apache.org/jira/browse/HDFS-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937897#comment-14937897
]
Hudson commented on HDFS-8366:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8366. Erasure Coding: Make the timeout parameter of polling (zhezhang: rev
8c95673db47763650c4bb9f9bf39ef770aa73cdf)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
> Erasure Coding: Make the timeout parameter of polling blocking queue
> configurable in DFSStripedOutputStream
> ---
>
> Key: HDFS-8366
> URL: https://issues.apache.org/jira/browse/HDFS-8366
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Li Bo
>Assignee: Li Bo
> Attachments: HDFS-8366-001.patch, HDFS-8366-HDFS-7285-02.patch
>
>
> The timeout of getting striped or ended block in
> {{DFSStripedOutputStream#Coodinator}} should be configurable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8563) Erasure Coding: fsck handles file smaller than a full stripe
[
https://issues.apache.org/jira/browse/HDFS-8563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937920#comment-14937920
]
Hudson commented on HDFS-8563:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8563. Erasure Coding: fsck handles file smaller than a full stripe.
(jing9: rev 2470a7bf88c7d7e0889905e35671c440c3ae649b)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
> Erasure Coding: fsck handles file smaller than a full stripe
>
>
> Key: HDFS-8563
> URL: https://issues.apache.org/jira/browse/HDFS-8563
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: HDFS-7285
>
> Attachments: HDFS-8563-HDFS-7285.01.patch,
> HDFS-8563-HDFS-7285.02.patch
>
>
> Uploaded a small file. Fsck shows it's UNRECOVERABLE. It's not correct.
> {noformat}
> Erasure Coded Block Groups:
> Total size:1366 B
> Total files: 1
> Total block groups (validated):1 (avg. block group size 1366 B)
>
> UNRECOVERABLE BLOCK GROUPS: 1 (100.0 %)
> MIN REQUIRED EC BLOCK:6
>
> Minimally erasure-coded block groups: 0 (0.0 %)
> Over-erasure-coded block groups: 0 (0.0 %)
> Under-erasure-coded block groups: 1 (100.0 %)
> Unsatisfactory placement block groups: 0 (0.0 %)
> Default schema:RS-6-3
> Average block group size: 4.0
> Missing block groups: 0
> Corrupt block groups: 0
> Missing ec-blocks: 5 (55.57 %)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8355) Erasure Coding: Refactor BlockInfo and BlockInfoUnderConstruction
[ https://issues.apache.org/jira/browse/HDFS-8355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937924#comment-14937924 ] Hudson commented on HDFS-8355: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8355. Erasure Coding: Refactor BlockInfo and (zhezhang: rev 51ea117f883f9c049de58987dc66e07e71a68ee4) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguousUnderConstruction.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStripedUnderConstruction.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > Erasure Coding: Refactor BlockInfo and BlockInfoUnderConstruction > - > > Key: HDFS-8355 > URL: https://issues.apache.org/jira/browse/HDFS-8355 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Minor > Fix For: HDFS-7285 > > Attachments: h8355_20150508.patch > > > A few static methods in BlockInfo can be declared in > BlockInfoUnderConstruction so that the subclasses could implement them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7749) Erasure Coding: Add striped block support in INodeFile
[ https://issues.apache.org/jira/browse/HDFS-7749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937933#comment-14937933 ] Hudson commented on HDFS-7749: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-7749. Erasure Coding: Add striped block support in INodeFile. (zhezhang: rev 9f2f583f401189c3f4a2687795a9e3e0b288322b) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguousUnderConstruction.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockgroup.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStripedUnderConstruction.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileWithStripedBlocksFeature.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotBlocksMap.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlock.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestBlockUnderConstruction.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotDeletion.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileUnderConstructionFeature.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/fsimage.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiffList.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java Addendum fix for HDFS-7749 to be compatible with HDFS-7993 (zhezhang: rev 9af1f4779b646fb2f09b5e36447c8b8abe920a7c) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java > Erasure Coding: Add striped block support in INodeFile > -- > > Key: HDFS-7749 > URL: https://issues.apache.org/jira/browse/HDFS-7749 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: HDFS-7285 > > Attachments: HDFS-7749.000.patch, HDFS-7749.001.patch > > > This jira
[jira] [Commented] (HDFS-8418) Fix the isNeededReplication calculation for Striped block in NN
[
https://issues.apache.org/jira/browse/HDFS-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937926#comment-14937926
]
Hudson commented on HDFS-8418:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8418. Fix the isNeededReplication calculation for Striped block in
(zhezhang: rev b008348dbf9bdd5070930be5d182116c5d370f6b)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
> Fix the isNeededReplication calculation for Striped block in NN
> ---
>
> Key: HDFS-8418
> URL: https://issues.apache.org/jira/browse/HDFS-8418
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
>Priority: Critical
> Fix For: HDFS-7285
>
> Attachments: HDFS-8418-HDFS-7285.001.patch
>
>
> Currently when calculating {{isNeededReplication}} for striped block, we use
> BlockCollection#getPreferredBlockReplication to get expected replica number
> for striped block. See an example:
> {code}
> public void checkReplication(BlockCollection bc) {
> final short expected = bc.getPreferredBlockReplication();
> for (BlockInfo block : bc.getBlocks()) {
> final NumberReplicas n = countNodes(block);
> if (isNeededReplication(block, expected, n.liveReplicas())) {
> neededReplications.add(block, n.liveReplicas(),
> n.decommissionedAndDecommissioning(), expected);
> } else if (n.liveReplicas() > expected) {
> processOverReplicatedBlock(block, expected, null, null);
> }
> }
> }
> {code}
> But actually it's not correct, for example, if the length of striped file is
> less than a cell, then the expected replica of the block should be {{1 +
> parityBlkNum}} instead of {{dataBlkNum + parityBlkNum}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8417) Erasure Coding: Pread failed to read data starting from not-first stripe
[ https://issues.apache.org/jira/browse/HDFS-8417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937928#comment-14937928 ] Hudson commented on HDFS-8417: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8417. Erasure Coding: Pread failed to read data starting from (zhezhang: rev 6c310db15939bf5199894060607ec10ecfbe0877) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java > Erasure Coding: Pread failed to read data starting from not-first stripe > > > Key: HDFS-8417 > URL: https://issues.apache.org/jira/browse/HDFS-8417 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Walter Su >Assignee: Walter Su > Fix For: HDFS-7285 > > Attachments: HDFS-8417-HDFS-7285.001.patch, > HDFS-8417-HDFS-7285.002.patch > > > When file length is cellSize * dataBlocks +123, and Pread start from cellSize > * dataBlocks + 1, it will fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8156) Add/implement necessary APIs even we just have the system default schema
[ https://issues.apache.org/jira/browse/HDFS-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937932#comment-14937932 ] Hudson commented on HDFS-8156: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8156. Add/implement necessary APIs even we just have the system (zhezhang: rev e8df2581c3293d0b6c43862edbf034f9e7851dbf) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECSchema.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ECSchemaManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingZoneManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingZones.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestSchemaLoader.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestECSchemas.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestECSchema.java > Add/implement necessary APIs even we just have the system default schema > > > Key: HDFS-8156 > URL: https://issues.apache.org/jira/browse/HDFS-8156 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HDFS-8156-v1.patch, HDFS-8156-v2.patch, > HDFS-8156-v3.patch, HDFS-8156-v4.patch, HDFS-8156-v5.patch, > HDFS-8156-v6.patch, HDFS-8156-v7.patch, HDFS-8156-v8.patch, HDFS-8156-v9.patch > > > According to the discussion here, this issue was repurposed and modified. > This is to add and implement some necessary APIs even we just have the system > default schema, to resolve some TODOs left for HDFS-7859 and HDFS-7866 as > they're still subject to further discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8189) ClientProtocol#createErasureCodingZone API was wrongly annotated as Idempotent
[
https://issues.apache.org/jira/browse/HDFS-8189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937931#comment-14937931
]
Hudson commented on HDFS-8189:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8189. ClientProtocol#createErasureCodingZone API was wrongly (zhezhang:
rev 5eb17e57265655b9c704e7fe664a29071f7902ae)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
> ClientProtocol#createErasureCodingZone API was wrongly annotated as Idempotent
> --
>
> Key: HDFS-8189
> URL: https://issues.apache.org/jira/browse/HDFS-8189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Vinayakumar B
> Fix For: HDFS-7285
>
> Attachments: HDFS-8189-01.patch
>
>
> Currently createErasureCodingZone was annotated as Idempotent
> But it should be annotated as @AtMostOnce as we handle retries via retryCache.
> {code}
> @Idempotent
> public void createErasureCodingZone(String src, ECSchema schema)
> throws IOException;
> {code}
> It will fail to create Zone if its already a zone. So, simply we can not
> retry by ignoring previous call success. So, we were using retryCache already
> for handling this situation.
> {code}
> if (getECSchema(srcIIP) != null) {
> throw new IOException("Directory " + src + " is already in an " +
> "erasure coding zone.");
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8203) Erasure Coding: Seek and other Ops in DFSStripedInputStream.
[
https://issues.apache.org/jira/browse/HDFS-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937939#comment-14937939
]
Hudson commented on HDFS-8203:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8203. Erasure Coding: Seek and other Ops in DFSStripedInputStream.
(zhezhang: rev a17cedb44c2a98d863407fab95c2b7f0893d0727)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteReadStripedFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
> Erasure Coding: Seek and other Ops in DFSStripedInputStream.
>
>
> Key: HDFS-8203
> URL: https://issues.apache.org/jira/browse/HDFS-8203
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: HDFS-7285
>
> Attachments: HDFS-8203.001.patch, HDFS-8203.002.patch,
> HDFS-8203.003.patch
>
>
> In HDFS-7782 and HDFS-8033, we handle pread and stateful read for
> {{DFSStripedInputStream}}, we also need handle other operations, such as
> {{seek}}, zerocopy read ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8212) DistributedFileSystem.createErasureCodingZone should pass schema in FileSystemLinkResolver
[
https://issues.apache.org/jira/browse/HDFS-8212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937917#comment-14937917
]
Hudson commented on HDFS-8212:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8212. DistributedFileSystem.createErasureCodingZone should pass (zhezhang:
rev 2f11109bb0db1f3baf802a345f2de840b1553493)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
> DistributedFileSystem.createErasureCodingZone should pass schema in
> FileSystemLinkResolver
> --
>
> Key: HDFS-8212
> URL: https://issues.apache.org/jira/browse/HDFS-8212
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Fix For: HDFS-7285
>
> Attachments: h8212_20150421.patch
>
>
> It current passes null and ignores the input schema.
> {code}
> public void createErasureCodingZone(final Path path, final ECSchema schema)
> throws IOException {
> Path absF = fixRelativePart(path);
> new FileSystemLinkResolver() {
> @Override
> public Void doCall(final Path p) throws IOException,
> UnresolvedLinkException {
> dfs.createErasureCodingZone(getPathName(p), null);
> return null;
> }
> ...
> }.resolve(this, absF);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8559) Erasure Coding: fix non-protobuf fsimage for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937938#comment-14937938
]
Hudson commented on HDFS-8559:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8559. Erasure Coding: fix non-protobuf fsimage for striped blocks. (yliu:
rev 49d5cff49011cc0878665204e22b5c832bc914ce)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
> Erasure Coding: fix non-protobuf fsimage for striped blocks
> ---
>
> Key: HDFS-8559
> URL: https://issues.apache.org/jira/browse/HDFS-8559
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
>Priority: Minor
> Attachments: HDFS-8559.000.patch
>
>
> For a legacy fsimage we currently always record its layoutversion as -51 so
> that we can make sure it cannot be processed by a protobuf-based image
> parser. Thus the following code in the parser always returns false and the
> parse will fail.
> {code}
> NameNodeLayoutVersion.supports(
> NameNodeLayoutVersion.Feature.ERASURE_CODING, imgVersion)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8702) Erasure coding: update BlockManager.blockHasEnoughRacks(..) logic for striped block
[ https://issues.apache.org/jira/browse/HDFS-8702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937841#comment-14937841 ] Hudson commented on HDFS-8702: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8702. Erasure coding: update BlockManager.blockHasEnoughRacks(..) (jing9: rev 6ff957be88d48a8b41e9fcbe4cf466d672cd7bc1) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > Erasure coding: update BlockManager.blockHasEnoughRacks(..) logic for striped > block > --- > > Key: HDFS-8702 > URL: https://issues.apache.org/jira/browse/HDFS-8702 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Walter Su >Assignee: Kai Sasaki > Fix For: HDFS-7285 > > Attachments: HDFS-8702-HDFS-7285.00.patch, > HDFS-8702-HDFS-7285.01.patch, HDFS-8702-HDFS-7285.02.patch, > HDFS-8702-HDFS-7285.03.patch, HDFS-8702-HDFS-7285.04.patch > > > Currently blockHasEnoughRacks(..) only guarantees 2 racks. Logic needs > updated for striped blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7672) Erasure Coding: consolidate streamer coordination logic and handle failure when writing striped blocks
[ https://issues.apache.org/jira/browse/HDFS-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937846#comment-14937846 ] Hudson commented on HDFS-7672: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-7672. Handle write failure for stripping blocks and refactor the (zhezhang: rev 220ca960bce970d5969b9af570a3ce43360b7e2b) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java > Erasure Coding: consolidate streamer coordination logic and handle failure > when writing striped blocks > -- > > Key: HDFS-7672 > URL: https://issues.apache.org/jira/browse/HDFS-7672 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Fix For: HDFS-7285 > > Attachments: h7672_20150504.patch, h7672_20150504b.patch, > h7672_20150504c.patch, h7672_20150505.patch, h7672_20150505b.patch > > > In *stripping* case, for (6, 3)-Reed-Solomon, a client writes to 6 data > blocks and 3 parity blocks concurrently. We need to handle datanode or > network failures when writing a EC BlockGroup. > We also refactor the existing code in DFSStripedOutputStream and > StripedDataStreamer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8937) Erasure coding: do not throw exception when setting replication factor on EC files
[ https://issues.apache.org/jira/browse/HDFS-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937817#comment-14937817 ] Hudson commented on HDFS-8937: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8937. Erasure coding: do not throw exception when setting (jing9: rev ddf4e785475affead2f7c070b9f151de0fcb9024) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingZones.java > Erasure coding: do not throw exception when setting replication factor on EC > files > -- > > Key: HDFS-8937 > URL: https://issues.apache.org/jira/browse/HDFS-8937 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: GAO Rui >Assignee: GAO Rui > Fix For: HDFS-7285 > > Attachments: HDFS-8937-HDFS-7285.00.patch, > HDFS-8937-HDFS-7285.01.patch > > > Setting replication to an EC file caused exception. We should simply ignore > the request, just like what we're currently doing for a setReplication > request against a directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8323) Bump GenerationStamp for write faliure in DFSStripedOutputStream
[ https://issues.apache.org/jira/browse/HDFS-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937882#comment-14937882 ] Hudson commented on HDFS-8323: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8323. Bump GenerationStamp for write faliure in (zhezhang: rev 45db1a0b727965ceb27411ea5567b3a76c63b181) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedStripedBlock.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt Addendum fix for HDFS-8323 (adding back a mistakenly deleted line). (zhezhang: rev 42a1d0dbde2582867669268a3f7ad2476dbf629e) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java > Bump GenerationStamp for write faliure in DFSStripedOutputStream > > > Key: HDFS-8323 > URL: https://issues.apache.org/jira/browse/HDFS-8323 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Fix For: HDFS-7285 > > Attachments: h8323_20150511.patch, h8323_20150511b.patch, > h8323_20150512.patch, h8323_20150518.patch, h8323_20150520.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7348) Erasure Coding: DataNode reconstruct striped blocks
[ https://issues.apache.org/jira/browse/HDFS-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937779#comment-14937779 ] Hudson commented on HDFS-7348: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-7348. Erasure Coding: DataNode reconstruct striped blocks. (zhezhang: rev 6616de24cb14f1c2d0d6568fd4382062618834bd) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocalLegacy.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java * hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReader.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java > Erasure Coding: DataNode reconstruct striped blocks > --- > > Key: HDFS-7348 > URL: https://issues.apache.org/jira/browse/HDFS-7348 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Kai Zheng >Assignee: Yi Liu > Fix For: HDFS-7285 > > Attachments: ECWorker.java, HDFS-7348.001.patch, HDFS-7348.002.patch, > HDFS-7348.003.patch > > > This JIRA is to recover one or more missed striped block in the striped block > group. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8920) Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance
[
https://issues.apache.org/jira/browse/HDFS-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937838#comment-14937838
]
Hudson commented on HDFS-8920:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8920. Erasure Coding: when recovering lost blocks, logs can be too
(zhezhang: rev 7bff8ca1c872ea534a96cbbc5f70134574e289ce)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
> Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt
> performance
> -
>
> Key: HDFS-8920
> URL: https://issues.apache.org/jira/browse/HDFS-8920
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
> Fix For: HDFS-7285
>
> Attachments: HDFS-8920-HDFS-7285.1.patch, HDFS-8920-HDFS-7285.2.patch
>
>
> When we test reading data with datanodes killed,
> {{DFSInputStream::getBestNodeDNAddrPair}} becomes a hot spot method and
> effectively blocks the client JVM. This log seems too verbose:
> {code}
> if (chosenNode == null) {
> DFSClient.LOG.warn("No live nodes contain block " + block.getBlock() +
> " after checking nodes = " + Arrays.toString(nodes) +
> ", ignoredNodes = " + ignoredNodes);
> return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8444) Erasure Coding: fix cannot rename a zone dir
[
https://issues.apache.org/jira/browse/HDFS-8444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937802#comment-14937802
]
Hudson commented on HDFS-8444:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8444. Erasure Coding: fix cannot rename a zone dir (Contributed by
(vinayakumarb: rev 2d847e7d620f97ebe8e9297361e2541d7d0ecd50)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingZoneManager.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingZones.java
> Erasure Coding: fix cannot rename a zone dir
>
>
> Key: HDFS-8444
> URL: https://issues.apache.org/jira/browse/HDFS-8444
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: HDFS-7285
>
> Attachments: HDFS-8444-HDFS-7285.001.patch,
> HDFS-8444-HDFS-7285.002.patch
>
>
> We create a EC zone {{/my_ec_zone}}.
> We want to rename it to {{/myZone}}.
> But it failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8027) Erasure Coding: Update CHANGES-HDFS-7285.txt with branch commits
[ https://issues.apache.org/jira/browse/HDFS-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937866#comment-14937866 ] Hudson commented on HDFS-8027: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8027. Erasure Coding: Update CHANGES-HDFS-7285.txt with branch (zhezhang: rev af8eaacdb40afba327b776e2b4f374be624289fc) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt HDFS-8027. Erasure Coding: Update CHANGES-HDFS-7285.txt with branch (zhezhang: rev 2c277802c1635837ec926862523be02e05e5649b) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > Erasure Coding: Update CHANGES-HDFS-7285.txt with branch commits > > > Key: HDFS-8027 > URL: https://issues.apache.org/jira/browse/HDFS-8027 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Fix For: HDFS-7285 > > Attachments: HDFS-8027-01.patch > > > Latest branch commits are not tracked in CHANGES-HDFS-7285.txt. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8167) BlockManager.addBlockCollectionWithCheck should check if the block is a striped block
[ https://issues.apache.org/jira/browse/HDFS-8167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937904#comment-14937904 ] Hudson commented on HDFS-8167: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8167. BlockManager.addBlockCollectionWithCheck should check if the (zhezhang: rev 64f8f0a14584cdd1fbbf6da349d1e4576c419d18) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java > BlockManager.addBlockCollectionWithCheck should check if the block is a > striped block > - > > Key: HDFS-8167 > URL: https://issues.apache.org/jira/browse/HDFS-8167 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Tsz Wo Nicholas Sze >Assignee: Hui Zheng > Attachments: HDFS-8167_001.patch, HDFS-8167_002.patch > > > This JIRA is to address [~zhz]'s > [comment|https://issues.apache.org/jira/browse/HDFS-7994?focusedCommentId=14498894=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14498894]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7969) Erasure coding: NameNode support for lease recovery of striped block groups
[ https://issues.apache.org/jira/browse/HDFS-7969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937760#comment-14937760 ] Hudson commented on HDFS-7969: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-7969. Erasure coding: NameNode support for lease recovery of (zhezhang: rev 146ce7a9784e52432b76164009336a4b2cf2860e) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoUnderConstruction.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStripedUnderConstruction.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguousUnderConstruction.java > Erasure coding: NameNode support for lease recovery of striped block groups > --- > > Key: HDFS-7969 > URL: https://issues.apache.org/jira/browse/HDFS-7969 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: HDFS-7285 > > Attachments: HDFS-7969-000.patch, HDFS-7969-001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7768) Separate Platform specific funtions
[ https://issues.apache.org/jira/browse/HDFS-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937763#comment-14937763 ] Hudson commented on HDFS-7768: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-7768. Change fsck to support EC files. Contributed by Takanobu (zhezhang: rev 0ed92e5b13f6bbc0ea7475bc67488359413a980e) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > Separate Platform specific funtions > --- > > Key: HDFS-7768 > URL: https://issues.apache.org/jira/browse/HDFS-7768 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Thanh Do >Assignee: Thanh Do > > Current code has several platform-specific parts (e.g., get environment > variables, get local addresses, print stack). We should separate these parts > into platform folders. > This issue will do just that. Posix systems will be able to compile > successfully. Windows will fail to compile due to unimplemented parts. The > implementation for the Windows parts will be handle at HDFS-7188 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8289) Erasure Coding: add ECSchema to HdfsFileStatus
[
https://issues.apache.org/jira/browse/HDFS-8289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937761#comment-14937761
]
Hudson commented on HDFS-8289:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8289. Erasure Coding: add ECSchema to HdfsFileStatus. Contributed
(zhezhang: rev 9da927540f0ea6698388a4e79ef32c4dc51495ea)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestJsonUtil.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/SnapshottableDirectoryStatus.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsLocatedFileStatus.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileStatusWithECschema.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/erasurecoding.proto
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsFileStatus.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
> Erasure Coding: add ECSchema to HdfsFileStatus
> --
>
> Key: HDFS-8289
> URL: https://issues.apache.org/jira/browse/HDFS-8289
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Yong Zhang
> Fix For: HDFS-7285
>
> Attachments: HDFS-8289.000.patch, HDFS-8289.001.patch,
> HDFS-8289.002.patch, HDFS-8289.003.patch, HDFS-8289.004.patch
>
>
> {code}
> // ECInfo is restored from NN just before writing striped files.
> ecInfo = dfsClient.getErasureCodingInfo(src);
> {code}
> The rpc call above can be avoided by adding ECSchema to HdfsFileStatus.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9113) ErasureCodingWorker#processErasureCodingTasks should not fail to process remaining tasks due to one invalid ECTask
[ https://issues.apache.org/jira/browse/HDFS-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937765#comment-14937765 ] Hudson commented on HDFS-9113: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-9113. ErasureCodingWorker#processErasureCodingTasks should not fail (umamahesh: rev b762199adbd10173c588df67bd227393c5bbcce9) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java > ErasureCodingWorker#processErasureCodingTasks should not fail to process > remaining tasks due to one invalid ECTask > -- > > Key: HDFS-9113 > URL: https://issues.apache.org/jira/browse/HDFS-9113 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Affects Versions: HDFS-7285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Minor > Fix For: HDFS-7285 > > Attachments: HDFS-9113-HDFS-7285-00.patch > > > Currently processErasureCodingTasks method submits ecTasks to thread pool > service for processing the tasks. While submitting we initialize > ReconstructAndTransferBlock with each ecTask and submit it. There are chances > ReconstructAndTransferBlock initialization can fail due to wrong values as we > had preconditions for parameter validations in Ctor. Anyway, whatever may be > the case, processErasureCodingTasks should not fail and throw exceptions out > as it could prevent processing other tasks in the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7339) Representing striped block groups in NameNode with hierarchical naming protocol
[
https://issues.apache.org/jira/browse/HDFS-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937764#comment-14937764
]
Hudson commented on HDFS-7339:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-7339. Allocating and persisting block groups in NameNode. (zhezhang: rev
bc2833b1c91e107d090619d755c584f6eae82327)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockIdGenerator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockgroup.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockGroupIdGenerator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
> Representing striped block groups in NameNode with hierarchical naming
> protocol
> ---
>
> Key: HDFS-7339
> URL: https://issues.apache.org/jira/browse/HDFS-7339
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-7339-001.patch, HDFS-7339-002.patch,
> HDFS-7339-003.patch, HDFS-7339-004.patch, HDFS-7339-005.patch,
> HDFS-7339-006.patch, HDFS-7339-007.patch, HDFS-7339-008.patch,
> Meta-striping.jpg, NN-stripping.jpg
>
>
> All erasure codec operations center around the concept of _block group_; they
> are formed in initial encoding and looked up in recoveries and conversions. A
> lightweight class {{BlockGroup}} is created to record the original and parity
> blocks in a coding group, as well as a pointer to the codec schema (pluggable
> codec schemas will be supported in HDFS-7337). With the striping layout, the
> HDFS client needs to operate on all blocks in a {{BlockGroup}} concurrently.
> Therefore we propose to extend a file’s inode to switch between _contiguous_
> and _striping_ modes, with the current mode recorded in a binary flag. An
> array of BlockGroups (or BlockGroup IDs) is added, which remains empty for
> “traditional” HDFS files with contiguous block layout.
> The NameNode creates and maintains {{BlockGroup}} instances through the new
> {{ECManager}} component; the attached figure has an illustration of the
> architecture. As a simple example, when a {_Striping+EC_} file is created and
> written to, it will serve requests from the client to allocate new
> {{BlockGroups}} and store them under the {{INodeFile}}. In the current phase,
> {{BlockGroups}} are allocated both in initial online encoding and in the
> conversion from replication to EC. {{ECManager}} also facilitates the lookup
> of {{BlockGroup}} information for block recovery work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8282) Erasure coding: move striped reading logic to StripedBlockUtil
[ https://issues.apache.org/jira/browse/HDFS-8282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937759#comment-14937759 ] Hudson commented on HDFS-8282: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8282. Erasure coding: move striped reading logic to (zhezhang: rev f0628280c351640a807b05eaf786d051b5a7bd66) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPlanReadPortions.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java > Erasure coding: move striped reading logic to StripedBlockUtil > -- > > Key: HDFS-8282 > URL: https://issues.apache.org/jira/browse/HDFS-8282 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: HDFS-7285 > > Attachments: HDFS-8282-HDFS-7285.00.patch, > HDFS-8282-HDFS-7285.01.patch, HDFS-8282-HDFS-7285.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7617) Add unit tests for editlog transactions for EC
[ https://issues.apache.org/jira/browse/HDFS-7617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937749#comment-14937749 ] Hudson commented on HDFS-7617: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-7617. Add unit tests for editlog transactions for EC. Contributed (zhezhang: rev 9edc73d1231e7eb773286eaa9a6302931faed9ca) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java > Add unit tests for editlog transactions for EC > -- > > Key: HDFS-7617 > URL: https://issues.apache.org/jira/browse/HDFS-7617 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Tsz Wo Nicholas Sze >Assignee: Hui Zheng > Fix For: HDFS-7285 > > Attachments: HDFS-7617.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8567) Erasure Coding: SafeMode handles file smaller than a full stripe
[ https://issues.apache.org/jira/browse/HDFS-8567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937752#comment-14937752 ] Hudson commented on HDFS-8567: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8567. Erasure Coding: SafeMode handles file smaller than a full (jing9: rev d920cdab91781e547932234d52807ba5a59eb120) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeModeWithStripedFile.java > Erasure Coding: SafeMode handles file smaller than a full stripe > > > Key: HDFS-8567 > URL: https://issues.apache.org/jira/browse/HDFS-8567 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Walter Su >Assignee: Walter Su > Fix For: HDFS-7285 > > Attachments: HDFS-8567-HDFS-7285.00.patch, > HDFS-8567-HDFS-7285.01.patch, HDFS-8567-HDFS-7285.02.patch, HDFS-8567.00.patch > > > Upload 3 small files, and restart NN. It can't leave safemode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7912) Erasure Coding: track BlockInfo instead of Block in UnderReplicatedBlocks and PendingReplicationBlocks
[
https://issues.apache.org/jira/browse/HDFS-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937875#comment-14937875
]
Hudson commented on HDFS-7912:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-7912. Erasure Coding: track BlockInfo instead of Block in (zhezhang: rev
a38a37c63417a3b19dcdf98251af196c9d7b8c31)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestRBWBlockInvalidation.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SafeMode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingReplication.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingReplicationBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNodeCount.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestReadOnlySharedStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestOverReplicatedBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestProcessCorruptBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
Addendum fix for HDFS-7912. (zhezhang: rev
c9e0268216584f1df1a7c6cd25d2cfb2bc6d1d3c)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingReplication.java
> Erasure Coding: track BlockInfo instead of Block in UnderReplicatedBlocks and
> PendingReplicationBlocks
> --
>
> Key: HDFS-7912
> URL: https://issues.apache.org/jira/browse/HDFS-7912
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Fix For: HDFS-7285
>
> Attachments: HDFS-7912.000.patch
>
>
> Now with striped blocks and the design that uses a single BlockInfoStriped
> object to track all the corresponding blocks, we need to clearly distinguish
> the type Block and BlockInfo in BlockManager. Specifically, data structures
> like {{UnderReplicatedBlocks}} and {{PendingReplicationBlocks}} should track
> BlockInfo instead of Block in order to support striped block recovery.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8242) Erasure Coding: XML based end-to-end test for ECCli commands
[
https://issues.apache.org/jira/browse/HDFS-8242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937844#comment-14937844
]
Hudson commented on HDFS-8242:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8242. Erasure Coding: XML based end-to-end test for ECCli commands
(zhezhang: rev 4392325546a49d29720fe705381447a0fdd5826e)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/cli/util/CLICommandErasureCodingCli.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/cli/CLITestCmdErasureCoding.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/cli/util/ErasureCodingCliCmdExecutor.java
* hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testErasureCodingConf.xml
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/cli/TestErasureCodingCLI.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/erasurecode/ECCommand.java
> Erasure Coding: XML based end-to-end test for ECCli commands
>
>
> Key: HDFS-8242
> URL: https://issues.apache.org/jira/browse/HDFS-8242
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
> Fix For: HDFS-7285
>
> Attachments: HDFS-8242-001.patch, HDFS-8242-002.patch,
> HDFS-8242-003.patch, HDFS-8242-HDFS-7285.04.patch,
> HDFS-8242-HDFS-7285.05.patch, HDFS-8242-HDFS-7285.05.patch,
> HDFS-8242-HDFS-7285.05.patch
>
>
> This JIRA to add test cases with CLI test f/w for the commands present in
> {{ECCli}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8853) Erasure Coding: Provide ECSchema validation when setting EC policy
[
https://issues.apache.org/jira/browse/HDFS-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937867#comment-14937867
]
Hudson commented on HDFS-8853:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8853. Erasure Coding: Provide ECSchema validation when setting EC
(zhezhang: rev 96d6b516b2468fce346490e0b95931d1759b3d33)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirErasureCodingOp.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingPolicies.java
> Erasure Coding: Provide ECSchema validation when setting EC policy
> --
>
> Key: HDFS-8853
> URL: https://issues.apache.org/jira/browse/HDFS-8853
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: J.Andreina
> Fix For: HDFS-7285
>
> Attachments: HDFS-8853-HDFS-7285-01.patch,
> HDFS-8853-HDFS-7285-05.patch, HDFS-8853-HDFS-7285-06.patch,
> HDFS-8853-HDFS-7285-merge-02.patch, HDFS-8853-HDFS-7285-merge-03.patch,
> HDFS-8853-HDFS-7285-merge-04.patch
>
>
> Presently the {{DFS#createErasureCodingZone(path, ecSchema, cellSize)}}
> doesn't have any validation that the given {{ecSchema}} is available in
> {{ErasureCodingSchemaManager#activeSchemas}} list. Now, if it doesn't exists
> then will create the ECZone with {{null}} schema. IMHO we could improve this
> by doing necessary basic sanity checks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8145) Fix the editlog corruption exposed by failed TestAddStripedBlocks
[
https://issues.apache.org/jira/browse/HDFS-8145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937850#comment-14937850
]
Hudson commented on HDFS-8145:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8145. Fix the editlog corruption exposed by failed (zhezhang: rev
f6e1160ef1e946a5f6c9503b06832e6b84c36edb)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingZoneManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoStriped.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
> Fix the editlog corruption exposed by failed TestAddStripedBlocks
> -
>
> Key: HDFS-8145
> URL: https://issues.apache.org/jira/browse/HDFS-8145
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Fix For: HDFS-7285
>
> Attachments: HDFS-8145.000.patch, HDFS-8145.001.patch,
> HDFS-8145.002.patch
>
>
> {{TestAddStripedBlocks}} failed with some editlog corruption. Did some
> debugging, I can see at least two issues:
> # DFSStripedOutputStream tries to send out an empty packet to close the block
> even if writing 0 bytes
> # Because of the above, NN tries to close the file. This exposes another bug
> in {{BlockInfoStriped}}, which writes its data/parity block numbers into the
> close editlog but do not read them while loading.
> This jira aims to only fix #2 and HDFS-8166 is used for fixing #1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8090) Erasure Coding: Add RPC to client-namenode to list all ECSchemas loaded in Namenode.
[ https://issues.apache.org/jira/browse/HDFS-8090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937845#comment-14937845 ] Hudson commented on HDFS-8090: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8090. Erasure Coding: Add RPC to client-namenode to list all (zhezhang: rev f53e402635a990458113ba0595655751d83c02e1) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestECSchemas.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECSchema.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto > Erasure Coding: Add RPC to client-namenode to list all ECSchemas loaded in > Namenode. > > > Key: HDFS-8090 > URL: https://issues.apache.org/jira/browse/HDFS-8090 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Fix For: HDFS-7285 > > Attachments: HDFS-8090-01.patch, HDFS-8090-02.patch, > HDFS-8090-03.patch > > > ECSchemas will be configured and loaded only at the Namenode to avoid > conflicts. > Client has to specify one of these schemas during creation of ecZones. > So, add an RPC to ClientProtocol to get all ECSchemas loaded at namenode, so > that client can choose only any one of these. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8364) Erasure coding: fix some minor bugs in EC CLI
[
https://issues.apache.org/jira/browse/HDFS-8364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937899#comment-14937899
]
Hudson commented on HDFS-8364:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8364. Erasure coding: fix some minor bugs in EC CLI (Contributed by
(zhezhang: rev bba15e06d29d79ab1c379f258e5efb13a1ea39ae)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/erasurecode/ECCommand.java
* hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testErasureCodingConf.xml
* hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
> Erasure coding: fix some minor bugs in EC CLI
> -
>
> Key: HDFS-8364
> URL: https://issues.apache.org/jira/browse/HDFS-8364
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Fix For: HDFS-7285
>
> Attachments: HDFS-8364-HDFS-7285.001.patch,
> HDFS-8364-HDFS-7285.002.patch
>
>
> issue 1: it's not user-friendly. We should tell user that `hdfs
> erasurecode` is the entrance to EC commands(HDFS-7349)
> {noformat}
> # bin/hdfs
> Usage: hdfs [--config confdir] [--daemon (start|stop|status)]
>[--loglevel loglevel] COMMAND
>where COMMAND is one of:
> balancer run a cluster balancing utility
> cacheadmin configure the HDFS cache
> crypto configure HDFS encryption zones
> ...
> zkfc run the ZK Failover Controller daemon
> {noformat}
> issue 2: it's not user-friendly
> {noformat}
> # bin/hdfs erasurecode -getZoneInfo /
> -getZoneInfo: Fatal internal error
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.tools.erasurecode.ECCommand$GetECZoneInfoCommand.processPath(ECCommand.java:168)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:321)
> at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:293)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:275)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:259)
> at
> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:205)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:166)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:291)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.hdfs.tools.erasurecode.ECCli.main(ECCli.java:45)
> {noformat}
> possible issue 3: We create zone twice at /
> {noformat}
> # bin/hdfs erasurecode -createZone /
> EC Zone created successfully at hdfs://ds-34:9000/
> # bin/hdfs erasurecode -createZone /
> createZone: Unable to create EC zone for the path hdfs://ds-34:9000/
> {noformat}
> It doesn't print detail error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8428) Erasure Coding: Fix the NullPointerException when deleting file
[
https://issues.apache.org/jira/browse/HDFS-8428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937853#comment-14937853
]
Hudson commented on HDFS-8428:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8428. Erasure Coding: Fix the NullPointerException when deleting
(zhezhang: rev 47ef869fa790dd096b576697c4245d2f3a3193fa)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
> Erasure Coding: Fix the NullPointerException when deleting file
> ---
>
> Key: HDFS-8428
> URL: https://issues.apache.org/jira/browse/HDFS-8428
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: HDFS-7285
>
> Attachments: HDFS-8428-HDFS-7285.001.patch
>
>
> In HDFS, when removing some file, NN will also remove all its blocks from
> {{BlocksMap}}, and send {{DNA_INVALIDATE}} (invalidate blocks) commands to
> datanodes. After datanodes successfully delete the block replicas, will
> report {{DELETED_BLOCK}} to NameNode.
> snippet code logic in {{BlockManager#processIncrementalBlockReport}} is as
> following
> {code}
> case DELETED_BLOCK:
> removeStoredBlock(storageInfo, getStoredBlock(rdbi.getBlock()), node);
> ...
> {code}
> {code}
> private void removeStoredBlock(DatanodeStorageInfo storageInfo, Block block,
> DatanodeDescriptor node) {
> if (shouldPostponeBlocksFromFuture &&
> namesystem.isGenStampInFuture(block)) {
> queueReportedBlock(storageInfo, block, null,
> QUEUE_REASON_FUTURE_GENSTAMP);
> return;
> }
> removeStoredBlock(getStoredBlock(block), node);
> }
> {code}
> In EC branch, we add {{getStoredBlock}}. There is {{NullPointerException}}
> when handling {{DELETED_BLOCK}} of incrementalBlockReport from DataNode after
> delete a file, since the block is already removed, we need to check.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937887#comment-14937887 ] Hudson commented on HDFS-8033: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8033. Erasure coding: stateful (non-positional) read from files in (zhezhang: rev 89d33785780f98a58e1e81eca2c27165840475df) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java > Erasure coding: stateful (non-positional) read from files in striped layout > --- > > Key: HDFS-8033 > URL: https://issues.apache.org/jira/browse/HDFS-8033 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: HDFS-7285 > > Attachments: ByteBufferStrategy_support_len.patch, > HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch, > HDFS-8033.003.patch, HDFS-8033.004.patch, HDFS-8033.005.patch, > HDFS-8033.006.patch, HDFS-8033.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7853) Erasure coding: extend LocatedBlocks to support reading from striped files
[
https://issues.apache.org/jira/browse/HDFS-7853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937859#comment-14937859
]
Hudson commented on HDFS-7853:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-7853. Erasure coding: extend LocatedBlocks to support reading from
(zhezhang: rev f05c21285ef23b6a973d69f045b1cb46c5abc039)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBrVariations.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedStripedBlock.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStripedUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
> Erasure coding: extend LocatedBlocks to support reading from striped files
> --
>
> Key: HDFS-7853
> URL: https://issues.apache.org/jira/browse/HDFS-7853
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Jing Zhao
> Fix For: HDFS-7285
>
> Attachments: HDFS-7853.000.patch, HDFS-7853.001.patch,
> HDFS-7853.002.patch
>
>
> We should extend {{LocatedBlocks}} class so {{getBlockLocations}} can work
> with striping layout (possibly an extra list specifying the index of each
> location in the group)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8909) Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use BlockUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937884#comment-14937884
]
Hudson commented on HDFS-8909:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8909. Erasure coding: update BlockInfoContiguousUC and (waltersu4549: rev
164cbe643988f878f0f4100a4de51783e5b6738e)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotTestHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStripedUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiffList.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirTruncateOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestBlockUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStripedINodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockUnderConstructionFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguousUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoUnderConstruction.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
> Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use
> BlockUnderConstructionFeature
>
>
> Key: HDFS-8909
> URL: https://issues.apache.org/jira/browse/HDFS-8909
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Jing Zhao
> Attachments: HDFS-8909-HDFS-7285.003.patch,
> HDFS-8909-HDFS-7285.004.patch, HDFS-8909.000.patch, HDFS-8909.001.patch,
> HDFS-8909.002.patch, HDFS-8909.003.patch
>
>
> HDFS-8801 converts {{BlockInfoUC}} as a feature. We should consolidate
> {{BlockInfoContiguousUC}} and {{BlockInfoStripedUC}} logics to use this
> feature.
--
[jira] [Commented] (HDFS-8230) Erasure Coding: Ignore DatanodeProtocol#DNA_ERASURE_CODING_RECOVERY commands from standbynode if any
[ https://issues.apache.org/jira/browse/HDFS-8230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937869#comment-14937869 ] Hudson commented on HDFS-8230: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8230. Erasure Coding: Ignore (zhezhang: rev f9eb95c31da9199b393d7994cf9f21250abe41b9) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > Erasure Coding: Ignore DatanodeProtocol#DNA_ERASURE_CODING_RECOVERY commands > from standbynode if any > > > Key: HDFS-8230 > URL: https://issues.apache.org/jira/browse/HDFS-8230 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Uma Maheswara Rao G >Assignee: Vinayakumar B >Priority: Minor > Fix For: HDFS-7285 > > Attachments: HDFS-8230-01.patch > > > Let's ignore DNA_ERASURE_CODING_RECOVERY command at DN if its coming from > standby namenode. Ideally we should not get this commands from standby node. > Since we handle to ignore the commands from standby node, we can add this > also to be ignored. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8328) Follow-on to update decode for DataNode striped blocks reconstruction
[ https://issues.apache.org/jira/browse/HDFS-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937878#comment-14937878 ] Hudson commented on HDFS-8328: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8328. Follow-on to update decode for DataNode striped blocks (yliu: rev a31eada33a598ebf9f78e48a3ab1ed031b9bbd27) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > Follow-on to update decode for DataNode striped blocks reconstruction > - > > Key: HDFS-8328 > URL: https://issues.apache.org/jira/browse/HDFS-8328 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Yi Liu >Assignee: Yi Liu > Attachments: HDFS-8328-HDFS-7285.001.patch, > HDFS-8328-HDFS-7285.002.patch, HDFS-8328-HDFS-7285.003.patch > > > Current the decode for DataNode striped blocks reconstruction is a > workaround, we need to update it after the decode fix in HADOOP-11847. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)
[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937889#comment-14937889 ] Hudson commented on HDFS-9040: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-9040. Erasure coding: coordinate data streamers in (jing9: rev 6419900ac24a5493827abf9b5d90373bc1043e0b) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteStripedFileWithFailure.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/StripedFileTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddOverReplicatedStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockUnderConstructionFeature.java > Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests > to Coordinator) > --- > > Key: HDFS-9040 > URL: https://issues.apache.org/jira/browse/HDFS-9040 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Walter Su >Assignee: Jing Zhao > Fix For: HDFS-7285 > > Attachments: HDFS-9040-HDFS-7285.002.patch, > HDFS-9040-HDFS-7285.003.patch, HDFS-9040-HDFS-7285.004.patch, > HDFS-9040-HDFS-7285.005.patch, HDFS-9040-HDFS-7285.005.patch, > HDFS-9040-HDFS-7285.006.patch, HDFS-9040-HDFS-7285.007.patch, > HDFS-9040.00.patch, HDFS-9040.001.wip.patch, HDFS-9040.02.bgstreamer.patch > > > The general idea is to simplify error handling logic. > -Proposal 1:- > -A BlockGroupDataStreamer to communicate with NN to allocate/update block, > and StripedDataStreamer s only have to stream blocks to DNs.- > Proposal 2: > See below the > [comment|https://issues.apache.org/jira/browse/HDFS-9040?focusedCommentId=14741388=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14741388] > from [~jingzhao]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8336) Expose some administrative erasure coding operations to HdfsAdmin
[ https://issues.apache.org/jira/browse/HDFS-8336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937852#comment-14937852 ] Hudson commented on HDFS-8336: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8336. Expose some administrative erasure coding operations to (vinayakumarb: rev 9b54e66f3ea6a5a6945d18ba534cfb4f2419facb) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java > Expose some administrative erasure coding operations to HdfsAdmin > - > > Key: HDFS-8336 > URL: https://issues.apache.org/jira/browse/HDFS-8336 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Minor > Fix For: HDFS-7285 > > Attachments: HDFS-8336-001.patch, HDFS-8336-HDFS-7285-001.patch, > HDFS-8336-HDFS-7285-002.patch > > > We have HdfsAdmin.java for exposing administrative functions. So, it would be > good, if we could expose EC related administrative functions as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8734) Erasure Coding: fix one cell need two packets
[
https://issues.apache.org/jira/browse/HDFS-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937865#comment-14937865
]
Hudson commented on HDFS-8734:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8734. Erasure Coding: fix one cell need two packets. Contributed by
(jing9: rev 0a93712f3b9b36d746577dca5da0f7f09756fcca)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> Erasure Coding: fix one cell need two packets
> -
>
> Key: HDFS-8734
> URL: https://issues.apache.org/jira/browse/HDFS-8734
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: HDFS-7285
>
> Attachments: HDFS-8734-HDFS-7285.01.patch, HDFS-8734.01.patch
>
>
> The default WritePacketSize is 64k
> Currently default cellSize is 64k
> We hope one cell consumes one packet. In fact it's not.
> By default,
> chunkSize = 516( 512 data + 4 checksum)
> packetSize = 64k
> chunksPerPacket = 126 ( See DFSOutputStream#computePacketChunkSize for
> details)
> numBytes of data in one packet = 64512
> cellSize = 65536
> When first packet is full ( with 64512 data), there are still 65536 - 64512 =
> 1024 bytes left.
> {code}
> super.writeChunk(bytes, offset, len, checksum, ckoff, cklen);
> // cell is full and current packet has not been enqueued,
> if (cellFull && currentPacket != null) {
> enqueueCurrentPacketFull();
> }
> {code}
> When the last 1024 bytes of the cell was written, we meet {{cellFull}} and
> create another packet.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8137) Send the EC schema to DataNode via EC encoding/recovering command
[ https://issues.apache.org/jira/browse/HDFS-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937848#comment-14937848 ] Hudson commented on HDFS-8137: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8137. Send the EC schema to DataNode via EC encoding/recovering (zhezhang: rev 436c14855aef58a551d5cab91eba6e88775c6797) * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/erasurecoding.proto * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockECRecoveryCommand.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java > Send the EC schema to DataNode via EC encoding/recovering command > - > > Key: HDFS-8137 > URL: https://issues.apache.org/jira/browse/HDFS-8137 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Uma Maheswara Rao G > Fix For: HDFS-7285 > > Attachments: HDFS-8137-0.patch, HDFS-8137-1.patch, HDFS-8137-2.patch > > > Discussed with [~umamaheswararao] and [~vinayrpet], we should also send the > EC schema to DataNode as well contained in the EC encoding/recovering > command. The target DataNode will use it to guide the executing of the task. > Another way would be, DataNode would just request schema actively thru a > separate RPC call, and as an optimization consideration, DataNode may cache > schemas to avoid repeatedly asking for the same schema twice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8744) Erasure Coding: the number of chunks in packet is not updated when writing parity data
[
https://issues.apache.org/jira/browse/HDFS-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937890#comment-14937890
]
Hudson commented on HDFS-8744:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8744. Erasure Coding: the number of chunks in packet is not updated
(bo.b.li: rev f4098dfae44b6dd287a3054f2c04658773b4f466)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
> Erasure Coding: the number of chunks in packet is not updated when writing
> parity data
> --
>
> Key: HDFS-8744
> URL: https://issues.apache.org/jira/browse/HDFS-8744
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Li Bo
>Assignee: Li Bo
> Attachments: HDFS-8744-HDFS-7285-001.patch,
> HDFS-8744-HDFS-7285-002.patch
>
>
> The member {{numChunks}} in {{DFSPacket}} is always zero if this packet
> contains parity data. The calling of {{getNumChunks}} may cause potential
> errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7652) Process block reports for erasure coded blocks
[
https://issues.apache.org/jira/browse/HDFS-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937883#comment-14937883
]
Hudson commented on HDFS-7652:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-7652. Process block reports for erasure coded blocks. Contributed
(zhezhang: rev 0c1da5a0300f015a7e39f2b40a73fb06c65a78c8)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
> Process block reports for erasure coded blocks
> --
>
> Key: HDFS-7652
> URL: https://issues.apache.org/jira/browse/HDFS-7652
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: HDFS-7285
>
> Attachments: HDFS-7652.001.patch, HDFS-7652.002.patch,
> HDFS-7652.003.patch, HDFS-7652.004.patch, HDFS-7652.005.patch,
> HDFS-7652.006.patch
>
>
> HDFS-7339 adds support in NameNode for persisting block groups. For memory
> efficiency, erasure coded blocks under the striping layout are not stored in
> {{BlockManager#blocksMap}}. Instead, entire block groups are stored in
> {{BlockGroupManager#blockGroups}}. When a block report arrives from the
> DataNode, it should be processed under the block group that it belongs to.
> The following naming protocol is used to calculate the group of a given block:
> {code}
> * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
> * Contiguous: {reserved block IDs | flag | block ID}
> * Striped: {reserved block IDs | flag | block group ID | index in group}
> *
> * Following n bits of reserved block IDs, The (n+1)th bit in an ID
> * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
> * bits (n+2) to (64-m) represent the ID of its block group, while the last m
> * bits represent its index of the group. The value m is determined by the
> * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8324) Add trace info to DFSClient#getErasureCodingZoneInfo(..)
[ https://issues.apache.org/jira/browse/HDFS-8324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937881#comment-14937881 ] Hudson commented on HDFS-8324: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8324. Add trace info to DFSClient#getErasureCodingZoneInfo(..). (zhezhang: rev e849be2d31b6b5cda14f3c19b5469ca4d2742d7d) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java > Add trace info to DFSClient#getErasureCodingZoneInfo(..) > > > Key: HDFS-8324 > URL: https://issues.apache.org/jira/browse/HDFS-8324 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Fix For: HDFS-7285 > > Attachments: HDFS-8324-HDFS-7285.01.patch > > > Add trace spans to DFSClient#getErasureCodingZoneInfo(..) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8375) Add cellSize as an XAttr to ECZone
[ https://issues.apache.org/jira/browse/HDFS-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937989#comment-14937989 ] Hudson commented on HDFS-8375: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-8375. Add cellSize as an XAttr to ECZone. Contributed by (zhezhang: rev 91c81fdc24709b3caf1f6281c8879ffee08db956) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsFileStatus.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestJsonUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/erasurecoding.proto * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestStripedBlockUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingZones.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/erasurecode/ECCommand.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/SnapshottableDirectoryStatus.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileStatusWithECschema.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewerWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteReadStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsLocatedFileStatus.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ErasureCodingZoneInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockECRecoveryCommand.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java *
[jira] [Commented] (HDFS-7768) Separate Platform specific funtions
[ https://issues.apache.org/jira/browse/HDFS-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937983#comment-14937983 ] Hudson commented on HDFS-7768: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-7768. Change fsck to support EC files. Contributed by Takanobu (zhezhang: rev 0ed92e5b13f6bbc0ea7475bc67488359413a980e) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java > Separate Platform specific funtions > --- > > Key: HDFS-7768 > URL: https://issues.apache.org/jira/browse/HDFS-7768 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Thanh Do >Assignee: Thanh Do > > Current code has several platform-specific parts (e.g., get environment > variables, get local addresses, print stack). We should separate these parts > into platform folders. > This issue will do just that. Posix systems will be able to compile > successfully. Windows will fail to compile due to unimplemented parts. The > implementation for the Windows parts will be handle at HDFS-7188 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8281) Erasure Coding: implement parallel stateful reading for striped layout
[
https://issues.apache.org/jira/browse/HDFS-8281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937971#comment-14937971
]
Hudson commented on HDFS-8281:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-8281. Erasure Coding: implement parallel stateful reading for (zhezhang:
rev 6dea01f1ee5ea3bf6e146e1b68616c2f43ba4792)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPlanReadPortions.java
> Erasure Coding: implement parallel stateful reading for striped layout
> --
>
> Key: HDFS-8281
> URL: https://issues.apache.org/jira/browse/HDFS-8281
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Fix For: HDFS-7285
>
> Attachments: HDFS-8281-HDFS-7285.001.patch,
> HDFS-8281-HDFS-7285.001.patch, HDFS-8281-HDFS-7285.002.patch,
> HDFS-8281.000.patch
>
>
> This jira aims to support parallel reading for stateful read in
> {{DFSStripedInputStream}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7348) Erasure Coding: DataNode reconstruct striped blocks
[ https://issues.apache.org/jira/browse/HDFS-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937998#comment-14937998 ] Hudson commented on HDFS-7348: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-7348. Erasure Coding: DataNode reconstruct striped blocks. (zhezhang: rev 6616de24cb14f1c2d0d6568fd4382062618834bd) * hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReader.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocalLegacy.java > Erasure Coding: DataNode reconstruct striped blocks > --- > > Key: HDFS-7348 > URL: https://issues.apache.org/jira/browse/HDFS-7348 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Kai Zheng >Assignee: Yi Liu > Fix For: HDFS-7285 > > Attachments: ECWorker.java, HDFS-7348.001.patch, HDFS-7348.002.patch, > HDFS-7348.003.patch > > > This JIRA is to recover one or more missed striped block in the striped block > group. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8571) Fix TestErasureCodingCli test
[ https://issues.apache.org/jira/browse/HDFS-8571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937987#comment-14937987 ] Hudson commented on HDFS-8571: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-8571. Fix TestErasureCodingCli test. Contributed by Vinayakumar B. (waltersu4549: rev b7d6ea8e91417a23e8c07c3db977277b485a2fdd) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testErasureCodingConf.xml > Fix TestErasureCodingCli test > - > > Key: HDFS-8571 > URL: https://issues.apache.org/jira/browse/HDFS-8571 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Fix For: HDFS-7285 > > Attachments: HDFS-8571-HDFS-7285-01.patch > > > TestErasureCodingCli test is failing due to changes done in HDFS-8556. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9113) ErasureCodingWorker#processErasureCodingTasks should not fail to process remaining tasks due to one invalid ECTask
[ https://issues.apache.org/jira/browse/HDFS-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937985#comment-14937985 ] Hudson commented on HDFS-9113: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-9113. ErasureCodingWorker#processErasureCodingTasks should not fail (umamahesh: rev b762199adbd10173c588df67bd227393c5bbcce9) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java > ErasureCodingWorker#processErasureCodingTasks should not fail to process > remaining tasks due to one invalid ECTask > -- > > Key: HDFS-9113 > URL: https://issues.apache.org/jira/browse/HDFS-9113 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Affects Versions: HDFS-7285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Minor > Fix For: HDFS-7285 > > Attachments: HDFS-9113-HDFS-7285-00.patch > > > Currently processErasureCodingTasks method submits ecTasks to thread pool > service for processing the tasks. While submitting we initialize > ReconstructAndTransferBlock with each ecTask and submit it. There are chances > ReconstructAndTransferBlock initialization can fail due to wrong values as we > had preconditions for parameter validations in Ctor. Anyway, whatever may be > the case, processErasureCodingTasks should not fail and throw exceptions out > as it could prevent processing other tasks in the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8058) Erasure coding: use BlockInfo[] for both striped and contiguous blocks in INodeFile
[
https://issues.apache.org/jira/browse/HDFS-8058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937982#comment-14937982
]
Hudson commented on HDFS-8058:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-8058. Erasure coding: use BlockInfo[] for both striped and (zhezhang: rev
7e091de1366f4b57b5433bc19d738199dc05313d)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiff.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiffList.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileAttributes.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestTruncateQuotaUpdate.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/fsimage.proto
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/CreateEditsLog.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStripedINodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileWithStripedBlocksFeature.java
> Erasure coding: use BlockInfo[] for both striped and contiguous blocks in
> INodeFile
> ---
>
> Key: HDFS-8058
> URL: https://issues.apache.org/jira/browse/HDFS-8058
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Yi Liu
>Assignee: Zhe Zhang
> Fix For: HDFS-7285
>
> Attachments: HDFS-8058-HDFS-7285.003.patch,
> HDFS-8058-HDFS-7285.004.patch, HDFS-8058-HDFS-7285.005.patch,
> HDFS-8058-HDFS-7285.006.patch, HDFS-8058-HDFS-7285.007.patch,
> HDFS-8058-HDFS-7285.008.patch, HDFS-8058-HDFS-7285.009.patch,
> HDFS-8058-HDFS-7285.010.patch, HDFS-8058.001.patch, HDFS-8058.002.patch
>
>
> This JIRA is to use {{BlockInfo[] blocks}} for both striped and contiguous
> blocks in INodeFile.
> Currently {{FileWithStripedBlocksFeature}} keeps separate list for striped
> blocks, and the methods there duplicate with those in INodeFile, and current
> code need to judge {{isStriped}} then do different things. Also if file is
> striped, the {{blocks}} in INodeFile occupy a reference memory space.
> These are not necessary, and we can use the same {{blocks}} to make code more
> clear.
> I keep {{FileWithStripedBlocksFeature}} as empty for follow use: I will file
> a new JIRA to move {{dataBlockNum}} and {{parityBlockNum}} from
> *BlockInfoStriped* to INodeFile, since ideally they are the same for all
> striped blocks in a file, and store them in block will waste NN memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8289) Erasure Coding: add ECSchema to HdfsFileStatus
[
https://issues.apache.org/jira/browse/HDFS-8289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937981#comment-14937981
]
Hudson commented on HDFS-8289:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-8289. Erasure Coding: add ECSchema to HdfsFileStatus. Contributed
(zhezhang: rev 9da927540f0ea6698388a4e79ef32c4dc51495ea)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsLocatedFileStatus.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/erasurecoding.proto
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsFileStatus.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileStatusWithECschema.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/SnapshottableDirectoryStatus.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestJsonUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> Erasure Coding: add ECSchema to HdfsFileStatus
> --
>
> Key: HDFS-8289
> URL: https://issues.apache.org/jira/browse/HDFS-8289
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Yong Zhang
> Fix For: HDFS-7285
>
> Attachments: HDFS-8289.000.patch, HDFS-8289.001.patch,
> HDFS-8289.002.patch, HDFS-8289.003.patch, HDFS-8289.004.patch
>
>
> {code}
> // ECInfo is restored from NN just before writing striped files.
> ecInfo = dfsClient.getErasureCodingInfo(src);
> {code}
> The rpc call above can be avoided by adding ECSchema to HdfsFileStatus.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8899) Erasure Coding: use threadpool for EC recovery tasks on DataNode
[
https://issues.apache.org/jira/browse/HDFS-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937997#comment-14937997
]
Hudson commented on HDFS-8899:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-8899. Erasure Coding: use threadpool for EC recovery tasks on (zhezhang:
rev ced438a4bf50fe0ac9072c128e18249e6742956a)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
> Erasure Coding: use threadpool for EC recovery tasks on DataNode
>
>
> Key: HDFS-8899
> URL: https://issues.apache.org/jira/browse/HDFS-8899
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Rakesh R
>Assignee: Rakesh R
> Fix For: HDFS-7285
>
> Attachments: HDFS-8899-HDFS-7285-01.patch,
> HDFS-8899-HDFS-7285-02.patch, HDFS-8899-HDFS-7285-merge-00.patch
>
>
> The idea is to use threadpool for processing erasure coding recovery tasks at
> the datanode.
> {code}
> new Daemon(new ReconstructAndTransferBlock(recoveryInfo)).start();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8481) Erasure coding: remove workarounds in client side stripped blocks recovering
[ https://issues.apache.org/jira/browse/HDFS-8481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938000#comment-14938000 ] Hudson commented on HDFS-8481: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-8481. Erasure coding: remove workarounds in client side stripped (zhezhang: rev 014bd32c58ccd7b4ff66ff1180f7fe9ae25484ef) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteReadStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > Erasure coding: remove workarounds in client side stripped blocks recovering > > > Key: HDFS-8481 > URL: https://issues.apache.org/jira/browse/HDFS-8481 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: HDFS-7285 > > Attachments: HDFS-8481-HDFS-7285.00.patch, > HDFS-8481-HDFS-7285.01.patch, HDFS-8481-HDFS-7285.02.patch, > HDFS-8481-HDFS-7285.03.patch, HDFS-8481-HDFS-7285.04.patch > > > After HADOOP-11847 and related fixes, we should be able to properly calculate > decoded contents. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8804) Erasure Coding: use DirectBufferPool in DFSStripedInputStream for buffer allocation
[ https://issues.apache.org/jira/browse/HDFS-8804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938002#comment-14938002 ] Hudson commented on HDFS-8804: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-8804. Erasure Coding: use DirectBufferPool in DFSStripedInputStream (jing9: rev 9312b168e2f152ebfb8a7c7b63e74a819adfc5d2) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java > Erasure Coding: use DirectBufferPool in DFSStripedInputStream for buffer > allocation > --- > > Key: HDFS-8804 > URL: https://issues.apache.org/jira/browse/HDFS-8804 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: HDFS-7285 > > Attachments: HDFS-8804.000.patch, HDFS-8804.001.patch > > > Currently we directly allocate direct ByteBuffer in DFSStripedInputstream for > the stripe buffer and the buffers holding parity data. It's better to get > ByteBuffer from DirectBufferPool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8813) Erasure Coding: Client no need to decode missing parity blocks
[ https://issues.apache.org/jira/browse/HDFS-8813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937986#comment-14937986 ] Hudson commented on HDFS-8813: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-8813. Erasure Coding: Client no need to decode missing parity (jing9: rev c2c26e6ea7967f02c284918f68554334d4af8561) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java > Erasure Coding: Client no need to decode missing parity blocks > -- > > Key: HDFS-8813 > URL: https://issues.apache.org/jira/browse/HDFS-8813 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Walter Su >Assignee: Walter Su >Priority: Minor > Fix For: HDFS-7285 > > Attachments: HDFS-8813-HDFS-7285.01.patch > > > Assume 6+3 schema. > Assume data block #2 is missing, then InputStream tries to read parity block > #6. > Assume parity block #6 is missing too. And InputStream successfully reads > parity block #7. > Then begin to decode. > Currently InputStream will decode #2 and #6. But client(user) only need #2. > The parity block #6 will be disposed. > The improvement is we only decode #2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8468) 2 RPC calls for every file read in DFSClient#open(..) resulting in double Audit log entries
[ https://issues.apache.org/jira/browse/HDFS-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937978#comment-14937978 ] Hudson commented on HDFS-8468: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-8468. 2 RPC calls for every file read in DFSClient#open(..) (vinayakumarb: rev 0b7af27b9a369d6abdb1fb6c216f50692267f3f4) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto > 2 RPC calls for every file read in DFSClient#open(..) resulting in double > Audit log entries > --- > > Key: HDFS-8468 > URL: https://issues.apache.org/jira/browse/HDFS-8468 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Fix For: HDFS-7285 > > Attachments: HDFS-8468-HDFS-7285-02.patch, HDFS-8468-HDFS-7285.patch > > > In HDFS-7285 branch, > To determine whether file is striped/not and get the Schema for the file, 2 > RPCs done to Namenode. > This is resulting in double audit logs for every file read for both > striped/non-striped. > This will be a major impact in size of audit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8316) Erasure coding: refactor EC constants to be consistent with HDFS-8249
[ https://issues.apache.org/jira/browse/HDFS-8316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938022#comment-14938022 ] Hudson commented on HDFS-8316: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-8316. Erasure coding: refactor EC constants to be consistent with (zhezhang: rev ea6c66ed5704dc8a9448b4c1b56e9664086c70fc) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStripedINodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockGroupIdGenerator.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > Erasure coding: refactor EC constants to be consistent with HDFS-8249 > - > > Key: HDFS-8316 > URL: https://issues.apache.org/jira/browse/HDFS-8316 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: HDFS-7285 > > Attachments: HDFS-8316-HDFS-7285.0.patch > > > HDFS-8249 separates server-internal constants and client-visible ones. This > JIRA takes care of EC constants from that perspective. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8580) Erasure coding: Persist cellSize in BlockInfoStriped and StripedBlocksFeature
[
https://issues.apache.org/jira/browse/HDFS-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938010#comment-14938010
]
Hudson commented on HDFS-8580:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-8580. Erasure coding: Persist cellSize in BlockInfoStriped and (jing9: rev
c12a974ccf5f52f63e4f825d8b4d2385953cd119)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/fsimage.proto
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoStriped.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStripedINodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStripedUnderConstruction.java
> Erasure coding: Persist cellSize in BlockInfoStriped and StripedBlocksFeature
> -
>
> Key: HDFS-8580
> URL: https://issues.apache.org/jira/browse/HDFS-8580
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: HDFS-7285
>
> Attachments: HDFS-8580-HDFS-7285.01.patch,
> HDFS-8580-HDFS-7285.02.patch, HDFS-8580-HDFS-7285.03.patch,
> HDFS-8580-HDFS-7285.04.patch, HDFS-8580.00.patch
>
>
> Zhe Zhang, Kai Zheng and I had a offline discussion. Here is what we thought:
> Add a cellSize field in BlockInfoStriped as a workaround, and deal with
> memory usage in follow-on.(HDFS-8059)
> discussion in HDFS-8494:
> from Walter Su:
> {quote}
> I think BlockInfoStriped needs to keep cellSize.
> {quote}
> from [~vinayrpet]:
> {quote}
> I too was thinking the same when the FSImageLoader problem has came up. This
> will increase the memory usage by ~4bytes for each block though.
> {quote}
> from [~jingzhao]
> {quote}
> -Also, we should consider adding a chunk size field to StripedBlockProto and
> removing the cell size field from HdfsFileStatus. In this way we can access
> the chunk size information in the storage layer.-
> {quote}
> ==
> update:
> from [~jingzhao]
> {quote}
> For fsimage part, since HDFS-8585 just removes StripedBlockProto, I guess
> what we can do here is to either 1) add the cellSize information into
> StripedBlocksFeature in fsimage.proto, or 2) bring StripedBlockProto back and
> put block info and cell size there.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7339) Representing striped block groups in NameNode with hierarchical naming protocol
[
https://issues.apache.org/jira/browse/HDFS-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937984#comment-14937984
]
Hudson commented on HDFS-7339:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-7339. Allocating and persisting block groups in NameNode. (zhezhang: rev
bc2833b1c91e107d090619d755c584f6eae82327)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockgroup.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockIdGenerator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockGroupIdGenerator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
> Representing striped block groups in NameNode with hierarchical naming
> protocol
> ---
>
> Key: HDFS-7339
> URL: https://issues.apache.org/jira/browse/HDFS-7339
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-7339-001.patch, HDFS-7339-002.patch,
> HDFS-7339-003.patch, HDFS-7339-004.patch, HDFS-7339-005.patch,
> HDFS-7339-006.patch, HDFS-7339-007.patch, HDFS-7339-008.patch,
> Meta-striping.jpg, NN-stripping.jpg
>
>
> All erasure codec operations center around the concept of _block group_; they
> are formed in initial encoding and looked up in recoveries and conversions. A
> lightweight class {{BlockGroup}} is created to record the original and parity
> blocks in a coding group, as well as a pointer to the codec schema (pluggable
> codec schemas will be supported in HDFS-7337). With the striping layout, the
> HDFS client needs to operate on all blocks in a {{BlockGroup}} concurrently.
> Therefore we propose to extend a file’s inode to switch between _contiguous_
> and _striping_ modes, with the current mode recorded in a binary flag. An
> array of BlockGroups (or BlockGroup IDs) is added, which remains empty for
> “traditional” HDFS files with contiguous block layout.
> The NameNode creates and maintains {{BlockGroup}} instances through the new
> {{ECManager}} component; the attached figure has an illustration of the
> architecture. As a simple example, when a {_Striping+EC_} file is created and
> written to, it will serve requests from the client to allocate new
> {{BlockGroups}} and store them under the {{INodeFile}}. In the current phase,
> {{BlockGroups}} are allocated both in initial online encoding and in the
> conversion from replication to EC. {{ECManager}} also facilitates the lookup
> of {{BlockGroup}} information for block recovery work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9086) Rename dfs.datanode.stripedread.threshold.millis to dfs.datanode.stripedread.timeout.millis
[ https://issues.apache.org/jira/browse/HDFS-9086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938016#comment-14938016 ] Hudson commented on HDFS-9086: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-9086. Rename dfs.datanode.stripedread.threshold.millis to (zhezhang: rev 82a88b92b46911c1b33d11898f8f678a134a9b69) * hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml > Rename dfs.datanode.stripedread.threshold.millis to > dfs.datanode.stripedread.timeout.millis > --- > > Key: HDFS-9086 > URL: https://issues.apache.org/jira/browse/HDFS-9086 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Affects Versions: HDFS-7285 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Trivial > Fix For: HDFS-7285 > > Attachments: HDFS-9086-HDFS-7285.001.patch, > hdfs-9086-hdfs-7285.001.patch > > > This config key is used to control the timeout for ECWorker reads, let's name > it with the standard term "timeout" rather than "threshold". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7826) Erasure Coding: Update INodeFile quota computation for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938004#comment-14938004
]
Hudson commented on HDFS-7826:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-7826. Erasure Coding: Update INodeFile quota computation for (zhezhang:
rev 26773d9d6c10479982a3cdbea3a0933f4476add3)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java
> Erasure Coding: Update INodeFile quota computation for striped blocks
> -
>
> Key: HDFS-7826
> URL: https://issues.apache.org/jira/browse/HDFS-7826
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Kai Sasaki
> Fix For: HDFS-7285
>
> Attachments: HDFS-7826.1.patch, HDFS-7826.2.patch, HDFS-7826.3.patch,
> HDFS-7826.4.patch, HDFS-7826.5.patch
>
>
> Currently INodeFile's quota computation only considers contiguous blocks
> (i.e., {{INodeFile#blocks}}). We need to update it to support striped blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8684) Erasure Coding: fix some block number calculation for striped block
[
https://issues.apache.org/jira/browse/HDFS-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938009#comment-14938009
]
Hudson commented on HDFS-8684:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-8684. Erasure Coding: fix some block number calculation for striped (yliu:
rev ee01a09500224136464f2c3e0a5d9ba53242d93f)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
> Erasure Coding: fix some block number calculation for striped block
> ---
>
> Key: HDFS-8684
> URL: https://issues.apache.org/jira/browse/HDFS-8684
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-8684-HDFS-7285.001.patch,
> HDFS-8684-HDFS-7285.002.patch
>
>
> in INodeFile#computeFileSize, the file size calucation for underconstruction
> striped block is incorrect.
> in BlockManager#chooseExcessReplicasStriped, the {{if (nonExcess.size() <=
> groupSize) {}} is incorrect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7347) Configurable erasure coding policy for individual files and directories
[ https://issues.apache.org/jira/browse/HDFS-7347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937995#comment-14937995 ] Hudson commented on HDFS-7347: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-7347. Configurable erasure coding policy for individual files and (zhezhang: rev f166e67a23bc49e932b23876ede6f8ab9c9f76d6) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInitialEncoding.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > Configurable erasure coding policy for individual files and directories > --- > > Key: HDFS-7347 > URL: https://issues.apache.org/jira/browse/HDFS-7347 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: HDFS-7285 > > Attachments: HDFS-7347-20141104.patch, HDFS-7347-20141105.patch > > > HDFS users and admins should be able to turn on and off erasure coding for > individual files or directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8585) Erasure Coding: Remove dataBlockNum and parityBlockNum from StripedBlockProto
[
https://issues.apache.org/jira/browse/HDFS-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938005#comment-14938005
]
Hudson commented on HDFS-8585:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-8585. Erasure Coding: Remove dataBlockNum and parityBlockNum from (jing9:
rev 683332b36de1040eb8901d676e666527e8c5f8fe)
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/fsimage.proto
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewerWithStripedBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageLoader.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
> Erasure Coding: Remove dataBlockNum and parityBlockNum from StripedBlockProto
> --
>
> Key: HDFS-8585
> URL: https://issues.apache.org/jira/browse/HDFS-8585
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: HDFS-7285
>
> Attachments: HDFS-8585-HDFS-7285.001.patch,
> HDFS-8585-HDFS-7285.002.patch
>
>
> Since in HDFS-8427, we remove {{dataBlockNum}} and {{parityBlockNum}}, and
> get it from {{ECSchema}}, so we also need it from {{StripedBlockProto}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8114) Erasure coding: Add auditlog FSNamesystem#createErasureCodingZone if this operation fails
[
https://issues.apache.org/jira/browse/HDFS-8114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938011#comment-14938011
]
Hudson commented on HDFS-8114:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-8114. Erasure coding: Add auditlog (zhezhang: rev
3b56a5aa9e82b5ef4fbafa832947eb80b77f0976)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
> Erasure coding: Add auditlog FSNamesystem#createErasureCodingZone if this
> operation fails
> -
>
> Key: HDFS-8114
> URL: https://issues.apache.org/jira/browse/HDFS-8114
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Minor
> Labels: reviewed
> Fix For: HDFS-7285
>
> Attachments: HDFS-8114-001.patch
>
>
> While reviewing, I've noticed {{createErasureCodingZone}} is not adding
> auditlog if this operation fails. IMHO its good to capture failure case also.
> {code}
> logAuditEvent(true, "createErasureCodingZone", srcArg, null, resultingStat);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7621) Erasure Coding: update the Balancer/Mover data migration logic
[ https://issues.apache.org/jira/browse/HDFS-7621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937994#comment-14937994 ] Hudson commented on HDFS-7621: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-7621. Erasure Coding: update the Balancer/Mover data migration (zhezhang: rev 673280df24f0228bf01777035ceeab8807da8c40) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlocksWithLocations.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java > Erasure Coding: update the Balancer/Mover data migration logic > -- > > Key: HDFS-7621 > URL: https://issues.apache.org/jira/browse/HDFS-7621 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Walter Su > Labels: HDFS-7285 > Fix For: HDFS-7285 > > Attachments: HDFS-7621.001.patch, HDFS-7621.002.patch, > HDFS-7621.003.patch, HDFS-7621.004.patch, HDFS-7621.005.patch, > HDFS-7621.006.patch, HDFS-7621.007.patch > > > Currently the Balancer/Mover only considers the distribution of replicas of > the same block during data migration: the migration cannot decrease the > number of racks. With EC the Balancer and Mover should also take into account > the distribution of blocks belonging to the same block group. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8391) NN should consider current EC tasks handling count from DN while assigning new tasks
[
https://issues.apache.org/jira/browse/HDFS-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938052#comment-14938052
]
Hudson commented on HDFS-8391:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-8391. NN should consider current EC tasks handling count from DN
(zhezhang: rev c99c3379282779e11ebda88d845bb89407f2f350)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> NN should consider current EC tasks handling count from DN while assigning
> new tasks
>
>
> Key: HDFS-8391
> URL: https://issues.apache.org/jira/browse/HDFS-8391
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Fix For: HDFS-7285
>
> Attachments: HDFS-8391-01.patch
>
>
> Currently NN will pick the (maxStreams-XmitsCount) number of ECtasks at a
> time for assigning to the respective DN.
> {code}
> //get datanode commands
> final int maxTransfer = blockManager.getMaxReplicationStreams()
> - xmitsInProgress;
> {code}
> But right now we increment xmitsInProgress count at DN only for regular
> replication tasks but not for ECWorker tasks.
> So, either we should treat this logic separately for EC or we should consider
> EC current task handling count form DN.
> This jira for discussing more for this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8146) Protobuf changes for BlockECRecoveryCommand and its fields for making it ready for transfer to DN
[ https://issues.apache.org/jira/browse/HDFS-8146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938068#comment-14938068 ] Hudson commented on HDFS-8146: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-8146. Protobuf changes for BlockECRecoveryCommand and its fields (zhezhang: rev a1c9425265d2c94bfc6afb39ab2c16b4ef9e874e) * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/erasurecoding.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtocol.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestRecoverStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockECRecoveryCommand.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java > Protobuf changes for BlockECRecoveryCommand and its fields for making it > ready for transfer to DN > -- > > Key: HDFS-8146 > URL: https://issues.apache.org/jira/browse/HDFS-8146 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: HDFS-7285 > > Attachments: HDFS-8146-1.patch, HDFS-8146-2.patch, HDFS-8146-3.patch, > HDFS-8146.0.patch > > > As part of working on HDFS-8137, we need to prepare BlockECRecoveryCommand, > BlockECRecoveryInfo, DatanodeStorageInfo, DatanodeDescripter (We can use > DatanodeInfo for proto trasfer) should be ready in proto format for > transferring them in command. Since all this code could be straight forward > and to have better focussed review on core part, I propose to separate this > part in to this JIRA. First I will prepare all this supported classes > protobuf ready and then trasnfer them to DN as part of HDFS-8137 by including > ECSchema. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9097) Erasure coding: update EC command "-s" flag to "-p" when specifying policy
[ https://issues.apache.org/jira/browse/HDFS-9097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938043#comment-14938043 ] Hudson commented on HDFS-9097: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-9097. Erasure coding: update EC command "-s" flag to "-p" when (zhezhang: rev 607bda2a21aebca7b363d57ddece8ce8b5db6c63) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/erasurecode/ECCommand.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testErasureCodingConf.xml > Erasure coding: update EC command "-s" flag to "-p" when specifying policy > -- > > Key: HDFS-9097 > URL: https://issues.apache.org/jira/browse/HDFS-9097 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: HDFS-7285 > > Attachments: HDFS-9097-HDFS-7285.00.patch > > > HDFS-8833 missed this update. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[ https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938038#comment-14938038 ] Hudson commented on HDFS-8833: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-8833. Erasure coding: store EC schema and cell size in INodeFile (zhz: rev f62237bc2f02afe11ce185e13aa51a60b5960037) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingZones.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingZoneManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/erasurecode/ECCommand.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingPolicies.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testErasureCodingConf.xml * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ErasureCodingZone.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStripedINodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestSequentialBlockGroupId.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeModeWithStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileAttributes.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithMissingBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileStatusWithECPolicy.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirErasureCodingOp.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecoding.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewerWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFSStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddOverReplicatedStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteStripedFileWithFailure.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java *
[jira] [Commented] (HDFS-8798) Erasure Coding: fix DFSStripedInputStream/DFSStripedOutputStream re-fetch token when expired
[ https://issues.apache.org/jira/browse/HDFS-8798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938048#comment-14938048 ] Hudson commented on HDFS-8798: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-8798. Erasure Coding: fix (jing9: rev 95b499a3671daae9018ae005c9384fb65aa37320) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFSStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java > Erasure Coding: fix DFSStripedInputStream/DFSStripedOutputStream re-fetch > token when expired > > > Key: HDFS-8798 > URL: https://issues.apache.org/jira/browse/HDFS-8798 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Walter Su >Assignee: Walter Su > Fix For: HDFS-7285 > > Attachments: HDFS-8798-HDFS-7285.02.patch, HDFS-8798.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8461) Erasure coding: fix priority level of UnderReplicatedBlocks for striped block
[
https://issues.apache.org/jira/browse/HDFS-8461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938055#comment-14938055
]
Hudson commented on HDFS-8461:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-8461. Erasure coding: fix priority level of UnderReplicatedBlocks (jing9:
rev 2c494a843699b478039f41336cf47bd4c635eb76)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestRecoverStripedBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
> Erasure coding: fix priority level of UnderReplicatedBlocks for striped block
> -
>
> Key: HDFS-8461
> URL: https://issues.apache.org/jira/browse/HDFS-8461
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: HDFS-7285
>
> Attachments: HDFS-8461-HDFS-7285.001.patch,
> HDFS-8461-HDFS-7285.002.patch
>
>
> Issues 1: correctly mark corrupted blocks.
> Issues 2: distinguish highest risk priority and normal risk priority.
> {code:title=UnderReplicatedBlocks.java}
> private int getPriority(int curReplicas,
> ...
> } else if (curReplicas == 1) {
> //only on replica -risk of loss
> // highest priority
> return QUEUE_HIGHEST_PRIORITY;
> ...
> {code}
> For stripe blocks, we should return QUEUE_HIGHEST_PRIORITY when curReplicas
> == 6( Suppose 6+3 schema).
> That's important. Because
> {code:title=BlockManager.java}
> DatanodeDescriptor[] chooseSourceDatanodes(BlockInfo block,
> ...
> if(priority != UnderReplicatedBlocks.QUEUE_HIGHEST_PRIORITY
> && !node.isDecommissionInProgress()
> && node.getNumberOfBlocksToBeReplicated() >= maxReplicationStreams)
> {
> continue; // already reached replication limit
> }
> ...
> {code}
> It may return not enough source DNs ( maybe 5), and failed to recover.
> A busy node should not be skiped if a block has highest risk/priority. The
> issue is the striped block doesn't have priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8937) Erasure coding: do not throw exception when setting replication factor on EC files
[ https://issues.apache.org/jira/browse/HDFS-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938036#comment-14938036 ] Hudson commented on HDFS-8937: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-8937. Erasure coding: do not throw exception when setting (jing9: rev ddf4e785475affead2f7c070b9f151de0fcb9024) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingZones.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java > Erasure coding: do not throw exception when setting replication factor on EC > files > -- > > Key: HDFS-8937 > URL: https://issues.apache.org/jira/browse/HDFS-8937 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: GAO Rui >Assignee: GAO Rui > Fix For: HDFS-7285 > > Attachments: HDFS-8937-HDFS-7285.00.patch, > HDFS-8937-HDFS-7285.01.patch > > > Setting replication to an EC file caused exception. We should simply ignore > the request, just like what we're currently doing for a setReplication > request against a directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8453) Erasure coding: properly handle start offset for internal blocks in a block group
[
https://issues.apache.org/jira/browse/HDFS-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938027#comment-14938027
]
Hudson commented on HDFS-8453:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-8453. Erasure coding: properly handle start offset for internal (jing9:
rev 5f15084bd530865d3e2641b709665b5b7971a74d)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecoding.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
> Erasure coding: properly handle start offset for internal blocks in a block
> group
> -
>
> Key: HDFS-8453
> URL: https://issues.apache.org/jira/browse/HDFS-8453
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: HDFS-7285
>
> Attachments: HDFS-8453-HDFS-7285.00.patch,
> HDFS-8453-HDFS-7285.01.patch
>
>
> {code}
> void actualGetFromOneDataNode(final DNAddrPair datanode,
> ...
> LocatedBlock block = getBlockAt(blockStartOffset);
> ...
> fetchBlockAt(block.getStartOffset());
> {code}
> The {{blockStartOffset}} here is from inner block. For parity blocks, the
> offset will overlap with the next block group, and we may end up with
> fetching wrong block. So we have to assign a meaningful start offset for
> internal blocks in a block group, especially for parity blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8368) Erasure Coding: DFS opening a non-existent file need to be handled properly
[
https://issues.apache.org/jira/browse/HDFS-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938008#comment-14938008
]
Hudson commented on HDFS-8368:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-8368. Erasure Coding: DFS opening a non-existent file need to be
(zhezhang: rev 54d28275226e0bc3b0d46142ce20f74a52e66395)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
> Erasure Coding: DFS opening a non-existent file need to be handled properly
> ---
>
> Key: HDFS-8368
> URL: https://issues.apache.org/jira/browse/HDFS-8368
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
> Fix For: HDFS-7285
>
> Attachments: HDFS-8368-HDFS-7285.00.patch
>
>
> This jira to address bad exceptions when opening a non-existent file. It
> throws NPE as shown below:
> {code}
> java.lang.NullPointerException: null
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1184)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:307)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:303)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:303)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
> at
> org.apache.hadoop.hdfs.TestDistributedFileSystem.testDFSClient(TestDistributedFileSystem.java:359)
> at
> org.apache.hadoop.hdfs.TestDistributedFileSystem.testAllWithNoXmlDefaults(TestDistributedFileSystem.java:666)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8781) Erasure Coding: Correctly handle BlockManager#InvalidateBlocks for striped block
[
https://issues.apache.org/jira/browse/HDFS-8781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937999#comment-14937999
]
Hudson commented on HDFS-8781:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-8781. Erasure Coding: Correctly handle (jing9: rev
5956d23b645e272748e2edca4c30231e729b96a9)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecoding.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
> Erasure Coding: Correctly handle BlockManager#InvalidateBlocks for striped
> block
>
>
> Key: HDFS-8781
> URL: https://issues.apache.org/jira/browse/HDFS-8781
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: HDFS-7285
>
> Attachments: HDFS-8781-HDFS-7285.001.patch,
> HDFS-8781-HDFS-7285.002.patch
>
>
> For {{BlockManager#InvalidateBlocks}}, currently some places track the
> internal block, some other places track the block group for striped block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8077) Erasure coding: fix bug in EC zone and symlinks
[
https://issues.apache.org/jira/browse/HDFS-8077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938018#comment-14938018
]
Hudson commented on HDFS-8077:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-8077. Erasure coding: fix bugs in EC zone and symlinks. Contributed
(zhezhang: rev 4d0bc724f29b646e252f53d1c654a23e8526a4bf)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestRecoverStripedBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingZoneManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStripedUnderConstruction.java
> Erasure coding: fix bug in EC zone and symlinks
> ---
>
> Key: HDFS-8077
> URL: https://issues.apache.org/jira/browse/HDFS-8077
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Jing Zhao
> Fix For: HDFS-7285
>
> Attachments: HDFS-8077-000.patch, HDFS-8077.001.patch,
> HDFS-8077.002.patch
>
>
> EC zone manager tries to get XAttr of an inode to determine the EC policy,
> which doesn't work with symlinks. This patch has a simple fix to get rid of
> test failures.
> Ideally we should also add logic to disallow creating symlinks in several
> EC-related scenarios. But since symlinks are disabled in branch-2 and will
> likely be disabled in trunk, this step is skipped now.
> The patch also fixes a small test error around {{getBlockReplication}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8484) Erasure coding: Two contiguous blocks occupy IDs belong to same striped group
[ https://issues.apache.org/jira/browse/HDFS-8484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938003#comment-14938003 ] Hudson commented on HDFS-8484: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-8484. Erasure coding: Two contiguous blocks occupy IDs belong to (jing9: rev e692c7dd92c0d0fef50ce08edd972000151a9fe4) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java > Erasure coding: Two contiguous blocks occupy IDs belong to same striped group > - > > Key: HDFS-8484 > URL: https://issues.apache.org/jira/browse/HDFS-8484 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Walter Su >Assignee: Walter Su >Priority: Trivial > Fix For: HDFS-7285 > > Attachments: HDFS-8484-HDFS-7285.001.patch > > > There's very very little chance: > Assume \[-1016,-1001\] is a block group ID. > A contiguous block has ID -1016. There is another contiguous block has ID > -1009. > When we want to get -1009 block, actually we get -1016. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8479) Erasure coding: fix striping related logic in FSDirWriteFileOp to sync with HDFS-8421
[ https://issues.apache.org/jira/browse/HDFS-8479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937992#comment-14937992 ] Hudson commented on HDFS-8479: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-8479. Erasure coding: fix striping related logic in (jing9: rev 1299357a05c52ad45513ed0ea854edc9c7ec3de8) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > Erasure coding: fix striping related logic in FSDirWriteFileOp to sync with > HDFS-8421 > - > > Key: HDFS-8479 > URL: https://issues.apache.org/jira/browse/HDFS-8479 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: HDFS-7285 > > Attachments: HDFS-8479-HDFS-7285.0.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7351) Document the HDFS Erasure Coding feature
[ https://issues.apache.org/jira/browse/HDFS-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938001#comment-14938001 ] Hudson commented on HDFS-7351: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-7351. Document the HDFS Erasure Coding feature. Contributed by Uma (wang: rev c2ebab67acf421f52dfa4879261982fed2768c82) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md * hadoop-project/src/site/site.xml * hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md > Document the HDFS Erasure Coding feature > > > Key: HDFS-7351 > URL: https://issues.apache.org/jira/browse/HDFS-7351 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: HDFS-7285 > > Attachments: HDFS-7351-HDFS-7285-01.patch, > HDFS-7351-HDFS-7285-02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8272) Erasure Coding: simplify the retry logic in DFSStripedInputStream (stateful read)
[ https://issues.apache.org/jira/browse/HDFS-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937993#comment-14937993 ] Hudson commented on HDFS-8272: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-8272. Erasure Coding: simplify the retry logic in (zhezhang: rev b00c66387709bd7c2411eb9778b5d15de53672fc) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > Erasure Coding: simplify the retry logic in DFSStripedInputStream (stateful > read) > - > > Key: HDFS-8272 > URL: https://issues.apache.org/jira/browse/HDFS-8272 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: HDFS-7285 > > Attachments: HDFS-8272.002.patch, h8272-HDFS-7285.000.patch, > h8272-HDFS-7285.001.patch > > > Currently in DFSStripedInputStream the retry logic is still the same with > DFSInputStream. More specifically, every failed read will try to search for > another source node. And an exception is thrown when no new source node can > be identified. This logic is not appropriate for EC inputstream and can be > simplified. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files
[
https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938014#comment-14938014
]
Hudson commented on HDFS-7889:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-7889 Subclass DFSOutputStream to support writing striping layout
(zhezhang: rev 4970f2a2ab8f898e85ddc6e5dd477e844424a999)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> Subclass DFSOutputStream to support writing striping layout files
> -
>
> Key: HDFS-7889
> URL: https://issues.apache.org/jira/browse/HDFS-7889
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Li Bo
>Assignee: Li Bo
> Fix For: HDFS-7285
>
> Attachments: HDFS-7889-001.patch, HDFS-7889-002.patch,
> HDFS-7889-003.patch, HDFS-7889-004.patch, HDFS-7889-005.patch,
> HDFS-7889-006.patch, HDFS-7889-007.patch, HDFS-7889-008.patch,
> HDFS-7889-009.patch, HDFS-7889-010.patch, HDFS-7889-011.patch,
> HDFS-7889-012.patch, HDFS-7889-013.patch, HDFS-7889-014.patch
>
>
> After HDFS-7888, we can subclass {{DFSOutputStream}} to support writing
> striping layout files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
