[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937787#comment-14937787 ] Hudson commented on HDFS-7827: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-7827. Erasure Coding: support striped blocks in non-protobuf (zhezhang: rev d0d75a833907f6cf723a42a007ca04e0004a8e52) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoStriped.java > Erasure Coding: support striped blocks in non-protobuf fsimage > -- > > Key: HDFS-7827 > URL: https://issues.apache.org/jira/browse/HDFS-7827 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Hui Zheng > Fix For: HDFS-7285 > > Attachments: HDFS-7827.000.patch, HDFS-7827.002.patch, > HDFS-7827.003.patch, HDFS-7827.004.patch > > > HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. > We should also add this support to the non-protobuf fsimage since it is still > used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938006#comment-14938006 ] Hudson commented on HDFS-7827: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-7827. Erasure Coding: support striped blocks in non-protobuf (zhezhang: rev d0d75a833907f6cf723a42a007ca04e0004a8e52) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java > Erasure Coding: support striped blocks in non-protobuf fsimage > -- > > Key: HDFS-7827 > URL: https://issues.apache.org/jira/browse/HDFS-7827 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Hui Zheng > Fix For: HDFS-7285 > > Attachments: HDFS-7827.000.patch, HDFS-7827.002.patch, > HDFS-7827.003.patch, HDFS-7827.004.patch > > > HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. > We should also add this support to the non-protobuf fsimage since it is still > used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938576#comment-14938576 ] Hudson commented on HDFS-7827: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #465 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/465/]) HDFS-7827. Erasure Coding: support striped blocks in non-protobuf (zhezhang: rev d0d75a833907f6cf723a42a007ca04e0004a8e52) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java > Erasure Coding: support striped blocks in non-protobuf fsimage > -- > > Key: HDFS-7827 > URL: https://issues.apache.org/jira/browse/HDFS-7827 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Hui Zheng > Fix For: HDFS-7285 > > Attachments: HDFS-7827.000.patch, HDFS-7827.002.patch, > HDFS-7827.003.patch, HDFS-7827.004.patch > > > HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. > We should also add this support to the non-protobuf fsimage since it is still > used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938382#comment-14938382 ] Hudson commented on HDFS-7827: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2408 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2408/]) HDFS-7827. Erasure Coding: support striped blocks in non-protobuf (zhezhang: rev d0d75a833907f6cf723a42a007ca04e0004a8e52) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java > Erasure Coding: support striped blocks in non-protobuf fsimage > -- > > Key: HDFS-7827 > URL: https://issues.apache.org/jira/browse/HDFS-7827 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Hui Zheng > Fix For: HDFS-7285 > > Attachments: HDFS-7827.000.patch, HDFS-7827.002.patch, > HDFS-7827.003.patch, HDFS-7827.004.patch > > > HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. > We should also add this support to the non-protobuf fsimage since it is still > used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937530#comment-14937530 ] Hudson commented on HDFS-7827: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1203 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1203/]) HDFS-7827. Erasure Coding: support striped blocks in non-protobuf (zhezhang: rev d0d75a833907f6cf723a42a007ca04e0004a8e52) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoStriped.java > Erasure Coding: support striped blocks in non-protobuf fsimage > -- > > Key: HDFS-7827 > URL: https://issues.apache.org/jira/browse/HDFS-7827 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Hui Zheng > Fix For: HDFS-7285 > > Attachments: HDFS-7827.000.patch, HDFS-7827.002.patch, > HDFS-7827.003.patch, HDFS-7827.004.patch > > > HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. > We should also add this support to the non-protobuf fsimage since it is still > used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937280#comment-14937280 ] Hudson commented on HDFS-7827: -- FAILURE: Integrated in Hadoop-trunk-Commit #8548 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8548/]) HDFS-7827. Erasure Coding: support striped blocks in non-protobuf (zhezhang: rev d0d75a833907f6cf723a42a007ca04e0004a8e52) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java > Erasure Coding: support striped blocks in non-protobuf fsimage > -- > > Key: HDFS-7827 > URL: https://issues.apache.org/jira/browse/HDFS-7827 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Hui Zheng > Fix For: HDFS-7285 > > Attachments: HDFS-7827.000.patch, HDFS-7827.002.patch, > HDFS-7827.003.patch, HDFS-7827.004.patch > > > HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. > We should also add this support to the non-protobuf fsimage since it is still > used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937085#comment-14937085 ] Hudson commented on HDFS-7827: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2379 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2379/]) HDFS-7827. Erasure Coding: support striped blocks in non-protobuf (zhezhang: rev d0d75a833907f6cf723a42a007ca04e0004a8e52) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java > Erasure Coding: support striped blocks in non-protobuf fsimage > -- > > Key: HDFS-7827 > URL: https://issues.apache.org/jira/browse/HDFS-7827 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Hui Zheng > Fix For: HDFS-7285 > > Attachments: HDFS-7827.000.patch, HDFS-7827.002.patch, > HDFS-7827.003.patch, HDFS-7827.004.patch > > > HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. > We should also add this support to the non-protobuf fsimage since it is still > used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[
https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577605#comment-14577605
]
Jing Zhao commented on HDFS-7827:
-
I just found that for a legacy fsimage we currently always record its
layoutversion as -51 so that we can make sure it cannot be processed by a
protobuf-based image parser. Thus the following code in the parser always
returns false and the parse will fail. I will create a jira to fix this.
{code}
NameNodeLayoutVersion.supports(
NameNodeLayoutVersion.Feature.ERASURE_CODING, imgVersion)
{code}
Erasure Coding: support striped blocks in non-protobuf fsimage
--
Key: HDFS-7827
URL: https://issues.apache.org/jira/browse/HDFS-7827
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Hui Zheng
Fix For: HDFS-7285
Attachments: HDFS-7827.000.patch, HDFS-7827.002.patch,
HDFS-7827.003.patch, HDFS-7827.004.patch
HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage.
We should also add this support to the non-protobuf fsimage since it is still
used for use cases like offline image processing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375309#comment-14375309 ] Hui Zheng commented on HDFS-7827: - Hi Jing Thank you for your help.The 004 patch looks good to me. Also I have known where is not following the coding convention by the patch. Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch, HDFS-7827.002.patch, HDFS-7827.003.patch, HDFS-7827.004.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[
https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364100#comment-14364100
]
Jing Zhao commented on HDFS-7827:
-
Currently we still have non-protobuf image for offline processing. Thus this
change can still be helpful especially for storage usage analysis.
Some comments on the current patch:
# After updating {{writeINodeUnderConstruction}}, we also need to update
{{readINodeUnderConstruction}} accordingly.
# We also need a unit test to cover the under construction file with striped
blocks.
# Instead of the following code, we can simply use
out.writeBoolean(cons.isWithStripedBlocks()). Whether we keep using
EC_STORAGE_POLICY_ID is still under construction.
{code}
+if (cons.isStriped()){
+ out.writeByte(HdfsConstants.EC_STORAGE_POLICY_ID);
+}else{
+ out.writeByte(0);
+}
{code}
# Please correct the coding format according to the project conventions.
Besides, as [~wheat9] pointed out, we also need to update WebImageViewer for
file size calculation. Please open a separate jira to do this.
Erasure Coding: support striped blocks in non-protobuf fsimage
--
Key: HDFS-7827
URL: https://issues.apache.org/jira/browse/HDFS-7827
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Hui Zheng
Attachments: HDFS-7827.000.patch, HDFS-7827.001.patch,
HDFS-7827.002.patch
HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage.
We should also add this support to the non-protobuf fsimage since it is still
used for use cases like offline image processing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14362853#comment-14362853 ] Hui Zheng commented on HDFS-7827: - I have add a testcase for saving and loading a INodeFile. Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch, HDFS-7827.001.patch, HDFS-7827.002.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14362863#comment-14362863 ] Hui Zheng commented on HDFS-7827: - Thanks [~wheat9] Is there any plan to remove the non-protobuf fsimage? Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch, HDFS-7827.001.patch, HDFS-7827.002.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360678#comment-14360678 ] Haohui Mai commented on HDFS-7827: -- I'm not sure whether this is the right thing to do as the old fsimage is quite out-of-dated compared to the features we have today. It might make sense to add the information to WebImageViewer though. Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch, HDFS-7827.001.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358398#comment-14358398 ] Hui Zheng commented on HDFS-7827: - Hi Jing I made a mistake and found that the INodeFile does not serialize its header.So we need to record the type of the blocks in the fsimage as you said. Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356515#comment-14356515 ] Hui Zheng commented on HDFS-7827: - Hi Jing I agree that we need a flag to identify the striped block, but could we use the STORAGE_POLICY_ID in header of INodeFile? Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355284#comment-14355284 ] Jing Zhao commented on HDFS-7827: - Yes, you're right. Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355752#comment-14355752 ] Jing Zhao commented on HDFS-7827: - Hi Hui, the current patch looks good to me. One comment is that we may need to explicitly record the type of the blocks in the fsimage (e.g., using a byte in the fsimage to indicate if the file has striped block). Otherwise the offline image viewer or fsimage loader has no way to identify the correct block type when reading the block list. As you mentioned, we also need to add support in offline image viewer. Besides, though we may not have a use case anymore, it will be more complete to update the corresponding FSImage Loader code (more specifically, FImageFormat#loadINode). Note that we need to check the NameNodeLayoutVersion there. Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354549#comment-14354549 ] Hui Zheng commented on HDFS-7827: - I think we don't need to implement the loading of a non-protobuf fsimage in Namenode,so it is not nessary to implement readFields method ofthe Writable inface. But we need to support it in tools(ImageLoaderCurrent.java). Is that right? Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352435#comment-14352435 ] Hui Zheng commented on HDFS-7827: - To support striped block in no-protobuf fsimage we only need to persist striped blocks in non-protobuf format. The BlockInfo class uses writable interface to persist itself in non-protobuf format. So we can override the write method to implement it. Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14336134#comment-14336134 ] Jing Zhao commented on HDFS-7827: - Sure. Assign the jira to you. Thanks for working on this, Hui! Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14336117#comment-14336117 ] Hui Zheng commented on HDFS-7827: - Hi Jing I would like to work on this jira. Could you assign it to me? Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Jing Zhao HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
