[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[ https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938113#comment-14938113 ] Hudson commented on HDFS-7949: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-7949. WebImageViewer need support file size calculation with (zhezhang: rev 77d94dd5ece2b12b0dfcc7e3b3094318115779b5) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewerWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageLoader.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > WebImageViewer need support file size calculation with striped blocks > - > > Key: HDFS-7949 > URL: https://issues.apache.org/jira/browse/HDFS-7949 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Hui Zheng >Assignee: Rakesh R > Fix For: HDFS-7285 > > Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch, > HDFS-7949-003.patch, HDFS-7949-004.patch, HDFS-7949-005.patch, > HDFS-7949-006.patch, HDFS-7949-007.patch, HDFS-7949-HDFS-7285.08.patch, > HDFS-7949-HDFS-7285.08.patch > > > The file size calculation should be changed when the blocks of the file are > striped in WebImageViewer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[ https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937893#comment-14937893 ] Hudson commented on HDFS-7949: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-7949. WebImageViewer need support file size calculation with (zhezhang: rev 77d94dd5ece2b12b0dfcc7e3b3094318115779b5) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageLoader.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewerWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > WebImageViewer need support file size calculation with striped blocks > - > > Key: HDFS-7949 > URL: https://issues.apache.org/jira/browse/HDFS-7949 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Hui Zheng >Assignee: Rakesh R > Fix For: HDFS-7285 > > Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch, > HDFS-7949-003.patch, HDFS-7949-004.patch, HDFS-7949-005.patch, > HDFS-7949-006.patch, HDFS-7949-007.patch, HDFS-7949-HDFS-7285.08.patch, > HDFS-7949-HDFS-7285.08.patch > > > The file size calculation should be changed when the blocks of the file are > striped in WebImageViewer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[ https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938682#comment-14938682 ] Hudson commented on HDFS-7949: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #465 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/465/]) HDFS-7949. WebImageViewer need support file size calculation with (zhezhang: rev 77d94dd5ece2b12b0dfcc7e3b3094318115779b5) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageLoader.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewerWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > WebImageViewer need support file size calculation with striped blocks > - > > Key: HDFS-7949 > URL: https://issues.apache.org/jira/browse/HDFS-7949 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Hui Zheng >Assignee: Rakesh R > Fix For: HDFS-7285 > > Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch, > HDFS-7949-003.patch, HDFS-7949-004.patch, HDFS-7949-005.patch, > HDFS-7949-006.patch, HDFS-7949-007.patch, HDFS-7949-HDFS-7285.08.patch, > HDFS-7949-HDFS-7285.08.patch > > > The file size calculation should be changed when the blocks of the file are > striped in WebImageViewer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[ https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938490#comment-14938490 ] Hudson commented on HDFS-7949: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2408 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2408/]) HDFS-7949. WebImageViewer need support file size calculation with (zhezhang: rev 77d94dd5ece2b12b0dfcc7e3b3094318115779b5) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewerWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageLoader.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java > WebImageViewer need support file size calculation with striped blocks > - > > Key: HDFS-7949 > URL: https://issues.apache.org/jira/browse/HDFS-7949 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Hui Zheng >Assignee: Rakesh R > Fix For: HDFS-7285 > > Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch, > HDFS-7949-003.patch, HDFS-7949-004.patch, HDFS-7949-005.patch, > HDFS-7949-006.patch, HDFS-7949-007.patch, HDFS-7949-HDFS-7285.08.patch, > HDFS-7949-HDFS-7285.08.patch > > > The file size calculation should be changed when the blocks of the file are > striped in WebImageViewer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[ https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937639#comment-14937639 ] Hudson commented on HDFS-7949: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1203 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1203/]) HDFS-7949. WebImageViewer need support file size calculation with (zhezhang: rev 77d94dd5ece2b12b0dfcc7e3b3094318115779b5) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageLoader.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewerWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt > WebImageViewer need support file size calculation with striped blocks > - > > Key: HDFS-7949 > URL: https://issues.apache.org/jira/browse/HDFS-7949 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Hui Zheng >Assignee: Rakesh R > Fix For: HDFS-7285 > > Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch, > HDFS-7949-003.patch, HDFS-7949-004.patch, HDFS-7949-005.patch, > HDFS-7949-006.patch, HDFS-7949-007.patch, HDFS-7949-HDFS-7285.08.patch, > HDFS-7949-HDFS-7285.08.patch > > > The file size calculation should be changed when the blocks of the file are > striped in WebImageViewer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[ https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937390#comment-14937390 ] Hudson commented on HDFS-7949: -- FAILURE: Integrated in Hadoop-trunk-Commit #8548 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8548/]) HDFS-7949. WebImageViewer need support file size calculation with (zhezhang: rev 77d94dd5ece2b12b0dfcc7e3b3094318115779b5) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewerWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageLoader.java > WebImageViewer need support file size calculation with striped blocks > - > > Key: HDFS-7949 > URL: https://issues.apache.org/jira/browse/HDFS-7949 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Hui Zheng >Assignee: Rakesh R > Fix For: HDFS-7285 > > Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch, > HDFS-7949-003.patch, HDFS-7949-004.patch, HDFS-7949-005.patch, > HDFS-7949-006.patch, HDFS-7949-007.patch, HDFS-7949-HDFS-7285.08.patch, > HDFS-7949-HDFS-7285.08.patch > > > The file size calculation should be changed when the blocks of the file are > striped in WebImageViewer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[ https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937193#comment-14937193 ] Hudson commented on HDFS-7949: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2379 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2379/]) HDFS-7949. WebImageViewer need support file size calculation with (zhezhang: rev 77d94dd5ece2b12b0dfcc7e3b3094318115779b5) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageLoader.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewerWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java > WebImageViewer need support file size calculation with striped blocks > - > > Key: HDFS-7949 > URL: https://issues.apache.org/jira/browse/HDFS-7949 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Hui Zheng >Assignee: Rakesh R > Fix For: HDFS-7285 > > Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch, > HDFS-7949-003.patch, HDFS-7949-004.patch, HDFS-7949-005.patch, > HDFS-7949-006.patch, HDFS-7949-007.patch, HDFS-7949-HDFS-7285.08.patch, > HDFS-7949-HDFS-7285.08.patch > > > The file size calculation should be changed when the blocks of the file are > striped in WebImageViewer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[ https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527073#comment-14527073 ] Rakesh R commented on HDFS-7949: Thanks [~zhz] and [~drankye] for the reviews and committing the patch! FYI: Newly added test cases are passing in the build [jenkins report|https://builds.apache.org/job/PreCommit-HDFS-Build/10776/testReport/org.apache.hadoop.hdfs.tools.offlineImageViewer/TestOfflineImageViewerWithStripedBlocks/] WebImageViewer need support file size calculation with striped blocks - Key: HDFS-7949 URL: https://issues.apache.org/jira/browse/HDFS-7949 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Hui Zheng Assignee: Rakesh R Fix For: HDFS-7285 Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch, HDFS-7949-003.patch, HDFS-7949-004.patch, HDFS-7949-005.patch, HDFS-7949-006.patch, HDFS-7949-007.patch, HDFS-7949-HDFS-7285.08.patch, HDFS-7949-HDFS-7285.08.patch The file size calculation should be changed when the blocks of the file are striped in WebImageViewer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[ https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524237#comment-14524237 ] Zhe Zhang commented on HDFS-7949: - Seems Jenkins is too busy. I read the patch again and it looks good. +1, and I just committed it. Thanks Rakesh for the contribution! I'll start our branch Jenkins just to make sure. WebImageViewer need support file size calculation with striped blocks - Key: HDFS-7949 URL: https://issues.apache.org/jira/browse/HDFS-7949 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Hui Zheng Assignee: Rakesh R Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch, HDFS-7949-003.patch, HDFS-7949-004.patch, HDFS-7949-005.patch, HDFS-7949-006.patch, HDFS-7949-007.patch, HDFS-7949-HDFS-7285.08.patch, HDFS-7949-HDFS-7285.08.patch The file size calculation should be changed when the blocks of the file are striped in WebImageViewer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[ https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14521014#comment-14521014 ] Zhe Zhang commented on HDFS-7949: - Thanks Rakesh,! The patch LGTM, +1 pending a Jenkins run. Do you mind Submit Patch and rename the patch as HDFS-7949-HDFS-7285.007.patch? WebImageViewer need support file size calculation with striped blocks - Key: HDFS-7949 URL: https://issues.apache.org/jira/browse/HDFS-7949 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Hui Zheng Assignee: Rakesh R Priority: Minor Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch, HDFS-7949-003.patch, HDFS-7949-004.patch, HDFS-7949-005.patch, HDFS-7949-006.patch, HDFS-7949-007.patch The file size calculation should be changed when the blocks of the file are striped in WebImageViewer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14521609#comment-14521609
]
Rakesh R commented on HDFS-7949:
Thanks [~zhz] for the feedback. I've attached patch which triggered the build,
it has successfully passed the newly written testcase.
{code}
TestOfflineImageViewerWithStripedBlocks 2.9 sec 0 0
7 +7 7 +7
{code}
Unfortunately jenkins has ABORTED due to timeout error. Please see the below
error:
{code}
Building hadoop-common with -Pnative in order to provide libhadoop.so to the
hadoop-hdfs unit tests.
/home/jenkins/tools/maven/latest/bin/mvn compile -Pnative -DhadoopPatchProcess
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/../patchprocess/testrun_native.txt
21
Elapsed time: 3m 14s
Running tests in hadoop-hdfs
/home/jenkins/tools/maven/latest/bin/mvn clean install -fae -Pnative
-Drequire.test.libhadoop -DhadoopPatchProcess
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/../patchprocess/testrun_hadoop-hdfs.txt
21
Build timed out (after 300 minutes). Marking the build as aborted.
Build was aborted
Archiving artifacts
Description set: HDFS-7949
Recording test results
Finished: ABORTED
{code}
Also, I could see this is not the first time, happened couple of times, please
see:
https://builds.apache.org/job/PreCommit-HDFS-Build/10474/console
https://builds.apache.org/job/PreCommit-HDFS-Build/10470/console
https://builds.apache.org/job/PreCommit-HDFS-Build/10461/console
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch,
HDFS-7949-003.patch, HDFS-7949-004.patch, HDFS-7949-005.patch,
HDFS-7949-006.patch, HDFS-7949-007.patch, HDFS-7949-HDFS-7285.08.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14513649#comment-14513649
]
Rakesh R commented on HDFS-7949:
Thanks again [~zhz] for giving more details. Now, I've modified the utility
method as {{StripedBlockUtil#spaceConsumedByStripedBlock}}. Also, I've added
few more tests to see different file sizes. Kindly review the changes!
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch,
HDFS-7949-003.patch, HDFS-7949-004.patch, HDFS-7949-005.patch,
HDFS-7949-006.patch, HDFS-7949-007.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14511884#comment-14511884
]
Zhe Zhang commented on HDFS-7949:
-
Thanks Rakesh for the updated patch; The overall logic looks pretty good to me.
The main concern now is that the new {{spaceConsumed}} method seems to work for
a set of striped block groups. This is ambiguous and probably won't work
correctly because {{getInternalBlockLength}} works for a single internal block.
For example, if {{numDataBlkBytes}} is equal to 2 cells, it could mean a single
data block with size of 2 cells or 2 data blocks each as large as a cell.
How about changing {{spaceConsumed}} to {{spaceConsumedByStripedBlock}} (which
works for a single striped block group)? The body of the method in
{{StripedBlockUtil}} doesn't need to be changed; the main change will be in
{{FSImageLoader}} I guess.
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch,
HDFS-7949-003.patch, HDFS-7949-004.patch, HDFS-7949-005.patch,
HDFS-7949-006.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506409#comment-14506409
]
Rakesh R commented on HDFS-7949:
Thanks [~zhz] for the review. Attached another patch where computed the file
size = datablkbytes + parityblksize.
IIUC {{BlockInfoStriped}} represents blockgroup and getNumBytes() method will
return the total of actual data block size. In the new patch, I'm computing the
parityblksize and multipled with the number of parity blks, does this makes
sense?
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch,
HDFS-7949-003.patch, HDFS-7949-004.patch, HDFS-7949-005.patch,
HDFS-7949-006.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503592#comment-14503592
]
Zhe Zhang commented on HDFS-7949:
-
Thanks Rakesh for the work! The patch looks good overall.
# It needs to be rebased
# {{StripedBlockUtil#getInternalBlockLength}} calculates the num of bytes for
an _internal_ block. The space consumption of a striped block group should be
the sum of the sizes of internal blocks.
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch,
HDFS-7949-003.patch, HDFS-7949-004.patch, HDFS-7949-005.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499478#comment-14499478
]
Rakesh R commented on HDFS-7949:
Thanks [~zhz] and [~drankye] for the review comments. I've tried using
{{StripedBlockUtil#getInternalBlockLength}} for the file size computation.
Please review when you get a chance.
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch,
HDFS-7949-003.patch, HDFS-7949-004.patch, HDFS-7949-005.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497646#comment-14497646
]
Rakesh R commented on HDFS-7949:
Thanks [~zhz] for the review.
bq. Maybe we should wait for that and then move the static {{spaceConsumed}}
method to there?
IIUC your suggestion is to move the {{#spaceConsumed}} function to
{{StripedBlockUtil}} utility, isn't it?
Thanks [~drankye] for the review.
bq. below codes, the assertion isn't clear to me.
For assertion, I'm using {{fileStatus.toString()}} representation which
contains {{length}} attribute info. Now, its hard coded value. As per the
[~zhz] suggestion, I will update using {{#spaceConsumed}} function directly.
Does this sound good to you?
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch,
HDFS-7949-003.patch, HDFS-7949-004.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497829#comment-14497829
]
Kai Zheng commented on HDFS-7949:
-
For the assertion, it would be good not to use hard-coded value, or give it a
nice name. For example, if you write {{TEST_WRITE_BYTES}}, then you will get
that number after reading back, and then compare with it.
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch,
HDFS-7949-003.patch, HDFS-7949-004.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[ https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14498471#comment-14498471 ] Zhe Zhang commented on HDFS-7949: - bq. IIUC your suggestion is to move the #spaceConsumed function to StripedBlockUtil utility, isn't it? Yes that's what I meant. Now HDFS-8120 is already in. WebImageViewer need support file size calculation with striped blocks - Key: HDFS-7949 URL: https://issues.apache.org/jira/browse/HDFS-7949 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Hui Zheng Assignee: Rakesh R Priority: Minor Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch, HDFS-7949-003.patch, HDFS-7949-004.patch The file size calculation should be changed when the blocks of the file are striped in WebImageViewer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496627#comment-14496627
]
Zhe Zhang commented on HDFS-7949:
-
Thanks Rakesh for the update!
bq. Also, updated the hard-coded assert value.
What I meant in the previous review was to remove this hard-coded number
{{13107200}} and use something like the below instead:
{code}
long expected = BlockInfoStriped.spaceConsumed(xxx);
// use expected in assertion
{code}
I just noticed that {{spaceConsumed}} needs to be updated too. Please see the
latest patch under HDFS-8120. We should call
{{StripedBlockUtil#getInternalBlockLength}} to calculate size of parity blocks.
I think that patch will be committed soon. Maybe we should wait for that and
then move the static {{spaceConsumed}} method to there?
Thanks for the good work on this JIRA and sorry about the back and forth.
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch,
HDFS-7949-003.patch, HDFS-7949-004.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497589#comment-14497589
]
Kai Zheng commented on HDFS-7949:
-
The patch looks good. Besides what's commented by Zhe above, ref. below codes,
the assertion isn't clear to me.
{code}
+ assertTrue(
+ Wrongly computed file size contains striped blocks, file status:
+ + fileStatus, fileStatus.contains(length\:13107200));
{code}
bq.can we use a null for the schema here? (when calling
{{createErasureCodingZone}})
Yes, null is allowed, and the system default schema will be used.
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch,
HDFS-7949-003.patch, HDFS-7949-004.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[ https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497364#comment-14497364 ] Kai Zheng commented on HDFS-7949: - Sorry for late on this. I will take a look at the patch today. Thanks for working on this! WebImageViewer need support file size calculation with striped blocks - Key: HDFS-7949 URL: https://issues.apache.org/jira/browse/HDFS-7949 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Hui Zheng Assignee: Rakesh R Priority: Minor Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch, HDFS-7949-003.patch, HDFS-7949-004.patch The file size calculation should be changed when the blocks of the file are striped in WebImageViewer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494390#comment-14494390
]
Zhe Zhang commented on HDFS-7949:
-
Thanks Rakesh for the update!
The latest patch looks good overall.
It needs a minor rebase. The below now needs to specify a schema. [~drankye]
and [~vinayrpet]: can we use a null for the schema here? I tried and the test
failed. Maybe the hard-coded value was based on the old default schema (3+2)?
Now we have (6+3). We should update the hard-coded assert value to link to
system default schema.
{code}
fs.getClient().getNamenode().createErasureCodingZone(/eczone);
{code}
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch,
HDFS-7949-003.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495454#comment-14495454
]
Rakesh R commented on HDFS-7949:
Thanks [~zhz]. I've attached new patch using null for the schema. Also,
updated the hard-coded assert value. Kindly review!
{code}
fs.getClient().getNamenode().createErasureCodingZone(/eczone, null);
{code}
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch,
HDFS-7949-003.patch, HDFS-7949-004.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485578#comment-14485578
]
Zhe Zhang commented on HDFS-7949:
-
Thanks for the update Rakesh.
# I think the static method should calculate the consumed space of a striped
block, instead of a file. Something like the below under {{BlockInfoStriped}}:
{code}
public static long spaceConsumed(long numBytes, short dataBlockNum, short
parityBlockNum) {
// In case striped blocks, total usage by this striped blocks should
// be the total of data blocks and parity blocks because
// `getNumBytes` is the total of actual data block size.
return ((getNumBytes() - 1) / (dataBlockNum * BLOCK_STRIPED_CELL_SIZE) + 1)
* BLOCK_STRIPED_CELL_SIZE * parityBlockNum + getNumBytes();
}
{code}
The file level logic should still be in {{FSImageLoader}}.
# Javadoc format issue:
{code}
+ * @param f
+ * inode file
+ * @return file size
{code}
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[ https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485786#comment-14485786 ] Rakesh R commented on HDFS-7949: Thanks a lot [~zhz] for the comments. Updated new patch addressing the same. Please have a look when you get a chance. WebImageViewer need support file size calculation with striped blocks - Key: HDFS-7949 URL: https://issues.apache.org/jira/browse/HDFS-7949 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Hui Zheng Assignee: Rakesh R Priority: Minor Attachments: HDFS-7949-001.patch, HDFS-7949-002.patch, HDFS-7949-003.patch The file size calculation should be changed when the blocks of the file are striped in WebImageViewer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14481067#comment-14481067
]
Rakesh R commented on HDFS-7949:
Thats good point. I agree exposing a static function to compute this. This
function takes an argument
{{org.apache.hadoop.hdfs.server.namenode.FsImageProto.INodeSection.INodeFile}},
I hope there won't be any concern of introducing module/package dependency
bewteen {{BlockInfo}} and {{FsImageProto}}.
bq.Could you also add a test for the change?
Sure, I'll add unit test case.
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
Attachments: HDFS-7949-001.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14394848#comment-14394848
]
Zhe Zhang commented on HDFS-7949:
-
To maintain a consistent way of calculating this, I think we should add a
static method. E.g., in {{BlockInfo}}. Let me know if you agree.
Could you also add a test for the change? Thanks!
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
Attachments: HDFS-7949-001.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14394421#comment-14394421
]
Rakesh R commented on HDFS-7949:
Thanks [~zhz] for the interest. Yes, this is same as {{#spaceConsumed}}
computation. Does this sound good?
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
Attachments: HDFS-7949-001.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14394100#comment-14394100
]
Zhe Zhang commented on HDFS-7949:
-
Thanks Rakesh for working on this!
The below calculation looks quite complex. Could you explain a little more? Is
it consistent with how file size is calculated in other places?
{code}
+size += ((p.getBlock().getNumBytes() - 1)
+/ (p.getDataBlockNum() * chunkSize) + 1)
+* chunkSize * p.getParityBlockNum() + p.getBlock().getNumBytes();
{code}
Would also be better to add a simple test for the change.
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
Attachments: HDFS-7949-001.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14394104#comment-14394104
]
Zhe Zhang commented on HDFS-7949:
-
Sorry I just note the calculation is from {{BlockInfoStriped#spaceConsumed}}.
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
Attachments: HDFS-7949-001.patch
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14385703#comment-14385703
]
Rakesh R commented on HDFS-7949:
I'd like to take this ahead. Below is my initial thought and could you please
have a look at this when you get a chance. Thanks!
For the file size computation we should consider both data blocks and partity
blocks (for a m + k striped block i.e., m data blocks and k parity blocks).
Since our current strip {{chunkSize}} is fixed (64KB), we should be able to
compute the file size on m, k, and numBytes. I think, we can do the computation
similar to the {{spaceConsumed}} logic. Please correct me if I'm missing
anything.
{code}
for (StripedBlockProto p : f.getStripedBlocks().getBlocksList()) {
size += ((p.getBlock().getNumBytes() - 1) / (p.getDataBlockNum() *
chunkSize) + 1)
* chunkSize * p.getParityBlockNum() + p.getBlock().getNumBytes();
}
{code}
WebImageViewer need support file size calculation with striped blocks
-
Key: HDFS-7949
URL: https://issues.apache.org/jira/browse/HDFS-7949
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
Assignee: Rakesh R
Priority: Minor
The file size calculation should be changed when the blocks of the file are
striped in WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
