[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338429#comment-14338429
]
Hudson commented on HDFS-7467:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2048 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2048/])
HDFS-7467. Provide storage tier information for a directory via fsck. (Benoy
Antony) (benoy: rev d140d76a43c88e326b9c2818578f22bd3563b969)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySummary.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySummary.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Fix For: 2.7.0
Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
HDFS-7467-004.patch, HDFS-7467.patch, storagepolicydisplay.pdf
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338549#comment-14338549
]
Hudson commented on HDFS-7467:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2066 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2066/])
HDFS-7467. Provide storage tier information for a directory via fsck. (Benoy
Antony) (benoy: rev d140d76a43c88e326b9c2818578f22bd3563b969)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySummary.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySummary.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Fix For: 2.7.0
Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
HDFS-7467-004.patch, HDFS-7467.patch, storagepolicydisplay.pdf
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338510#comment-14338510
]
Hudson commented on HDFS-7467:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #116 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/116/])
HDFS-7467. Provide storage tier information for a directory via fsck. (Benoy
Antony) (benoy: rev d140d76a43c88e326b9c2818578f22bd3563b969)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySummary.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySummary.java
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Fix For: 2.7.0
Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
HDFS-7467-004.patch, HDFS-7467.patch, storagepolicydisplay.pdf
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338446#comment-14338446
]
Hudson commented on HDFS-7467:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #107 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/107/])
HDFS-7467. Provide storage tier information for a directory via fsck. (Benoy
Antony) (benoy: rev d140d76a43c88e326b9c2818578f22bd3563b969)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySummary.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySummary.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Fix For: 2.7.0
Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
HDFS-7467-004.patch, HDFS-7467.patch, storagepolicydisplay.pdf
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338257#comment-14338257
]
Hudson commented on HDFS-7467:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #116 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/116/])
HDFS-7467. Provide storage tier information for a directory via fsck. (Benoy
Antony) (benoy: rev d140d76a43c88e326b9c2818578f22bd3563b969)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySummary.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySummary.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Fix For: 2.7.0
Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
HDFS-7467-004.patch, HDFS-7467.patch, storagepolicydisplay.pdf
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338283#comment-14338283
]
Hudson commented on HDFS-7467:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #850 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/850/])
HDFS-7467. Provide storage tier information for a directory via fsck. (Benoy
Antony) (benoy: rev d140d76a43c88e326b9c2818578f22bd3563b969)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySummary.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySummary.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Fix For: 2.7.0
Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
HDFS-7467-004.patch, HDFS-7467.patch, storagepolicydisplay.pdf
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14337573#comment-14337573
]
Hudson commented on HDFS-7467:
--
FAILURE: Integrated in Hadoop-trunk-Commit #7201 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7201/])
HDFS-7467. Provide storage tier information for a directory via fsck. (Benoy
Antony) (benoy: rev d140d76a43c88e326b9c2818578f22bd3563b969)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStoragePolicySummary.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySummary.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
HDFS-7467-004.patch, HDFS-7467.patch, storagepolicydisplay.pdf
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14337588#comment-14337588
]
Benoy Antony commented on HDFS-7467:
committed to trunk and branch-2.
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Fix For: 2.7.0
Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
HDFS-7467-004.patch, HDFS-7467.patch, storagepolicydisplay.pdf
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335606#comment-14335606
]
Benoy Antony commented on HDFS-7467:
Thanks for the review [~szetszwo].
If there are no further comments, I'll commit the patch tomorrow.
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
HDFS-7467-004.patch, HDFS-7467.patch, storagepolicydisplay.pdf
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14333138#comment-14333138
]
Tsz Wo Nicholas Sze commented on HDFS-7467:
---
The patch looks good. Just some minor comments:
- In StoragePolicySummary.process(), when allBlocksFollowPolicy is set to
false, it can return immediately.
- Need to fix StorageType import. It was moved to fs.StorageType.
- Add @Overreide tag for hashCode(), equals(Object another) and toString().
- StorageTypeAllocation.getStorageAllocationAsString(MapStorageType, Integer
storageType_countmap) should be static.
- The indentation for StorageTypeAllocation.getStorageAllocationAsString() is
off. The for-loop is indented too deep.
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
HDFS-7467.patch, storagepolicydisplay.pdf
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14334378#comment-14334378
]
Hadoop QA commented on HDFS-7467:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12700303/HDFS-7467-004.patch
against trunk revision 16bd79e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9648//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9648//console
This message is automatically generated.
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
HDFS-7467-004.patch, HDFS-7467.patch, storagepolicydisplay.pdf
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312484#comment-14312484
]
Benoy Antony commented on HDFS-7467:
The findbugs warning and test failure are unrelated.
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
HDFS-7467.patch, storagepolicydisplay.pdf
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312185#comment-14312185
]
Hadoop QA commented on HDFS-7467:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12697366/HDFS-7467-003.patch
against trunk revision 7e42088.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.balancer.TestBalancer
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9499//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9499//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9499//console
This message is automatically generated.
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
HDFS-7467.patch, storagepolicydisplay.pdf
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311128#comment-14311128
]
Hadoop QA commented on HDFS-7467:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12697294/HDFS-7467-002.patch
against trunk revision 350b520.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9490//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9490//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9490//console
This message is automatically generated.
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: HDFS-7467-002.patch, HDFS-7467.patch,
storagepolicydisplay.pdf
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268598#comment-14268598
]
Benoy Antony commented on HDFS-7467:
{quote}
If a file does not satisfies the specified policy, fsck should show such
information.
{quote}
That's good for a file. What do we do for directories which can potentially
have children with different policies ?
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: HDFS-7467.patch
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268661#comment-14268661
]
Tsz Wo Nicholas Sze commented on HDFS-7467:
---
I think the fsck output can have two sections, one for the blocks that
satisfying the specified policy and one for the others. E.g.
{noformat}
Blocks satisfy the specified policy:
Storage Policy # of blocks % of blocks
hot(DISK:3) 340730 97.7393%
hot(DISK:4) 39281.1268%
frozen(ARCHIVE:3) 31220.8956%
Blocks DO NOT satisfy the specified policy:
Storage Policy # of blocks % of blocks
DISK:3 440.0126%
DISK:1,ARCHIVE:2300.0086%
ARCHIVE:390.0026%
{noformat}
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: HDFS-7467.patch
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14266871#comment-14266871
]
Benoy Antony commented on HDFS-7467:
1.
{quote}
Are all storage policies in fallback storage equivalent to other storage
policies that this output can always be fully described by the percentages that
Tsz has suggested?
{quote}
There is a possibility that some storage tier combination may not belong to a
storage policy.
My recommendation is to display the policy along with the combination if
possible. If not, display the combination. Lowercase for policy name is
intentional.
{code}
Storage Policy # of blocks % of blocks
cold (DISK:1,ARCHIVE:2) 340730 97.7393%
frozen (ARCHIVE:3) 39281.1268%
DISK:2,ARCHIVE:231220.8956%
warm (DISK:2,ARCHIVE:1) 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
2.
{quote}
There should also be some warning messages as well in fsck for all files that
are unable to meet the requested ideal for their storage policy and are using
fallback storage, perhaps with a switch since that could become overly volumous
output.
{quote}
This is a nice feature. Will look into that .
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: HDFS-7467.patch
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267222#comment-14267222
]
Tsz Wo Nicholas Sze commented on HDFS-7467:
---
My recommendation is to display the policy along with the combination if
possible. ...
It is a good idea. We should also consider a file's specified storage policy
and actually storage media. If a file does not satisfies the specified policy,
fsck should show such information. E.g. the specified storage policy of file
foo is hot but all the replicas are stored in ARCHIVE, then it should not be
counted as frozen. It should be counted as ARCHIVE:3 in order to indicate
that it does not satisfies the specified policy.
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: HDFS-7467.patch
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258172#comment-14258172
]
Hari Sekhon commented on HDFS-7467:
---
There does need to be a way to figure out if a given file or directory of files
are using fallback storage.
There should also be a global method of seeing if any files are using fallback
storage as an indicator that there isn't enough SSD for example.
Adding this information to fsck seems like a sensible way to go - the main
question is how to represent that information concisely.
Are all storage policies in fallback storage equivalent to other storage
policies that this output can always be fully described by the percentages that
Tsz has suggested?
There should also be some warning messages as well in fsck for all files that
are unable to meet the requested ideal for their storage policy and are using
fallback storage, perhaps with a switch since that could become overly volumous
output.
Regards,
Hari Sekhon
http://www.linkedin.com/in/harisekhon
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: HDFS-7467.patch
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252571#comment-14252571
]
Tsz Wo Nicholas Sze commented on HDFS-7467:
---
Hi Benoy, I think it may be better to show storage policies than storage
combinations, i.e.
{noformat}
Storage Policy # of blocks % of blocks
HOT340730 97.7393%
WARM 39281.1268%
COLD 31220.8956%
{noformat}
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Attachments: HDFS-7467.patch
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14240268#comment-14240268
]
Benoy Antony commented on HDFS-7467:
Could I please get a feedback on this whether it makes sense to add this
information as part of fsck optionally?
Provide storage tier information for a directory via fsck
-
Key: HDFS-7467
URL: https://issues.apache.org/jira/browse/HDFS-7467
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Currently _fsck_ provides information regarding blocks for a directory.
It should be augmented to provide storage tier information (optionally).
The sample report could be as follows :
{code}
Storage Tier Combination# of blocks % of blocks
DISK:1,ARCHIVE:2 340730 97.7393%
ARCHIVE:3 39281.1268%
DISK:2,ARCHIVE:231220.8956%
DISK:2,ARCHIVE:1 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
