[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity
[ https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085445#comment-15085445 ] Rajat Jain commented on HDFS-9279: -- While it makes sense to not include decommissioning nodes in configured capacity, but they should still be used for calculating used capacity. Because the data present in the decommissioning nodes would eventually be transferred over to the live nodes. Is this understanding correct? > Decomissioned capacity should not be considered for configured/used capacity > > > Key: HDFS-9279 > URL: https://issues.apache.org/jira/browse/HDFS-9279 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Fix For: 3.0.0, 2.8.0 > > Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, > HDFS-9279-v3.patch, HDFS-9279-v4.patch > > > Capacity of a decommissioned node is being accounted as configured and used > capacity metrics. This gives incorrect perception of cluster usage. > Once a node is decommissioned, its capacity should be considered similar to a > dead node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity
[ https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085577#comment-15085577 ] Kihwal Lee commented on HDFS-9279: -- bq. Because the data present in the decommissioning nodes would eventually be transferred over to the live nodes. Is this understanding correct? The replicas are not invalidated on decommissioning nodes even after replicating, so the capacity tracking was not accurate either. It ended up double counting the used space toward the end, at which the process seems to stall more frequently nowadays (this is another topic). If a significant portion of a cluster is decommissioned, the stat will look very strange and confuse people. That actually happened to us multiple times. The free/total ratio will look considerably smaller than the actual value. Monitoring tools cannot easily dismiss it as 'Nah.. it's a temporary discrepancy caused by decommissioning.' With this change, the storage capacity stat has become more like regular under-replication scenario caused by node/disk outages. Additional space will be used for re-replicating those blocks, but it is not yet allocated to those blocks. That's the actual state of used/usable storage and the stat reflects that now. If we want the stat to reflect what would be used in the future, we are talking space reservation feature. > Decomissioned capacity should not be considered for configured/used capacity > > > Key: HDFS-9279 > URL: https://issues.apache.org/jira/browse/HDFS-9279 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Fix For: 3.0.0, 2.8.0 > > Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, > HDFS-9279-v3.patch, HDFS-9279-v4.patch > > > Capacity of a decommissioned node is being accounted as configured and used > capacity metrics. This gives incorrect perception of cluster usage. > Once a node is decommissioned, its capacity should be considered similar to a > dead node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity
[ https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978664#comment-14978664 ] Kihwal Lee commented on HDFS-9279: -- The jira was originally to exclude "live" decommissioned nodes from stat, but it seems decommissioning nodes are excluded as well. > Decomissioned capacity should not be considered for configured/used capacity > > > Key: HDFS-9279 > URL: https://issues.apache.org/jira/browse/HDFS-9279 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, > HDFS-9279-v3.patch, HDFS-9279-v4.patch > > > Capacity of a decommissioned node is being accounted as configured and used > capacity metrics. This gives incorrect perception of cluster usage. > Once a node is decommissioned, its capacity should be considered similar to a > dead node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity
[ https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978744#comment-14978744 ] Kihwal Lee commented on HDFS-9279: -- +1 the patch looks good. > Decomissioned capacity should not be considered for configured/used capacity > > > Key: HDFS-9279 > URL: https://issues.apache.org/jira/browse/HDFS-9279 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, > HDFS-9279-v3.patch, HDFS-9279-v4.patch > > > Capacity of a decommissioned node is being accounted as configured and used > capacity metrics. This gives incorrect perception of cluster usage. > Once a node is decommissioned, its capacity should be considered similar to a > dead node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity
[ https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978708#comment-14978708 ] Kihwal Lee commented on HDFS-9279: -- Actually, I think this makes more sense. Assuming roughly even distribution of data, excluding everything from dicomm'ing nodes makes the used/total unchanged initially. As blocks are replicated used/total will increase. I think this tracks the reality better. > Decomissioned capacity should not be considered for configured/used capacity > > > Key: HDFS-9279 > URL: https://issues.apache.org/jira/browse/HDFS-9279 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, > HDFS-9279-v3.patch, HDFS-9279-v4.patch > > > Capacity of a decommissioned node is being accounted as configured and used > capacity metrics. This gives incorrect perception of cluster usage. > Once a node is decommissioned, its capacity should be considered similar to a > dead node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity
[ https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978758#comment-14978758 ] Kihwal Lee commented on HDFS-9279: -- I've committed this to trunk and branch-2. Thank you for working on the issue, Kuhu. > Decomissioned capacity should not be considered for configured/used capacity > > > Key: HDFS-9279 > URL: https://issues.apache.org/jira/browse/HDFS-9279 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Fix For: 3.0.0, 2.8.0 > > Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, > HDFS-9279-v3.patch, HDFS-9279-v4.patch > > > Capacity of a decommissioned node is being accounted as configured and used > capacity metrics. This gives incorrect perception of cluster usage. > Once a node is decommissioned, its capacity should be considered similar to a > dead node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity
[ https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978926#comment-14978926 ] Hudson commented on HDFS-9279: -- FAILURE: Integrated in Hadoop-trunk-Commit #8720 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8720/]) HDFS-9279. Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Decomissioned capacity should not be considered for configured/used capacity > > > Key: HDFS-9279 > URL: https://issues.apache.org/jira/browse/HDFS-9279 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Fix For: 3.0.0, 2.8.0 > > Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, > HDFS-9279-v3.patch, HDFS-9279-v4.patch > > > Capacity of a decommissioned node is being accounted as configured and used > capacity metrics. This gives incorrect perception of cluster usage. > Once a node is decommissioned, its capacity should be considered similar to a > dead node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity
[ https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979002#comment-14979002 ] Hudson commented on HDFS-9279: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #596 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/596/]) HDFS-9279. Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java > Decomissioned capacity should not be considered for configured/used capacity > > > Key: HDFS-9279 > URL: https://issues.apache.org/jira/browse/HDFS-9279 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Fix For: 3.0.0, 2.8.0 > > Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, > HDFS-9279-v3.patch, HDFS-9279-v4.patch > > > Capacity of a decommissioned node is being accounted as configured and used > capacity metrics. This gives incorrect perception of cluster usage. > Once a node is decommissioned, its capacity should be considered similar to a > dead node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity
[ https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979081#comment-14979081 ] Hudson commented on HDFS-9279: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #609 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/609/]) HDFS-9279. Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java > Decomissioned capacity should not be considered for configured/used capacity > > > Key: HDFS-9279 > URL: https://issues.apache.org/jira/browse/HDFS-9279 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Fix For: 3.0.0, 2.8.0 > > Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, > HDFS-9279-v3.patch, HDFS-9279-v4.patch > > > Capacity of a decommissioned node is being accounted as configured and used > capacity metrics. This gives incorrect perception of cluster usage. > Once a node is decommissioned, its capacity should be considered similar to a > dead node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity
[ https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979100#comment-14979100 ] Hudson commented on HDFS-9279: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1332 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1332/]) HDFS-9279. Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java > Decomissioned capacity should not be considered for configured/used capacity > > > Key: HDFS-9279 > URL: https://issues.apache.org/jira/browse/HDFS-9279 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Fix For: 3.0.0, 2.8.0 > > Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, > HDFS-9279-v3.patch, HDFS-9279-v4.patch > > > Capacity of a decommissioned node is being accounted as configured and used > capacity metrics. This gives incorrect perception of cluster usage. > Once a node is decommissioned, its capacity should be considered similar to a > dead node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity
[ https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979240#comment-14979240 ] Hudson commented on HDFS-9279: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2539 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2539/]) HDFS-9279. Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java > Decomissioned capacity should not be considered for configured/used capacity > > > Key: HDFS-9279 > URL: https://issues.apache.org/jira/browse/HDFS-9279 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Fix For: 3.0.0, 2.8.0 > > Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, > HDFS-9279-v3.patch, HDFS-9279-v4.patch > > > Capacity of a decommissioned node is being accounted as configured and used > capacity metrics. This gives incorrect perception of cluster usage. > Once a node is decommissioned, its capacity should be considered similar to a > dead node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity
[ https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979549#comment-14979549 ] Hudson commented on HDFS-9279: -- ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #547 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/547/]) HDFS-9279. Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Decomissioned capacity should not be considered for configured/used capacity > > > Key: HDFS-9279 > URL: https://issues.apache.org/jira/browse/HDFS-9279 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Fix For: 3.0.0, 2.8.0 > > Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, > HDFS-9279-v3.patch, HDFS-9279-v4.patch > > > Capacity of a decommissioned node is being accounted as configured and used > capacity metrics. This gives incorrect perception of cluster usage. > Once a node is decommissioned, its capacity should be considered similar to a > dead node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity
[ https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979465#comment-14979465 ] Hudson commented on HDFS-9279: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2485 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2485/]) HDFS-9279. Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Decomissioned capacity should not be considered for configured/used capacity > > > Key: HDFS-9279 > URL: https://issues.apache.org/jira/browse/HDFS-9279 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Fix For: 3.0.0, 2.8.0 > > Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, > HDFS-9279-v3.patch, HDFS-9279-v4.patch > > > Capacity of a decommissioned node is being accounted as configured and used > capacity metrics. This gives incorrect perception of cluster usage. > Once a node is decommissioned, its capacity should be considered similar to a > dead node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity
[
https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977236#comment-14977236
]
Hadoop QA commented on HDFS-9279:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 24m 18s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 10m 49s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 13m 43s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 32s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 54s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 59s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 45s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 18s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 4m 13s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 82m 24s | Tests failed in hadoop-hdfs. |
| | | 144m 0s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestAppendSnapshotTruncate |
| | hadoop.hdfs.web.TestWebHdfsUrl |
| | hadoop.hdfs.web.TestWebHdfsTokens |
| | hadoop.hdfs.server.datanode.TestBlockReplacement |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12769047/HDFS-9279-v4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 5c24fe7 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13230/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/13230/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13230/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13230/console |
This message was automatically generated.
> Decomissioned capacity should not be considered for configured/used capacity
>
>
> Key: HDFS-9279
> URL: https://issues.apache.org/jira/browse/HDFS-9279
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch,
> HDFS-9279-v3.patch, HDFS-9279-v4.patch
>
>
> Capacity of a decommissioned node is being accounted as configured and used
> capacity metrics. This gives incorrect perception of cluster usage.
> Once a node is decommissioned, its capacity should be considered similar to a
> dead node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity
[ https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977333#comment-14977333 ] Kuhu Shukla commented on HDFS-9279: --- The latest test failures are unrelated. From a previous failure, TestDataNodeHotSwapVolumes is failing intermittently on trunk(HDFS-9310). Requesting [~kihwal] to review and share comments. Thanks. > Decomissioned capacity should not be considered for configured/used capacity > > > Key: HDFS-9279 > URL: https://issues.apache.org/jira/browse/HDFS-9279 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, > HDFS-9279-v3.patch, HDFS-9279-v4.patch > > > Capacity of a decommissioned node is being accounted as configured and used > capacity metrics. This gives incorrect perception of cluster usage. > Once a node is decommissioned, its capacity should be considered similar to a > dead node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity
[
https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975088#comment-14975088
]
Hadoop QA commented on HDFS-9279:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 23m 4s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 10m 23s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 14m 1s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 38s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 53s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 53s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 43s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 59s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 4m 15s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 64m 57s | Tests failed in hadoop-hdfs. |
| | | 124m 50s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.TestDFSOutputStream |
| | hadoop.hdfs.TestWriteReadStripedFile |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.fs.TestSymlinkHdfsFileContext |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12768762/HDFS-9279-v3.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 67e3d75 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13196/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/13196/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13196/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13196/console |
This message was automatically generated.
> Decomissioned capacity should not be considered for configured/used capacity
>
>
> Key: HDFS-9279
> URL: https://issues.apache.org/jira/browse/HDFS-9279
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch,
> HDFS-9279-v3.patch
>
>
> Capacity of a decommissioned node is being accounted as configured and used
> capacity metrics. This gives incorrect perception of cluster usage.
> Once a node is decommissioned, its capacity should be considered similar to a
> dead node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity
[
https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972309#comment-14972309
]
Hadoop QA commented on HDFS-9279:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 29s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 4s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 35s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 31s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 40s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 33s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 15s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 64m 26s | Tests failed in hadoop-hdfs. |
| | | 108m 34s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestReplaceDatanodeOnFailure |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
| | hadoop.hdfs.TestLeaseRecovery2 |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12768433/HDFS-9279-v2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 5679e46 |
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13176/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/13176/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13176/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13176/console |
This message was automatically generated.
> Decomissioned capacity should not be considered for configured/used capacity
>
>
> Key: HDFS-9279
> URL: https://issues.apache.org/jira/browse/HDFS-9279
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch
>
>
> Capacity of a decommissioned node is being accounted as configured and used
> capacity metrics. This gives incorrect perception of cluster usage.
> Once a node is decommissioned, its capacity should be considered similar to a
> dead node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity
[
https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971986#comment-14971986
]
Hadoop QA commented on HDFS-9279:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12768373/HDFS-9279-v1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 15eb84b |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13169/console |
This message was automatically generated.
> Decomissioned capacity should not be considered for configured/used capacity
>
>
> Key: HDFS-9279
> URL: https://issues.apache.org/jira/browse/HDFS-9279
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: HDFS-9279-v1.patch
>
>
> Capacity of a decommissioned node is being accounted as configured and used
> capacity metrics. This gives incorrect perception of cluster usage.
> Once a node is decommissioned, its capacity should be considered similar to a
> dead node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
