[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[ https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17459024#comment-17459024 ] Yao Lei commented on HDFS-11194: Hi, [~arp] Is this feature availble in release 3.1.0 ? I can't find SlowNodeDetector.java in 3.1.0 source code. > Maintain aggregated peer performance metrics on NameNode > > > Key: HDFS-11194 > URL: https://issues.apache.org/jira/browse/HDFS-11194 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Reporter: Xiaobing Zhou >Assignee: Arpit Agarwal >Priority: Major > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: HDFS-11194-03-04.delta, HDFS-11194.01.patch, > HDFS-11194.02.patch, HDFS-11194.03.patch, HDFS-11194.04.patch, > HDFS-11194.05.patch, HDFS-11194.06.patch > > > The metrics collected in HDFS-10917 should be reported to and aggregated on > NameNode as part of heart beat messages. This will make is easy to expose it > through JMX to users who are interested in them. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[ https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332689#comment-16332689 ] Arpit Agarwal commented on HDFS-11194: -- There's some dependent Jiras also, the back-porting work may be complex. > Maintain aggregated peer performance metrics on NameNode > > > Key: HDFS-11194 > URL: https://issues.apache.org/jira/browse/HDFS-11194 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Reporter: Xiaobing Zhou >Assignee: Arpit Agarwal >Priority: Major > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: HDFS-11194-03-04.delta, HDFS-11194.01.patch, > HDFS-11194.02.patch, HDFS-11194.03.patch, HDFS-11194.04.patch, > HDFS-11194.05.patch, HDFS-11194.06.patch > > > The metrics collected in HDFS-10917 should be reported to and aggregated on > NameNode as part of heart beat messages. This will make is easy to expose it > through JMX to users who are interested in them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[ https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332099#comment-16332099 ] Brahma Reddy Battula commented on HDFS-11194: - Nice work!! Good to have in branch-2.8 and branch-2.7 too (even HDFS-10917)?. More useful for debugging in large cluster. cc to [~shv]/[~arpitagarwal] > Maintain aggregated peer performance metrics on NameNode > > > Key: HDFS-11194 > URL: https://issues.apache.org/jira/browse/HDFS-11194 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Reporter: Xiaobing Zhou >Assignee: Arpit Agarwal >Priority: Major > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: HDFS-11194-03-04.delta, HDFS-11194.01.patch, > HDFS-11194.02.patch, HDFS-11194.03.patch, HDFS-11194.04.patch, > HDFS-11194.05.patch, HDFS-11194.06.patch > > > The metrics collected in HDFS-10917 should be reported to and aggregated on > NameNode as part of heart beat messages. This will make is easy to expose it > through JMX to users who are interested in them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[ https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326124#comment-16326124 ] Leo Chen commented on HDFS-11194: - thanks,[~hanishakoneru] :D > Maintain aggregated peer performance metrics on NameNode > > > Key: HDFS-11194 > URL: https://issues.apache.org/jira/browse/HDFS-11194 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Reporter: Xiaobing Zhou >Assignee: Arpit Agarwal >Priority: Major > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: HDFS-11194-03-04.delta, HDFS-11194.01.patch, > HDFS-11194.02.patch, HDFS-11194.03.patch, HDFS-11194.04.patch, > HDFS-11194.05.patch, HDFS-11194.06.patch > > > The metrics collected in HDFS-10917 should be reported to and aggregated on > NameNode as part of heart beat messages. This will make is easy to expose it > through JMX to users who are interested in them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[
https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322782#comment-16322782
]
Hanisha Koneru commented on HDFS-11194:
---
[~Leo Chen], in the Namenode JMX, the metric is {{SlowPeersReport}}.
> Maintain aggregated peer performance metrics on NameNode
>
>
> Key: HDFS-11194
> URL: https://issues.apache.org/jira/browse/HDFS-11194
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: namenode
>Reporter: Xiaobing Zhou
>Assignee: Arpit Agarwal
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HDFS-11194-03-04.delta, HDFS-11194.01.patch,
> HDFS-11194.02.patch, HDFS-11194.03.patch, HDFS-11194.04.patch,
> HDFS-11194.05.patch, HDFS-11194.06.patch
>
>
> The metrics collected in HDFS-10917 should be reported to and aggregated on
> NameNode as part of heart beat messages. This will make is easy to expose it
> through JMX to users who are interested in them.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[ https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319891#comment-16319891 ] Leo Chen commented on HDFS-11194: - hi Arpit, what's metrics name in namenode jmx info that user can search? > Maintain aggregated peer performance metrics on NameNode > > > Key: HDFS-11194 > URL: https://issues.apache.org/jira/browse/HDFS-11194 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Reporter: Xiaobing Zhou >Assignee: Arpit Agarwal > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: HDFS-11194-03-04.delta, HDFS-11194.01.patch, > HDFS-11194.02.patch, HDFS-11194.03.patch, HDFS-11194.04.patch, > HDFS-11194.05.patch, HDFS-11194.06.patch > > > The metrics collected in HDFS-10917 should be reported to and aggregated on > NameNode as part of heart beat messages. This will make is easy to expose it > through JMX to users who are interested in them. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[ https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843597#comment-15843597 ] Andrew Wang commented on HDFS-11194: Great, thanks Arpit! > Maintain aggregated peer performance metrics on NameNode > > > Key: HDFS-11194 > URL: https://issues.apache.org/jira/browse/HDFS-11194 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Reporter: Xiaobing Zhou >Assignee: Arpit Agarwal > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: HDFS-11194.01.patch, HDFS-11194.02.patch, > HDFS-11194-03-04.delta, HDFS-11194.03.patch, HDFS-11194.04.patch, > HDFS-11194.05.patch, HDFS-11194.06.patch > > > The metrics collected in HDFS-10917 should be reported to and aggregated on > NameNode as part of heart beat messages. This will make is easy to expose it > through JMX to users who are interested in them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[ https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843026#comment-15843026 ] Arpit Agarwal commented on HDFS-11194: -- We'll have a release note and docs out in time for alpha3. > Maintain aggregated peer performance metrics on NameNode > > > Key: HDFS-11194 > URL: https://issues.apache.org/jira/browse/HDFS-11194 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Xiaobing Zhou >Assignee: Arpit Agarwal > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: HDFS-11194.01.patch, HDFS-11194.02.patch, > HDFS-11194-03-04.delta, HDFS-11194.03.patch, HDFS-11194.04.patch, > HDFS-11194.05.patch, HDFS-11194.06.patch > > > The metrics collected in HDFS-10917 should be reported to and aggregated on > NameNode as part of heart beat messages. This will make is easy to expose it > through JMX to users who are interested in them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[ https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840607#comment-15840607 ] Andrew Wang commented on HDFS-11194: This is a really cool feature. Would someone mind adding a release note on how to configure and view these new metrics? > Maintain aggregated peer performance metrics on NameNode > > > Key: HDFS-11194 > URL: https://issues.apache.org/jira/browse/HDFS-11194 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Xiaobing Zhou >Assignee: Arpit Agarwal > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: HDFS-11194.01.patch, HDFS-11194.02.patch, > HDFS-11194-03-04.delta, HDFS-11194.03.patch, HDFS-11194.04.patch, > HDFS-11194.05.patch, HDFS-11194.06.patch > > > The metrics collected in HDFS-10917 should be reported to and aggregated on > NameNode as part of heart beat messages. This will make is easy to expose it > through JMX to users who are interested in them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[ https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836997#comment-15836997 ] Hudson commented on HDFS-11194: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11168 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11168/]) HDFS-11194. Maintain aggregated peer performance metrics on NameNode. (arp: rev b57368b6f893cb27d77fc9425e116f1312f4790f) * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/SlowNodeDetector.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBpServiceActorScheduler.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestStorageReport.java * (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/protocol/SlowPeerReports.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestFsDatasetCache.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeLifeline.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SlowPeerTracker.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestRollingAverages.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/metrics/TestDataNodeOutlierDetectionViaMetrics.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/metrics/TestSlowNodeDetector.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/RollingAverages.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtocol.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodePeerMetrics.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodePeerMetrics.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDatanodeProtocolRetryPolicy.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/InternalDataNodeTestUtils.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNameNodePrunesMissingStorages.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeStatusMXBean.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestHdfsConfigFields.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java * (edit)
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[
https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836925#comment-15836925
]
Hadoop QA commented on HDFS-11194:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
44s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 20 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 10m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
36s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 56s{color} | {color:orange} root: The patch generated 9 new + 1666 unchanged
- 5 fixed = 1675 total (was 1671) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
41s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
8s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 98m 36s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}180m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.server.namenode.ha.TestHAAppend |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11194 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12849166/HDFS-11194.06.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc xml |
| uname | Linux d9da0c59cf56 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[ https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836624#comment-15836624 ] Xiaobing Zhou commented on HDFS-11194: -- Thanks for v5 patch. [~arpitagarwal]. Having some minor comments, otherwise LGTM. # there is minor diff of 'S' in AVERAGE DFSConfigKeys.DFS_METRICS_ROLLING_AVERAGE_WINDOW_LENGTH_KEY DFSConfigKeys.DFS_METRICS_ROLLING_AVERAGES_WINDOW_LENGTH_DEFAULT # some check style issues > Maintain aggregated peer performance metrics on NameNode > > > Key: HDFS-11194 > URL: https://issues.apache.org/jira/browse/HDFS-11194 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.8.0 >Reporter: Xiaobing Zhou >Assignee: Arpit Agarwal > Attachments: HDFS-11194.01.patch, HDFS-11194.02.patch, > HDFS-11194-03-04.delta, HDFS-11194.03.patch, HDFS-11194.04.patch, > HDFS-11194.05.patch > > > The metrics collected in HDFS-10917 should be reported to and aggregated on > NameNode as part of heart beat messages. This will make is easy to expose it > through JMX to users who are interested in them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[ https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813493#comment-15813493 ] Xiaobing Zhou commented on HDFS-11194: -- Thank you [~arpitagarwal] for the patch. I've some comments. # RollingAverages#getStats is using non-rolling mean states, it should use rolling ones instead. See RollingAverages#snapshot for calculation of rolling averages. # DFS_METRICS_ROLLING_AVERAGES_WINDOW_SIZE_DEFAULT should be changed to the same naming with DFS_METRICS_ROLLING_AVERAGE_WINDOW_LENGTH_KEY # These parameters can be changed to be configurable. SlowNodeDetector#minOutlierDetectionPeers DataNodePeerMetrics#LOW_THRESHOLD_MS SlowPeerTracker#MAX_NODES_TO_REPORT # In BlockReceiver#receivePacket, after trackSendPacketToLastNodeInPipeline(duration); It may need to change a bit, e.g. if (duration > DataNodePeerMetrics#LOW_THRESHOLD_MS). or remove the warning msg at all. > Maintain aggregated peer performance metrics on NameNode > > > Key: HDFS-11194 > URL: https://issues.apache.org/jira/browse/HDFS-11194 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.8.0 >Reporter: Xiaobing Zhou >Assignee: Arpit Agarwal > Attachments: HDFS-11194-03-04.delta, HDFS-11194.01.patch, > HDFS-11194.02.patch, HDFS-11194.03.patch, HDFS-11194.04.patch > > > The metrics collected in HDFS-10917 should be reported to and aggregated on > NameNode as part of heart beat messages. This will make is easy to expose it > through JMX to users who are interested in them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[
https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15803341#comment-15803341
]
Hadoop QA commented on HDFS-11194:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 20 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 9m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 53s{color} | {color:orange} root: The patch generated 7 new + 1666 unchanged
- 5 fixed = 1673 total (was 1671) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
27s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
0s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 45s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}154m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11194 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12845901/HDFS-11194.04.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc xml |
| uname | Linux d2dc9ae0acbf 3.13.0-106-generic #153-Ubuntu SMP
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[ https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15803009#comment-15803009 ] Arpit Agarwal commented on HDFS-11194: -- HDFS-11194-03-04.delta has the changes from the v03 -> v04 patch for reviewing. > Maintain aggregated peer performance metrics on NameNode > > > Key: HDFS-11194 > URL: https://issues.apache.org/jira/browse/HDFS-11194 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.8.0 >Reporter: Xiaobing Zhou >Assignee: Arpit Agarwal > Attachments: HDFS-11194-03-04.delta, HDFS-11194.01.patch, > HDFS-11194.02.patch, HDFS-11194.03.patch, HDFS-11194.04.patch > > > The metrics collected in HDFS-10917 should be reported to and aggregated on > NameNode as part of heart beat messages. This will make is easy to expose it > through JMX to users who are interested in them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[
https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796899#comment-15796899
]
Xiaoyu Yao commented on HDFS-11194:
---
TestHeartbeatHandling.java
Line 60: is the 300_000 a typo or special usage of timeout rule?
{code}
public Timeout testTimeout = new Timeout(300_000);
{code}
TestSlowPeerTracker.java
Line 54: same as above.
> Maintain aggregated peer performance metrics on NameNode
>
>
> Key: HDFS-11194
> URL: https://issues.apache.org/jira/browse/HDFS-11194
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Xiaobing Zhou
>Assignee: Arpit Agarwal
> Attachments: HDFS-11194.01.patch, HDFS-11194.02.patch,
> HDFS-11194.03.patch
>
>
> The metrics collected in HDFS-10917 should be reported to and aggregated on
> NameNode as part of heart beat messages. This will make is easy to expose it
> through JMX to users who are interested in them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[ https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796663#comment-15796663 ] Xiaoyu Yao commented on HDFS-11194: --- Thanks [~arpitagarwal] for working on this and all for the discussion. I have the following comments on the production side changes. Still reviewing the unit test changes and will post my comments on that soon. 1. BlockReceiver.java NIT: Line 848: "&& mirrorAddr != null" can be removed. Line 849: can be simplified with "peerMetrics.addSendPacketDownstream" 2. BPServiceActor.java Line 1146: NIT: heatbeatTime can be changed to slowPeerReportTime or remove the parameter by hiding the montonicNow() call inside scheduleNextSlowPeerReport(). 3. DatanodeManager.java Line 52-53: NIT: avoid import * import org.apache.hadoop.util.*; import org.apache.hadoop.util.Timer; Line 180. The comments seems incomplete. Line 212. we should instantiate slowPeerTracker only if dataNodePeerStatsEnabled is true. Line 1653-1660: NIT: can we tweak the code to avoid calling slowPeers.getSlowPeers() multiple times in the worst case and maybe avoid the if (LOG.isDebugEnabled()) with parameterized logging? Line 1659: can we use nodeinfo.getIpcAddr() sicne the datanode has registered? 4. DataNodePeerMetrics.java Line 142-143: Correct me if I'm wrong, looks like the comments is for stats Map in Line 137. 5. DatanodeProtocol.proto Line 398-405. This is a very good document. Can we add a field indicating the DN aggregate mechanism? This way the NN can enforce consistent aggregation across all the datanodes. This can be done in a separate ticket. 6. DFSConfigKeys.java Line 677: document for dfs.datanode.slow.peers.report.interval? We can open separate ticket for it. 7. RollingAverage.java Great catch on some missing synchronized on rollOverAvgs. NIT: Line 264: missing @param for minSamples 8. SlowNodeDetector.java Line 99-108: We can make this an interface to allow different aggregation methods (median, 90th percentile) for outlier detection. This can be done in a separate ticket. We can also use Median/Percentile class from apache common to implement different aggregation. Line 127: we need to guard the tracing with if (LOG.isTraceEnabled()) to avoid the implicit sorted.toString() overhead. 9. SlowPeerReports.java Line 44: NIT: typo consistenly -> consistently Line 144: NIT: the document needs to update to match the code which returns a map -> sortedset of string. Line 190: Can we make MAX_NODES_TO_REPORT configurable? This can be fixed in a separate ticket. > Maintain aggregated peer performance metrics on NameNode > > > Key: HDFS-11194 > URL: https://issues.apache.org/jira/browse/HDFS-11194 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.8.0 >Reporter: Xiaobing Zhou >Assignee: Arpit Agarwal > Attachments: HDFS-11194.01.patch, HDFS-11194.02.patch, > HDFS-11194.03.patch > > > The metrics collected in HDFS-10917 should be reported to and aggregated on > NameNode as part of heart beat messages. This will make is easy to expose it > through JMX to users who are interested in them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[
https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15786590#comment-15786590
]
Hadoop QA commented on HDFS-11194:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 20 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 9m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
18s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 21s{color} | {color:orange} root: The patch generated 7 new + 1666 unchanged
- 5 fixed = 1673 total (was 1671) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 8 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
27s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
0s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 62m
30s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}135m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11194 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12845117/HDFS-11194.03.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc xml |
| uname | Linux 20947df75f90 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 95c2c24 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[
https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15786317#comment-15786317
]
Hadoop QA commented on HDFS-11194:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 20 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 9m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 9m 10s{color}
| {color:red} root generated 1 new + 690 unchanged - 0 fixed = 691 total (was
690) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 54s{color} | {color:orange} root: The patch generated 12 new + 1666
unchanged - 5 fixed = 1678 total (was 1671) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 8 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
25s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
1s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 55s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeUUID |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11194 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12845104/HDFS-11194.02.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc xml |
| uname | Linux 190b84f76e01 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[
https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784508#comment-15784508
]
Hadoop QA commented on HDFS-11194:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 18 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 9m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 9m 34s{color}
| {color:red} root generated 1 new + 690 unchanged - 0 fixed = 691 total (was
690) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 52s{color} | {color:orange} root: The patch generated 22 new + 1604
unchanged - 5 fixed = 1626 total (was 1609) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 19 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
38s{color} | {color:red} hadoop-common-project/hadoop-common generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
44s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 2 new
+ 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
4s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
48s{color} | {color:red} hadoop-common-project_hadoop-common generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 9s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
1s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 47s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-common |
| | Inconsistent synchronization of
org.apache.hadoop.metrics2.lib.RollingAverages.currentSnapshot; locked 57% of
time
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[ https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747513#comment-15747513 ] Kai Zheng commented on HDFS-11194: -- For the trade-off, would it be good to put such new metrics into off-heap? We have lots of discussions like this but it's hard for existing inode things so blocked. Maybe it's good for new memory consuming data structures in NameNode. > Maintain aggregated peer performance metrics on NameNode > > > Key: HDFS-11194 > URL: https://issues.apache.org/jira/browse/HDFS-11194 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.8.0 >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > > The metrics collected in HDFS-10917 should be reported to and aggregated on > NameNode as part of heart beat messages. This will make is easy to expose it > through JMX to users who are interested in them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11194) Maintain aggregated peer performance metrics on NameNode
[ https://issues.apache.org/jira/browse/HDFS-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15740192#comment-15740192 ] Andrew Purtell commented on HDFS-11194: --- It would be an interesting exercise to make a back of the envelope calculation of how much online storage would be required for the aggregate metrics of say 2000 DataNodes. The patch on HDFS-10917 adds 6 MutableQuantiles. Would it be just these or all 60 or so metrics in DataNodeMetrics.java ? Assume each metric is a simple long counter for the sake of argument. That would require 8 * 60 * 2000 = ~1 MB of storage. Of course there will be data structure overheads, the quantile metrics are not single longs, etc. As an operator I can report that maintaining the stability of NameNodes (2.7.x) with respect to avoiding full GC or the Linux OOM killer is a nontrivial exercise. Perhaps this feature if implemented could be made optional, but it would be better to take an approach like Accumulo and implement a separate metrics aggregation service for the task that can be comfortably run elsewhere than the NN. (See https://accumulo.apache.org/1.7/accumulo_user_manual#_monitor) > Maintain aggregated peer performance metrics on NameNode > > > Key: HDFS-11194 > URL: https://issues.apache.org/jira/browse/HDFS-11194 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.8.0 >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > > The metrics collected in HDFS-10917 should be reported to and aggregated on > NameNode as part of heart beat messages. This will make is easy to expose it > through JMX to users who are interested in them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
