[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855141#comment-16855141 ] Erik Krogen commented on HADOOP-15481: -- Re-updated the fix version to reflect that this was committed to branch-2 > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Fix For: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 3.1.3 > > Attachments: HADOOP-15481-branch-2.003.patch, HADOOP-15481.001.patch, > HADOOP-15481.001.patch, HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16751065#comment-16751065 ] Takanobu Asanuma commented on HADOOP-15481: --- Updated the fix versions. > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3 > > Attachments: HADOOP-15481-branch-2.003.patch, HADOOP-15481.001.patch, > HADOOP-15481.001.patch, HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746624#comment-16746624 ] Chen Liang commented on HADOOP-15481: - Committed to branch 3.0, 3.1 and 3.2. > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15481-branch-2.003.patch, HADOOP-15481.001.patch, > HADOOP-15481.001.patch, HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746603#comment-16746603 ] Chen Liang commented on HADOOP-15481: - [~shv] You are right, I should have done that. Will commit to 3.0, 3.1 and 3.2 shortly. > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15481-branch-2.003.patch, HADOOP-15481.001.patch, > HADOOP-15481.001.patch, HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746589#comment-16746589 ] Konstantin Shvachko commented on HADOOP-15481: -- Hey [~vagarychen], if you commit to branch-2, then you should also commit to all branches above it, that is 3.0, 3.1, 3.2. And then update fix version(s). > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15481-branch-2.003.patch, HADOOP-15481.001.patch, > HADOOP-15481.001.patch, HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745528#comment-16745528 ] Chen Liang commented on HADOOP-15481: - I applied branch-2 patch locally and the test TestFairCallQueue ran successfully, I've committed the patch to branch-2. Thanks for the followup [~cgregori]! > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15481-branch-2.003.patch, HADOOP-15481.001.patch, > HADOOP-15481.001.patch, HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745408#comment-16745408 ] Christopher Gregorian commented on HADOOP-15481: [~vagarychen] Added: let me know if that works! > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15481-branch-2.003.patch, HADOOP-15481.001.patch, > HADOOP-15481.001.patch, HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744428#comment-16744428 ] Chen Liang commented on HADOOP-15481: - [~cgregori] I was trying to commit to branch-2 as well, but seems v003 patch does not apply to branch-2, would you mind providing a patch for branch-2 as well? > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15481.001.patch, HADOOP-15481.001.patch, > HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740830#comment-16740830 ] Chen Liang commented on HADOOP-15481: - I've committed v003 patch to trunk, thanks [~cgregori] for the contribution! > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Attachments: HADOOP-15481.001.patch, HADOOP-15481.001.patch, > HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740831#comment-16740831 ] Hudson commented on HADOOP-15481: - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #15765 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15765/]) HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by (cliang: rev bf08f4abae43d706a305af3f14e00f01c00dba7c) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestFairCallQueue.java * (edit) hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/FairCallQueue.java > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15481.001.patch, HADOOP-15481.001.patch, > HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[
https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740702#comment-16740702
]
Hadoop QA commented on HADOOP-15481:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 32 unchanged - 1 fixed = 32 total (was 33) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 59s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
18s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15481 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954612/HADOOP-15481.003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c0d82c5b1764 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / dddad98 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15776/testReport/ |
| Max. process+thread count | 1447 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15776/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Emit FairCallQueue stats as metrics
>
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740660#comment-16740660 ] Chen Liang commented on HADOOP-15481: - +1 on v003 patch, pending Jenkins. > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Attachments: HADOOP-15481.001.patch, HADOOP-15481.001.patch, > HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740595#comment-16740595 ] Erik Krogen commented on HADOOP-15481: -- Great, LGTM pending Jenkins. +1 (non-binding) [~vagarychen] or [~shv], care to help review & commit this? > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Attachments: HADOOP-15481.001.patch, HADOOP-15481.001.patch, > HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740586#comment-16740586 ] Christopher Gregorian commented on HADOOP-15481: Good call on the naming—updated the test and docs to reflect this as well. > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Attachments: HADOOP-15481.001.patch, HADOOP-15481.001.patch, > HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[
https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740563#comment-16740563
]
Erik Krogen commented on HADOOP-15481:
--
Looking great [~cgregori]! I have a few more minor nits:
* I don't think you need the {{@Metrics}} annotation anymore – the
{{DefaultMetricsSystem.instance().register()}} should take care of this, IIUC.
* In the definition of {{MetricsProxy}}, we usually use indentation like:
(hanging indent of 4 spaces)
{code:java}
private static final class MetricsProxy implements FairCallQueueMXBean,
MetricsSource {
{code}
There's some discussion of this
[here|https://www.oracle.com/technetwork/java/javase/documentation/codeconventions-136091.html],
though we typically use 4 spaces instead of 8.
* For the {{name}}, now you do "FairCallQueue.namespace", I'm thinking that
"namespace.FairCallQueue" might be better to match more closely with how the
configurations look?
* I'm starting to think that names like "FairCallQueueSize_p0" might be
better, to make it more clear what the number represents. Let me know what you
think.
> Emit FairCallQueue stats as metrics
> ---
>
> Key: HADOOP-15481
> URL: https://issues.apache.org/jira/browse/HADOOP-15481
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics, rpc-server
>Reporter: Erik Krogen
>Assignee: Christopher Gregorian
>Priority: Major
> Attachments: HADOOP-15481.001.patch, HADOOP-15481.001.patch,
> HADOOP-15481.002.patch
>
>
> Currently FairCallQueue has some statistics which are exported via JMX: the
> size of each queue, and the number of overflowed calls per queue. These are
> useful statistics to track over time to determine, for example, if queues
> need to be resized. We should emit them via the standard metrics system.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[
https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740013#comment-16740013
]
Hadoop QA commented on HADOOP-15481:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 35s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 32 unchanged - 1 fixed = 32 total (was 33) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 48s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem |
| | hadoop.security.ssl.TestSSLFactory |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15481 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954511/HADOOP-15481.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 02712d70f311 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 33c009a4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15775/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15775/testReport/ |
| Max. process+thread count | 1511 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[
https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740006#comment-16740006
]
Hadoop QA commented on HADOOP-15481:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 32 unchanged - 1 fixed = 32 total (was 33) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 1s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 10s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.ssl.TestSSLFactory |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15481 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954510/HADOOP-15481.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 00ef5086abec 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 33c009a4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15774/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15774/testReport/ |
| Max. process+thread count | 1521 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[
https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739959#comment-16739959
]
Christopher Gregorian commented on HADOOP-15481:
Updated! I think that covers everything, as well as the style suggestions and
docs changes.
Example line output:
{{1547173433343 rpc.FairCallQueue: Context=rpc, namespace=ipc.8020,
Hostname=hostname, FairCallQueueSize_0=0, FairCallQueueOverflowedCalls_0=0,
FairCallQueueSize_1=0, FairCallQueueOverflowedCalls_1=0, FairCallQueueSize_2=0,
FairCallQueueOverflowedCalls_2=0, FairCallQueueSize_3=0,
FairCallQueueOverflowedCalls_3=0}}
Thanks for the review!
> Emit FairCallQueue stats as metrics
> ---
>
> Key: HADOOP-15481
> URL: https://issues.apache.org/jira/browse/HADOOP-15481
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics, rpc-server
>Reporter: Erik Krogen
>Assignee: Christopher Gregorian
>Priority: Major
> Attachments: HADOOP-15481.001.patch, HADOOP-15481.001.patch,
> HADOOP-15481.002.patch
>
>
> Currently FairCallQueue has some statistics which are exported via JMX: the
> size of each queue, and the number of overflowed calls per queue. These are
> useful statistics to track over time to determine, for example, if queues
> need to be resized. We should emit them via the standard metrics system.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[
https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739858#comment-16739858
]
Erik Krogen commented on HADOOP-15481:
--
Thanks for tackling this [~cgregori]! My comments on the v001 patch are below.
* I think the metrics context should be "rpc" rather than "dfs", since FCQ
lives at the IPC layer of the stack and isn't HDFS-specific
* Making this implement {{MetricsSource}} seems like a good move to be able to
easily use the current info available in conjunction with the metrics proxy,
but I don't understand why both {{FairCallQueue}} and {{MetricsProxy}}
implement it. It seems like only the {{MetricsProxy}} should be a
{{MetricsSource}}?
* In {{Metrics.md}}, I think this needs to be under the "rpc" H1 section (as
discussed above), and should have its own H2 header -- there's one of these for
each class which emits metrics. It should probably include some discussion of
the fact that these will only exist if FCQ is enabled.
* The name of the metric should probably use {{getClass().getSimpleName()}}
instead of {{getClass().getName()}} so that it looks like
{{context.FairCallQueue}} instead of including the package name
* Since a process can theoretically have multiple FairCallQueue instances
running simultaneously, we need to include the port information. Take a look at
{{RpcMetrics}} for how it includes the port number as a "tag" that we can later
use to differentiate metrics from different FCQ instances.
* I know checkstyle didn't complain, but we typically discourage the use of
star-imports like you used for {{MetricsAsserts}} in your test, can you make
them individual static imports instead?
* The test looks great! My one comment would be that it's a little confusing
right now that the assertions appear within the catch block -- it would read
more easily if the catch block just had a comment saying that the exception was
expected, then the assertions outside of the try-catch.
> Emit FairCallQueue stats as metrics
> ---
>
> Key: HADOOP-15481
> URL: https://issues.apache.org/jira/browse/HADOOP-15481
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics, rpc-server
>Reporter: Erik Krogen
>Assignee: Christopher Gregorian
>Priority: Major
> Attachments: HADOOP-15481.001.patch, HADOOP-15481.001.patch
>
>
> Currently FairCallQueue has some statistics which are exported via JMX: the
> size of each queue, and the number of overflowed calls per queue. These are
> useful statistics to track over time to determine, for example, if queues
> need to be resized. We should emit them via the standard metrics system.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[
https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739810#comment-16739810
]
Hadoop QA commented on HADOOP-15481:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 53s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 8 new + 32 unchanged - 1 fixed = 40 total (was 33) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 23s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.ssl.TestSSLFactory |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15481 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954480/HADOOP-15481.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 1698a664 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 33c009a4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15766/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15766/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15766/testReport/ |
| Max. process+thread count | 1467 (vs.
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739729#comment-16739729 ] Christopher Gregorian commented on HADOOP-15481: Added a patch to emit the two metrics for each queue mentioned in the description, as FairCallQueueSize_# and FairCallQueueOverflowedCalls_#. > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Attachments: Emit_size_and_overflow_metrics_from_FairCallQueue.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
