[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369711#comment-15369711 ] Hudson commented on MAPREDUCE-6424: --- SUCCESS: Integrated in Hadoop-trunk-Commit #10074 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10074/]) MAPREDUCE-6424. Store MR counters as timeline metrics instead of event. (sjlee: rev f0dbd7a40f0dd9b619ca13b0a0648636c9e21b3b) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFinishedEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptStartedEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestEvents.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/UtilsForTests.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskUpdatedEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/AMStartedEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskFailedEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobInfoChangeEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobQueueChangeEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobSubmittedEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/JobHistoryEventUtils.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskStartedEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/NormalizedResourceEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/HistoryEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobFinishedEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobPriorityChangeEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptUnsuccessfulCompletionEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptFinishedEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobUnsuccessfulCompletionEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobInitedEvent.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRTimelineEventHandling.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobStatusChangedEvent.java > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Fix For: YARN-2928 > > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDU
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265744#comment-15265744 ] Naganarasimha G R commented on MAPREDUCE-6424: -- Thanks [~varun_saxena] for reviewing and committing the patch and [~sjlee0] & [~gtCarrera9] for the review . > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Fix For: YARN-2928 > > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, > MAPREDUCE-6424-YARN-2928.v1.004.patch, MAPREDUCE-6424-YARN-2928.v1.005.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265741#comment-15265741 ] Varun Saxena commented on MAPREDUCE-6424: - Committed this to YARN-2928 branch. Thanks [~Naganarasimha] for your contribution. Thanks [~sjlee0] and [~gtCarrera9] for your reviews. And thanks [~djp] for filing this JIRA. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Fix For: YARN-2928 > > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, > MAPREDUCE-6424-YARN-2928.v1.004.patch, MAPREDUCE-6424-YARN-2928.v1.005.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265734#comment-15265734 ] Varun Saxena commented on MAPREDUCE-6424: - bq. Could you guys review it, and commit it if you are +1 too? [~sjlee0], you meant go ahead and commit it as well ? Will do so. That will help me invoke build for MAPREDUCE-6688 and have a QA report by the time its morning there. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, > MAPREDUCE-6424-YARN-2928.v1.004.patch, MAPREDUCE-6424-YARN-2928.v1.005.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265650#comment-15265650 ] Varun Saxena commented on MAPREDUCE-6424: - LGTM too... > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, > MAPREDUCE-6424-YARN-2928.v1.004.patch, MAPREDUCE-6424-YARN-2928.v1.005.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265647#comment-15265647 ] Li Lu commented on MAPREDUCE-6424: -- +1. Latest patch lgtm. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, > MAPREDUCE-6424-YARN-2928.v1.004.patch, MAPREDUCE-6424-YARN-2928.v1.005.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265640#comment-15265640 ] Sangjin Lee commented on MAPREDUCE-6424: Thanks [~Naganarasimha]! I'm +1 on the latest patch. I'd like to give a little time for [~gtCarrera9] and [~varun_saxena]. Could you guys review it, and commit it if you are +1 too? > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, > MAPREDUCE-6424-YARN-2928.v1.004.patch, MAPREDUCE-6424-YARN-2928.v1.005.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265611#comment-15265611
]
Hadoop QA commented on MAPREDUCE-6424:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 21s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
12s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 28s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 32s
{color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
47s {color} | {color:green} YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 9s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in YARN-2928 has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 23s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s {color} | {color:green} hadoop-mapreduce-project/hadoop-mapreduce-client:
patch generated 0 new + 637 unchanged - 4 fixed = 637 total (was 641) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 49s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 7s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 104m 38s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed
with JDK v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 16s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_95. {color} |
| {col
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265391#comment-15265391
]
Hadoop QA commented on MAPREDUCE-6424:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 16s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 29s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 31s
{color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
47s {color} | {color:green} YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 9s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in YARN-2928 has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 37s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 31s
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: patch
generated 2 new + 637 unchanged - 4 fixed = 639 total (was 641) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 49s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 11s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 104m 36s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed
with JDK v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 15s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_95. {color} |
| {color:green}+1{col
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265362#comment-15265362 ] Sangjin Lee commented on MAPREDUCE-6424: Thanks for tracking down the unit test failure until late/early. The checkstyle issues look trivial to correct. [~Naganarasimha], could I bother you to fix them? Then I'd be +1 on the patch. What do other think? > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, > MAPREDUCE-6424-YARN-2928.v1.004.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265344#comment-15265344
]
Hadoop QA commented on MAPREDUCE-6424:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 19s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
11s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 13s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 12s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 50s
{color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
54s {color} | {color:green} YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 27s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in YARN-2928 has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 37s
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: patch
generated 2 new + 637 unchanged - 4 fixed = 639 total (was 641) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 33s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_92. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 1s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.8.0_92. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 122m 39s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed
with JDK v1.8.0_92. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 45s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_95. {color} |
| {color:red}-1{co
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265271#comment-15265271
]
Naganarasimha G R commented on MAPREDUCE-6424:
--
Thanks [~varun_saxena], for the analysis but anyway i too had narrowed down to
it.
bq. It would however be better to change it to guard against any future
changes in FS based impl. But frankly FS based impl is unlikely to change.
Even i too feel the same as to read each line as entity from the file,
currently i am using
{{FileSystemTimelineReaderImpl.getTimelineRecordFromJSON}}. Test case was
failing because i had assumed that there would no other entities without any
events other than the one published by the MR job, but if the method returns
once we have found the metrics i think it should be sufficient enough. thoughts?
Uploaded the patch for the same, if req can modify the test case to use
{{FileSystemTimelineReaderImpl}}.
> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
> Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch,
> MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch,
> MAPREDUCE-6424-YARN-2928.v1.004.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from
> JobFinishedEvent as timeline events with counters details in JSON format.
> We need to store framework specific counters as metrics in timeline service
> to support query, aggregation, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265241#comment-15265241 ] Varun Saxena commented on MAPREDUCE-6424: - Test case is failing for me every time too. So looked into it. The test case is failing because you are checking for a flag "jobMetricsFoundForAppEntity" being true. But you are not breaking out of the while loop which is iterating over the thist file. So this flag can be reset if any record of YARN_APPLICATION exists in thist file and has no metrics in it. If you break out of the loop as soon as jobMetricsFoundForAppEntity flag becomes true, test case would pass even if we do not address issue with aggregation updates. And I guess thats not required to be done for this test IIUC. I do not think we need to invoke file system based timeline reader explicitly from the test case as of now(if you do not have bandwidth to change it). It would however be better to change it to guard against any future changes in FS based impl. But frankly FS based impl is unlikely to change. There are some NPEs' in the test case output though. We can track that in a separate JIRA. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, > MAPREDUCE-6424-YARN-2928.v1.004.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265106#comment-15265106 ] Naganarasimha G R commented on MAPREDUCE-6424: -- Thanks [~gtCarrera], for looking into it let me try if i can use *file system based timeline reader*... > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, > MAPREDUCE-6424-YARN-2928.v1.004.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265083#comment-15265083 ] Naganarasimha G R commented on MAPREDUCE-6424: -- Oops had slept of by then!, now i realize that my local copy is one commit behind the branch hence may be its passing locally > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, > MAPREDUCE-6424-YARN-2928.v1.004.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265023#comment-15265023 ] Li Lu commented on MAPREDUCE-6424: -- Hmm I think I found the root cause of the UT failure on testMRNewTimelineServiceEventHandling. Note that in the unit test we're directly reading the entity file from the file system. This is different to the implementation of file system based timeline reader, where we merge all updates to the same entity before we return. With the newly added aggregation logic, there are aggregation updates to the application entity. Therefore, the processing logic in verifyMetricsWhenEvent needs to handle this kind of entities (right now it's an empty update). Maybe we'd like to use the file system timeline reader to do this, since there we have a more comprehensive reader implementation? Right now the logic in verifyMetricsWhenEvent seems to be a little bit fragile to future changes? There are certainly improvements we can do on the aggregation side. Let's address that after the first milestone. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, > MAPREDUCE-6424-YARN-2928.v1.004.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264887#comment-15264887 ] Li Lu commented on MAPREDUCE-6424: -- I'm looking into the test failure. Right now I found a timing issue for aggregations, but unfortunately this is not the root cause of the UT failure (those two components appear to be independent). Will keep looking into this issue. [~Naganarasimha] if you have time would you please also try to reproduce the problem? Thanks! > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, > MAPREDUCE-6424-YARN-2928.v1.004.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264818#comment-15264818
]
Li Lu commented on MAPREDUCE-6424:
--
bq. So thought better to postpone this decision to Varun Saxena's patch.
Thoughts?
Sure. Once we plan to refactor it I'm fine.
bq. TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling is
passing locally
I can reproduce the failure on my local machine:
{code}
Running org.apache.hadoop.mapred.TestMRTimelineEventHandling
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 56.764 sec <<<
FAILURE! - in org.apache.hadoop.mapred.TestMRTimelineEventHandling
testMRNewTimelineServiceEventHandling(org.apache.hadoop.mapred.TestMRTimelineEventHandling)
Time elapsed: 56.57 sec <<< FAILURE!
java.lang.AssertionError: Job Metrics are not published
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.mapred.TestMRTimelineEventHandling.verifyMetricsWhenEvent(TestMRTimelineEventHandling.java:361)
at
org.apache.hadoop.mapred.TestMRTimelineEventHandling.checkNewTimelineEvent(TestMRTimelineEventHandling.java:292)
at
org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling(TestMRTimelineEventHandling.java:210)
Results :
Failed tests:
TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling:210->checkNewTimelineEvent:292->verifyMetricsWhenEvent:361
Job Metrics are not published
{code}
>From the log not sure if it's related to some aggregation operations. I'm
>looking into it.
bq. we need to decide if we want to split configs right now and if yes is an
approach based on size configuration or a internal fixed value fine ?
Let's make the sizes configurable? Adding a config should be quick and then we
can decide on their names?
> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
> Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch,
> MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch,
> MAPREDUCE-6424-YARN-2928.v1.004.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from
> JobFinishedEvent as timeline events with counters details in JSON format.
> We need to store framework specific counters as metrics in timeline service
> to support query, aggregation, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264799#comment-15264799 ] Varun Saxena commented on MAPREDUCE-6424: - Ok, will have a look at this in my patch tomorrow as its quite late here now. Moreover, we need to decide if we want to split configs right now and if yes is an approach based on size configuration or a internal fixed value fine ? > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, > MAPREDUCE-6424-YARN-2928.v1.004.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264756#comment-15264756
]
Naganarasimha G R commented on MAPREDUCE-6424:
--
I have uploaded a patch without consideration of above points as i felt though
{{HistoryEvent.getTimelineEntitiesToPublish}} would be ideal but has far too
many changes and need to consider how to handle configs too. So thought better
to postpone this decision to [~varun_saxena]'s patch. Thoughts?
bq. So in general, the penalty for making fine-grained putEntitiesAsync() calls
shouldn't be too big.
What i was considering was if we go for
{{HistoryEvent.getTimelineEntitiesToPublish}}/ {{HistoryEvent.addDataToEntity}}
approach and if config size is too big then we will not be able to address it
by splitting into multiple entities because :
* All entities passed to {{timelineClient.putEntities/putEntitiesAsync}} is
published as a single REST call, currently not considering number of
entities/size.
* *HistoryEvent.addDataToEntity* approach doesn't model multiple entities which
might be required for publishing ApplicationEntity's metrics and configs.
* Based on security concerns we might take different approach in publishing
metrics and configs as part of ApplicationEntity in YARN-3401
TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling is passing
locally
{code}
---
T E S T S
---
Running org.apache.hadoop.mapred.TestMRTimelineEventHandling
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 69.195 sec - in
org.apache.hadoop.mapred.TestMRTimelineEventHandling
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] BUILD SUCCESS
[INFO]
{code}
would be helpful if someone validates it for me
> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
> Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch,
> MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch,
> MAPREDUCE-6424-YARN-2928.v1.004.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from
> JobFinishedEvent as timeline events with counters details in JSON format.
> We need to store framework specific counters as metrics in timeline service
> to support query, aggregation, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264755#comment-15264755 ] Li Lu commented on MAPREDUCE-6424: -- Sure. My main concern is not on the cost of multiple writes, but on the way we convert a HistoryEvent into timeline entities. With the current way, we have to build getMetrics, getConfigs, get for every HistoryEvent types. For most of these types, the getters may probably be trivial. So we're wondering if we'd like to have something like getTimelineEntitiesToPublish to generate related timeline entities out from each event. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, > MAPREDUCE-6424-YARN-2928.v1.004.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264742#comment-15264742
]
Hadoop QA commented on MAPREDUCE-6424:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m 26s
{color} | {color:red} Docker failed to build yetus/hadoop:0ca8df7. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12801522/MAPREDUCE-6424-YARN-2928.v1.004.patch
|
| JIRA Issue | MAPREDUCE-6424 |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6470/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
> Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch,
> MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch,
> MAPREDUCE-6424-YARN-2928.v1.004.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from
> JobFinishedEvent as timeline events with counters details in JSON format.
> We need to store framework specific counters as metrics in timeline service
> to support query, aggregation, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264714#comment-15264714 ] Sangjin Lee commented on MAPREDUCE-6424: I'm fine with the suggestion. Be reminded that the timeline client combines multiple (async) writes to arrive at a reasonable level of writes to the timeline collector. So in general, the penalty for making fine-grained putEntitiesAsync() calls shouldn't be too big. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264702#comment-15264702
]
Li Lu commented on MAPREDUCE-6424:
--
bq. If we plan to handle configs separately, then we can take this approach
thoughts?
Sure. [~varun_saxena] you guys can decide which JIRA you want to put this
change in. {{HistoryEvent.getTimelineEntitiesToPublish}} sounds like a nice
one.
> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
> Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch,
> MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from
> JobFinishedEvent as timeline events with counters details in JSON format.
> We need to store framework specific counters as metrics in timeline service
> to support query, aggregation, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264686#comment-15264686
]
Sangjin Lee commented on MAPREDUCE-6424:
Yeah that might be a better fit. In general, any place where we have a need to
loop to wait until a test condition is satisfied is a candidate to consider
{{GenericTestUtils.waitFor()}}.
> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
> Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch,
> MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from
> JobFinishedEvent as timeline events with counters details in JSON format.
> We need to store framework specific counters as metrics in timeline service
> to support query, aggregation, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264670#comment-15264670
]
Naganarasimha G R commented on MAPREDUCE-6424:
--
Well actually i was more looking at an API like
{{HistoryEvent.getTimelineEntitiesToPublish}} which returns array of
*TimelineEntities* which can be taken by JobHistoryEventHandler and directly
published using timelineclient
> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
> Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch,
> MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from
> JobFinishedEvent as timeline events with counters details in JSON format.
> We need to store framework specific counters as metrics in timeline service
> to support query, aggregation, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264540#comment-15264540
]
Naganarasimha G R commented on MAPREDUCE-6424:
--
bq. l.337-359: Is it possible to use GenericTestUtils.waitFor() for this
[~sjlee0], is this right class and method you are pointing to ? IIUC i think
you meant {{UtilsForTests}} l.613-635, waitForAppFinished right ?
> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
> Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch,
> MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from
> JobFinishedEvent as timeline events with counters details in JSON format.
> We need to store framework specific counters as metrics in timeline service
> to support query, aggregation, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264538#comment-15264538 ] Naganarasimha G R commented on MAPREDUCE-6424: -- bq. My proposal would be generalize the toTimelineEvent method into something like addDataToEntity, Ahh, i had this in back of my mind after i finished the patch, but thought anyway multiple entities needs to be sent out and particularly in the case of configs we might need to split one entity into multiple to avoid the size of one large REST call and at the same time we merge all asyncs hence we might need to send some as sync and some as async, hence held back. If we plan to handle configs separately, then we can take this approach thoughts? > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263283#comment-15263283
]
Hadoop QA commented on MAPREDUCE-6424:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
36s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 4s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 9s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 48s
{color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
56s {color} | {color:green} YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 29s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in YARN-2928 has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 12s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: patch
generated 2 new + 637 unchanged - 4 fixed = 639 total (was 641) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 43s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_92. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 54s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.8.0_92. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 122m 15s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed
with JDK v1.8.0_92. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 58s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_95. {color} |
| {color:red}-1{color} |
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263116#comment-15263116
]
Sangjin Lee commented on MAPREDUCE-6424:
(JobHistoryEventHandler.java)
- l.1053: I believe this causes a checkstyle violation (line too long)
- l.1054: {{timestamp}} here is no longer used, and should be removed
(JobFinishedEvent.java)
- l.161: Sorry I should have been clearer, but since we have a clear timestamp
associated with this event ({{finishTime}}), I think we ought to use that time
over the current time millis. Thoughts? Same for {{MapAttemptFinishedEvent}},
{{ReduceAttemptFinishedEvent}}, {{TaskAttemptFinishedEvent}},
{{TaskAttemptUnsuccessfulCompletionEvent}}, {{TaskFailedEvent}}, and
{{TaskFinishedEvent}}
(TestMRTimelineEventHandling.java)
- l.337-359: Is it possible to use {{GenericTestUtils.waitFor()}} for this?
> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
> Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch,
> MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from
> JobFinishedEvent as timeline events with counters details in JSON format.
> We need to store framework specific counters as metrics in timeline service
> to support query, aggregation, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262973#comment-15262973 ] Li Lu commented on MAPREDUCE-6424: -- Hi [~Naganarasimha], thanks for the work! I took a look at the patch. Most part of the patch LGTM. One general concern: given the fact that we're returning null for most of the HistoryEvents, maybe there's a better way to do this? My proposal would be generalize the toTimelineEvent method into something like addDataToEntity, so that we can attach any kinds of timeline v2 data to entities (info, config, metric, event, etc,.) in different kinds of events. Due to the nature of those HistoryEvents, it's quite possible that each event may only touch a part of a timeline entity, so maybe we don't want to generate events and/or metrics separately? I'm raising this point also because the MapReduce code may act as an example for how to use the v2 APIs in general, so we may want to figure out a "scalable" way to handle all cases. I'm terribly sorry not getting back on this point earlier. Do we still have time to find a solution here given the time is quite limited to our 1st milestone? Thanks! nits: JobHistoryEventHandler l.1053 style, parameter timestamp not used? l.1223 I agree with Sangjin's comments that we may probably want to use a blocking call to post final data. On the performance side, however, is it possible to post all contents in one batch, instead of making two calls? > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262545#comment-15262545 ] Sangjin Lee commented on MAPREDUCE-6424: Please ignore my comment about the findbugs warnings. They appear unrelated to our branch. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262537#comment-15262537
]
Sangjin Lee commented on MAPREDUCE-6424:
Thanks [~Naganarasimha] for the updated patch! It looks good for the most part.
Some feedback:
(JobHistoryEventHandler.java)
- l.1055-1067: it might be better if we do this only if the metrics is not null
(i.e. moving the {{if (timelineMetrics != null)}}) check to the outside
- l.1057, 1061: I don't think this event is needed at the application entity
(all we need are the metrics)
- l.1231: I think this is a debatable point, but since these are final metric
values would it be better to write them via {{putEntities()}}?
(JobHistoryEventUtils.java)
- l.68: This might be a small point, but how about adding {{long timestamp}} as
an input argument so that all metrics can have the same timestamp? That way we
can precisely set the timestamp if needed.
(TestMRTimelineEventHandling.java)
- can we add a test to check the metrics for the application entity?
It looks like we have a couple of pre-existing findbugs warnings. Did you have
a chance to see if they are straightforward to fix? Also, the checkstyle
violations with the patch should be easy to fix. Are the unit test failures
unrelated?
> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
> Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch,
> MAPREDUCE-6424-YARN-2928.v1.002.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from
> JobFinishedEvent as timeline events with counters details in JSON format.
> We need to store framework specific counters as metrics in timeline service
> to support query, aggregation, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15261897#comment-15261897
]
Hadoop QA commented on MAPREDUCE-6424:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m 53s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 53s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
13s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 31s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 24s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 0s
{color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
5s {color} | {color:green} YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 45s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in YARN-2928 has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 27s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 18s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 18s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 39s
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: patch
generated 2 new + 637 unchanged - 4 fixed = 639 total (was 641) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 47s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_92. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 40s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.8.0_92. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 135m 8s {color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed with JDK
v1.8.0_92. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 46s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_95. {color} |
| {color:green}+1{color
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15261515#comment-15261515 ] Naganarasimha G R commented on MAPREDUCE-6424: -- bq. If we don't do this, then we need to find a way to transfer metrics from certain framework-specific entities to the application entities so that they can be aggregated. I would think that would be considerably more involved. Agree, i was thinking of linking this jira to YARN-3401 and comment there to provide alternate approach to publish app metrics (& configuration as part of YARN-3401) from app and currrently applications are publishing System entities directlty. bq. As you mentioned, we can carry on the discussion in a new JIRA. Ok Have uploaded the patch with fixes for you suggestion [~sjlee0]. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259517#comment-15259517
]
Sangjin Lee commented on MAPREDUCE-6424:
{quote}
This seems to be a good idea for oozie like flows but just one query, would it
be good for app to publish metrics in SystemEntities ? IIRC YARN-3401 was
raised to block it later on ? so do we need to support some other mechanism to
publish only metrics ?
{quote}
We may need to refine what we do on YARN-3401. YARN-3401 notwithstanding, we
still have a need to support aggregating framework-owned metrics. Consider our
famous example: HDFS bytes written for the job. If we emit it for the MR job
entity, then it would not benefit from aggregation onto the flow and so on. One
can query for the job entities and get the HDFS bytes written for individual
jobs, but there would be no flow aggregation. By adding the metrics on the YARN
application entity, the flow aggregation will kick in naturally and be able to
show it at the flow run level.
If we don't do this, then we need to find a way to transfer metrics from
certain framework-specific entities to the application entities so that they
can be aggregated. I would think that would be considerably more involved.
{quote}
Ok basically was trying to understand for the next jira, may be we can discuss
more there. Query was : Apart from metrics getting published at the end of each
task(M/R) we need to publish metrics when task(M/R) is running or have some way
to share running task's (M/R) metrics to AM(via umbilical proto) and AM publish
aggregated running tasks (M/R) metrics periodically ?
{quote}
I see. Yeah, I haven't given a lot of thoughts on exactly how we might do this.
As you mentioned, we can carry on the discussion in a new JIRA.
> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Naganarasimha G R
> Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from
> JobFinishedEvent as timeline events with counters details in JSON format.
> We need to store framework specific counters as metrics in timeline service
> to support query, aggregation, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259427#comment-15259427 ] Naganarasimha G R commented on MAPREDUCE-6424: -- Thanks for the comments [~sjlee0], bq. I think we said it is OK to emit them as is and capture the work of emitting them periodically in a later JIRA Ok basically was trying to understand for the next jira, may be we can discuss more there. Query was : Apart from metrics getting published at the end of each task(M/R) we need to publish metrics when task(M/R) is *running* or have some way to share running task's (M/R) metrics to AM(via umbilical proto) and AM publish aggregated running tasks (M/R) metrics periodically ? bq. but we also need emit the metrics at the YARN application level if we want them aggregated to the flows. Therefore, we should add code that emits the metrics for the YARN application entity in addition to the MR job entity. This seems to be a good idea for oozie like flows but just one query, would it be good for app to publish metrics in SystemEntities ? IIRC YARN-3401 was raised to block it later on ? so do we need to support some other mechanism to publish only metrics ? > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259240#comment-15259240 ] Sangjin Lee commented on MAPREDUCE-6424: Thanks [~Naganarasimha] for your patch! I think we said it is OK to emit them as is and capture the work of emitting them periodically in a later JIRA. One thing to add, though: I mentioned it elsewhere but we also need to emit the metrics at the *YARN application* level if we want them aggregated to the flows. Therefore, we should add code that emits the metrics for the YARN application entity in addition to the MR job entity. I'd say that's actually even more important than metrics at the job entity. The metrics at the MR-specific entities are there to view, but they will not be aggregated. On the other hand, the YARN application entity is the focal point of real time flow aggregation. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259183#comment-15259183
]
Hadoop QA commented on MAPREDUCE-6424:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 46s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 6s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
12s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 59s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 54s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 43s
{color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
51s {color} | {color:green} YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 20s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in YARN-2928 has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 38s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 54s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 54s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 33s
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: patch
generated 1 new + 637 unchanged - 4 fixed = 638 total (was 641) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 3s {color} |
{color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.8.0_92. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 47s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.8.0_92. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 111m 35s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed
with JDK v1.8.0_92. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 18s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_95. {color} |
| {color:green}+1{colo
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15258635#comment-15258635 ] Naganarasimha G R commented on MAPREDUCE-6424: -- Hi [~sjlee0] & [~vrushalic], I have captured all the metrics emitted by the job/task/task attempt. Apart from these, was metrics planned to be published periodically also or were you referring to task metrics itself ? > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230682#comment-15230682 ] Vrushali C commented on MAPREDUCE-6424: --- Yes, emitting periodically can be done in another jira. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15227647#comment-15227647 ] Naganarasimha G R commented on MAPREDUCE-6424: -- Hi [~vrushalic] & [~sjlee0], regarding vrushali's comment bq. Need to ensure timeline metrics are emitted periodically as the app is running not just at completion. yes i agree this is required but can we track this in another jira and make *"Store MR counters as timeline metrics instead of event"* at the end of the application as metrics for the 1st milestone ? > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208591#comment-15208591 ] Naganarasimha G R commented on MAPREDUCE-6424: -- Thanks for assigning :) > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208571#comment-15208571 ] Junping Du commented on MAPREDUCE-6424: --- Sure. I just assign this JIRA to you. Thanks for checking. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208568#comment-15208568 ] Naganarasimha G R commented on MAPREDUCE-6424: -- Hi [~djp], Shall i work on this ? > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Junping Du > Labels: yarn-2928-1st-milestone > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208567#comment-15208567 ] Naganarasimha G R commented on MAPREDUCE-6424: -- Hi [~djp], Shall i work on this ? > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Junping Du > Labels: yarn-2928-1st-milestone > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134785#comment-15134785 ] Naganarasimha G R commented on MAPREDUCE-6424: -- Hi [~djp], I would like to work on this issue if you are not planning to work on this. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Junping Du > Labels: yarn-2928-1st-milestone > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132960#comment-15132960 ] Vrushali C commented on MAPREDUCE-6424: --- Need to ensure timeline metrics are emitted periodically as the app is running not just at completion. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Junping Du > Labels: yarn-2928-1st-milestone > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
