[jira] [Commented] (YARN-6375) App level aggregation should not consider metric values reported in the previous aggregation cycle
[ https://issues.apache.org/jira/browse/YARN-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997555#comment-15997555 ] Hudson commented on YARN-6375: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11686 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11686/]) YARN-6375 App level aggregation should not consider metric values (vrushali: rev 54e2b9e876fd91712c14ffbc4c49cd946f305aeb) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestTimelineCollector.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollector.java > App level aggregation should not consider metric values reported in the > previous aggregation cycle > -- > > Key: YARN-6375 > URL: https://issues.apache.org/jira/browse/YARN-6375 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Fix For: YARN-5355, YARN-5355-branch-2, 3.0.0-alpha3 > > Attachments: YARN-6375-YARN-5355.01.patch, > YARN-6375-YARN-5355.02.patch > > > Currently app level aggregation is done every 15 seconds. > And we consider last reported metric value for each entity belonging to an > app for aggregation. > We however merely update the corresponding metric values for the entity on > put. We never remove the entries. > But it is possible that multiple entities finish during lifetime of an > application. We however continue to consider them till the end. > We should however not consider metric values of entities unless reported > within the 15 second period. > Consider containers. For a long running app, several containers would start > and end at various times during the lifetime of an app. > To consider metrics for all the containers throughout the lifetime of app, > hence wont be correct. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6375) App level aggregation should not consider metric values reported in the previous aggregation cycle
[ https://issues.apache.org/jira/browse/YARN-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997521#comment-15997521 ] Vrushali C commented on YARN-6375: -- Thanks [~varun_saxena] for the 002 patch. LGTM, I will commit it shortly. > App level aggregation should not consider metric values reported in the > previous aggregation cycle > -- > > Key: YARN-6375 > URL: https://issues.apache.org/jira/browse/YARN-6375 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6375-YARN-5355.01.patch, > YARN-6375-YARN-5355.02.patch > > > Currently app level aggregation is done every 15 seconds. > And we consider last reported metric value for each entity belonging to an > app for aggregation. > We however merely update the corresponding metric values for the entity on > put. We never remove the entries. > But it is possible that multiple entities finish during lifetime of an > application. We however continue to consider them till the end. > We should however not consider metric values of entities unless reported > within the 15 second period. > Consider containers. For a long running app, several containers would start > and end at various times during the lifetime of an app. > To consider metrics for all the containers throughout the lifetime of app, > hence wont be correct. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6375) App level aggregation should not consider metric values reported in the previous aggregation cycle
[ https://issues.apache.org/jira/browse/YARN-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996490#comment-15996490 ] Varun Saxena commented on YARN-6375: Updated the patch. Findbugs is unrelated to this patch. Will file another JIRA for it > App level aggregation should not consider metric values reported in the > previous aggregation cycle > -- > > Key: YARN-6375 > URL: https://issues.apache.org/jira/browse/YARN-6375 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6375-YARN-5355.01.patch, > YARN-6375-YARN-5355.02.patch > > > Currently app level aggregation is done every 15 seconds. > And we consider last reported metric value for each entity belonging to an > app for aggregation. > We however merely update the corresponding metric values for the entity on > put. We never remove the entries. > But it is possible that multiple entities finish during lifetime of an > application. We however continue to consider them till the end. > We should however not consider metric values of entities unless reported > within the 15 second period. > Consider containers. For a long running app, several containers would start > and end at various times during the lifetime of an app. > To consider metrics for all the containers throughout the lifetime of app, > hence wont be correct. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6375) App level aggregation should not consider metric values reported in the previous aggregation cycle
[
https://issues.apache.org/jira/browse/YARN-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995095#comment-15995095
]
Hadoop QA commented on YARN-6375:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
48s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} YARN-5355 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
28s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
in YARN-5355 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
41s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-6375 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12866194/YARN-6375-YARN-5355.02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 15d58c1d884c 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5355 / 1f98134 |
| Default Java | 1.8.0_121 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/15809/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15809/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15809/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> App level aggregation should not consider metric values reported in the
> previous aggregation cycle
>
[jira] [Commented] (YARN-6375) App level aggregation should not consider metric values reported in the previous aggregation cycle
[
https://issues.apache.org/jira/browse/YARN-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989872#comment-15989872
]
Hadoop QA commented on YARN-6375:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
3s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} YARN-5355 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
29s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
in YARN-5355 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 9s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice:
The patch generated 5 new + 1 unchanged - 0 fixed = 6 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
41s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-6375 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861223/YARN-6375-YARN-5355.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0ddca9ac4e60 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5355 / 1f98134 |
| Default Java | 1.8.0_121 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/15780/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/15780/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15780/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
|
| Console
[jira] [Commented] (YARN-6375) App level aggregation should not consider metric values reported in the previous aggregation cycle
[
https://issues.apache.org/jira/browse/YARN-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989521#comment-15989521
]
Vrushali C commented on YARN-6375:
--
Hi [~varun_saxena]
I noticed some checkstyle issues in the new test code in the patch. Will it be
okay for you to fix those?
{code}
[ERROR]
src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestTimelineCollector.java:[218]
(indentation) Indentation: 'method def modifier' have incorrect indentation
level 9, expected level should be one of the following: 6, 8, 10.
[ERROR]
src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestTimelineCollector.java:[220]
(indentation) Indentation: 'method def' child have incorrect indentation level
11, expected level should be one of the following: 8, 10, 12.
[ERROR]
src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestTimelineCollector.java:[222]
(indentation) Indentation: 'method def rcurly' have incorrect indentation
level 9, expected level should be one of the following: 6, 8, 10.
[ERROR]
src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestTimelineCollector.java:[229,19]
(whitespace) NoWhitespaceBefore: ';' is preceded with whitespace.
[ERROR]
src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestTimelineCollector.java:[263,19]
(whitespace) NoWhitespaceBefore: ';' is preceded with whitespace.
{code}
thanks
Vrushali
> App level aggregation should not consider metric values reported in the
> previous aggregation cycle
> --
>
> Key: YARN-6375
> URL: https://issues.apache.org/jira/browse/YARN-6375
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-6375-YARN-5355.01.patch
>
>
> Currently app level aggregation is done every 15 seconds.
> And we consider last reported metric value for each entity belonging to an
> app for aggregation.
> We however merely update the corresponding metric values for the entity on
> put. We never remove the entries.
> But it is possible that multiple entities finish during lifetime of an
> application. We however continue to consider them till the end.
> We should however not consider metric values of entities unless reported
> within the 15 second period.
> Consider containers. For a long running app, several containers would start
> and end at various times during the lifetime of an app.
> To consider metrics for all the containers throughout the lifetime of app,
> hence wont be correct.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6375) App level aggregation should not consider metric values reported in the previous aggregation cycle
[ https://issues.apache.org/jira/browse/YARN-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987166#comment-15987166 ] Vrushali C commented on YARN-6375: -- +1 Thanks [~varun_saxena] for the patch. Yes, I think we would need some way of identifying the metric type (gauge vs counter) but that is outside the scope of this jira. I do agree that we don't want to aggregate all values all the time, so let's go ahead with this patch. I will try to commit this shortly. > App level aggregation should not consider metric values reported in the > previous aggregation cycle > -- > > Key: YARN-6375 > URL: https://issues.apache.org/jira/browse/YARN-6375 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6375-YARN-5355.01.patch > > > Currently app level aggregation is done every 15 seconds. > And we consider last reported metric value for each entity belonging to an > app for aggregation. > We however merely update the corresponding metric values for the entity on > put. We never remove the entries. > But it is possible that multiple entities finish during lifetime of an > application. We however continue to consider them till the end. > We should however not consider metric values of entities unless reported > within the 15 second period. > Consider containers. For a long running app, several containers would start > and end at various times during the lifetime of an app. > To consider metrics for all the containers throughout the lifetime of app, > hence wont be correct. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6375) App level aggregation should not consider metric values reported in the previous aggregation cycle
[ https://issues.apache.org/jira/browse/YARN-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15986299#comment-15986299 ] Varun Saxena commented on YARN-6375: Ping [~vrushalic]. Is the current approach fine? > App level aggregation should not consider metric values reported in the > previous aggregation cycle > -- > > Key: YARN-6375 > URL: https://issues.apache.org/jira/browse/YARN-6375 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6375-YARN-5355.01.patch > > > Currently app level aggregation is done every 15 seconds. > And we consider last reported metric value for each entity belonging to an > app for aggregation. > We however merely update the corresponding metric values for the entity on > put. We never remove the entries. > But it is possible that multiple entities finish during lifetime of an > application. We however continue to consider them till the end. > We should however not consider metric values of entities unless reported > within the 15 second period. > Consider containers. For a long running app, several containers would start > and end at various times during the lifetime of an app. > To consider metrics for all the containers throughout the lifetime of app, > hence wont be correct. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6375) App level aggregation should not consider metric values reported in the previous aggregation cycle
[ https://issues.apache.org/jira/browse/YARN-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951228#comment-15951228 ] Varun Saxena commented on YARN-6375: Thanks [~vrushalic] for the comments. I was primarily going by the metrics we currently report. And what can we support at this point of time The 2 examples you pointed out is basically the difference between how we should treat a gauge vs a counter. As you said for CPU and Memory, it probably makes sense in the way I have done in the patch. This is what I had primarily in mind while raising JIRA. But for counters the current mechanism might be better. Counters such as HDFS bytes read can be aggregated at the application level by AM itself as they are currently done by Mapreduce AM. However, I do see merit though in aggregating them on our own on the collector side. If we do say that we will sum up metrics across the lifetime of an app then we would need to handle that on collector restart as well. Do we store all the metrics in the state store which are read back upon restart? Or probably we would need some kind of edit log mechanism. Probably we need some way of identifying the metric type (gauge vs counter). And do not clear counters but clear gauges upon aggregation. Maybe indicate it in TimelineMetric object and rely on client reporting it in a consistent fashion? Or adopt the way we aggregate metrics for a flow run? This probably needs to be discussed further. bq. for CPU, perhaps we might want to know across this application what was the cpu used by all the containers in the lifetime of this app? This would be better dealt with time weighted accumulation which is in future scope. Just summing up metrics of all entities running across different timelines may not be very useful for metrics such as CPU / Memory. For instance, * App1 has 3 containers running one after the other for 10 second each i.e. app runs for 30 seconds. And these containers while they were running, report CPU of 20% each time. * App2 has 3 containers but they run parallely for 30 seconds. And similar to above these containers report CPU of 20% each time. Now, aggregation as it is done at 15 second period would lead to a value of 120% in both the cases. But both the apps and their resource requirements are very distinct. > App level aggregation should not consider metric values reported in the > previous aggregation cycle > -- > > Key: YARN-6375 > URL: https://issues.apache.org/jira/browse/YARN-6375 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6375-YARN-5355.01.patch > > > Currently app level aggregation is done every 15 seconds. > And we consider last reported metric value for each entity belonging to an > app for aggregation. > We however merely update the corresponding metric values for the entity on > put. We never remove the entries. > But it is possible that multiple entities finish during lifetime of an > application. We however continue to consider them till the end. > We should however not consider metric values of entities unless reported > within the 15 second period. > Consider containers. For a long running app, several containers would start > and end at various times during the lifetime of an app. > To consider metrics for all the containers throughout the lifetime of app, > hence wont be correct. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6375) App level aggregation should not consider metric values reported in the previous aggregation cycle
[ https://issues.apache.org/jira/browse/YARN-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950347#comment-15950347 ] Vrushali C commented on YARN-6375: -- Hello Varun, I am trying to understand the jira and I have some questions. I understand what you are saying by aggregation could aggregate only those metrics that are reported by entities in the current timeframe. I am trying to think of an example where we need aggregation for only currently reported metrics vs aggregation for lifetime metrics. CPU may be a good example of aggregating currently reported metrics? For instance, we might want to see what is the CPU usage while these containers are running but once a container is done, we need not consider it's last reported CPU for current CPU usage of this app. But, for other metrics, like say HDFS_BYTES_READ, perhaps we do want to think about how many bytes have been read across tasks? I am thinking out loud. Also, for CPU, perhaps we might want to know across this application what was the cpu used by all the containers in the lifetime of this app? What metrics are you thinking of.. > App level aggregation should not consider metric values reported in the > previous aggregation cycle > -- > > Key: YARN-6375 > URL: https://issues.apache.org/jira/browse/YARN-6375 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-6375-YARN-5355.01.patch > > > Currently app level aggregation is done every 15 seconds. > And we consider last reported metric value for each entity belonging to an > app for aggregation. > We however merely update the corresponding metric values for the entity on > put. We never remove the entries. > But it is possible that multiple entities finish during lifetime of an > application. We however continue to consider them till the end. > We should however not consider metric values of entities unless reported > within the 15 second period. > Consider containers. For a long running app, several containers would start > and end at various times during the lifetime of an app. > To consider metrics for all the containers throughout the lifetime of app, > hence wont be correct. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6375) App level aggregation should not consider metric values reported in the previous aggregation cycle
[
https://issues.apache.org/jira/browse/YARN-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15949271#comment-15949271
]
Hadoop QA commented on YARN-6375:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
24s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 10s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice:
The patch generated 5 new + 1 unchanged - 0 fixed = 6 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
40s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-6375 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861223/YARN-6375-YARN-5355.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0a28deccc8f9 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5355 / c0095d1 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/15436/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15436/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15436/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> App level aggregation should not consider metric values reported in the
> previous aggregation cycle
>
[jira] [Commented] (YARN-6375) App level aggregation should not consider metric values reported in the previous aggregation cycle
[
https://issues.apache.org/jira/browse/YARN-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15949027#comment-15949027
]
Hadoop QA commented on YARN-6375:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 10s{color}
| {color:red} YARN-6375 does not apply to YARN-5355. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6375 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861222/YARN-6375-YARN-5355.01.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15434/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> App level aggregation should not consider metric values reported in the
> previous aggregation cycle
> --
>
> Key: YARN-6375
> URL: https://issues.apache.org/jira/browse/YARN-6375
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-6375-YARN-5355.01.patch
>
>
> Currently app level aggregation is done every 15 seconds.
> And we consider last reported metric value for each entity belonging to an
> app for aggregation.
> We however merely update the corresponding metric values for the entity on
> put. We never remove the entries.
> But it is possible that multiple entities finish during lifetime of an
> application. We however continue to consider them till the end.
> We should however not consider metric values of entities unless reported
> within the 15 second period.
> Consider containers. For a long running app, several containers would start
> and end at various times during the lifetime of an app.
> To consider metrics for all the containers throughout the lifetime of app,
> hence wont be correct.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6375) App level aggregation should not consider metric values reported in the previous aggregation cycle
[
https://issues.apache.org/jira/browse/YARN-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937266#comment-15937266
]
Varun Saxena commented on YARN-6375:
Assume in the first 15 second period, entities reported with corresponding
metric values for metric1 are as under:
{entity1: metric1=50, entity2: metric1=100, entity3: metric1=150, entity4:
metric1=200, entity5: metric1=250}
When aggregation occurs(SUM operation) we would write out app entity with
metric1 having corresponding value of 750.
Let us say by next 15 second period, entity1 and entity2 finish and following
metric values are reported.
{entity3: metric1=50, entity4: metric1=70, entity5: metric1=90}
When aggregation occurs(SUM operation) we would write out app entity with
metric1 having corresponding value of 360.
I think the value reported should be 210.
> App level aggregation should not consider metric values reported in the
> previous aggregation cycle
> --
>
> Key: YARN-6375
> URL: https://issues.apache.org/jira/browse/YARN-6375
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Varun Saxena
>
> Currently app level aggregation is done every 15 seconds.
> And we consider last reported metric value for each entity belonging to an
> app for aggregation.
> We however merely update the corresponding metric values for the entity on
> put. We never remove the entries.
> But it is possible that multiple entities finish during lifetime of an
> application. We however continue to consider them till the end.
> We should however not consider metric values of entities unless reported
> within the 15 second period.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
