[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578769#comment-15578769 ] Hudson commented on YARN-5699: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10622 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10622/]) YARN-5699. Retrospect yarn entity fields which are publishing in events (sjlee: rev 1f304b0c7f261369dd68839507bb609a949965ad) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryManagerOnTimelineStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TimelineServiceV1Publisher.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TimelineServiceV2Publisher.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/metrics/AppAttemptMetricsConstants.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ContainerMetricsConstants.java > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Fix For: 3.0.0-alpha2 > > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch, 0002-YARN-5699.patch, > 0003-YARN-5699.YARN-5355.patch, 0003-YARN-5699.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[
https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578279#comment-15578279
]
Hadoop QA commented on YARN-5699:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 51s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
47s {color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 29s
{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 46s
{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
55s {color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
40s {color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server:
The patch generated 0 new + 102 unchanged - 11 fixed = 102 total (was 113)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 12s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 8s
{color} | {color:green} hadoop-yarn-server-applicationhistoryservice in the
patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 35m 55s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 8s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12833590/0003-YARN-5699.YARN-5355.patch
|
| JIRA Issue | YARN-5699 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b2780d2750ed 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5355 / 0c18631 |
|
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15577056#comment-15577056 ] Rohith Sharma K S commented on YARN-5699: - Yes, current patch given based on trunk. Sure, I will upload patch for YARN-5355 branch. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch, 0002-YARN-5699.patch, 0003-YARN-5699.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576895#comment-15576895 ] Sangjin Lee commented on YARN-5699: --- The latest patch LGTM. I'll wait until tomorrow to give folks time to chime in before I commit. [~rohithsharma], just to be clear, is the intent to commit this to trunk? If I take the latest patch (https://issues.apache.org/jira/secure/attachment/12833383/0003-YARN-5699.patch) it does not apply cleanly to the YARN-5355 branch. Could you please upload the corresponding YARN-5355 patch? > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch, 0002-YARN-5699.patch, 0003-YARN-5699.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575746#comment-15575746 ] Rohith Sharma K S commented on YARN-5699: - Test failures are unrelated to patch. YARN-5377 and YARN-5737 are tracking JIRA for test failures. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch, 0002-YARN-5699.patch, 0003-YARN-5699.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[
https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575722#comment-15575722
]
Hadoop QA commented on YARN-5699:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 55s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server:
The patch generated 0 new + 103 unchanged - 11 fixed = 103 total (was 114)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 30s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 59s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 45s {color}
| {color:red} hadoop-yarn-server-applicationhistoryservice in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 35m 36s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 34s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
|
| | hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12833383/0003-YARN-5699.patch |
| JIRA Issue | YARN-5699 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 748573d8db0b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575536#comment-15575536 ] Rohith Sharma K S commented on YARN-5699: - Updated the patch fixing review comment and checkstyle. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch, 0002-YARN-5699.patch, 0003-YARN-5699.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575534#comment-15575534 ] Rohith Sharma K S commented on YARN-5699: - Basically TestApplicationHistoryManagerOnTimelineStore runs on ATSv1. They publish the entities to store and read it back where they verify each fields for validating published value is read. In ATSv2, there is such framework exists as of now. I have raised a JIRA YARN-5737 to validate each fields that are published in the form of report. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch, 0002-YARN-5699.patch, 0003-YARN-5699.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574347#comment-15574347 ] Sangjin Lee commented on YARN-5699: --- OK it's more of curiosity on my part as what this test does is obviously different than what we are doing in this patch. I am fine with opening a different JIRA to look into this test. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch, 0002-YARN-5699.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573867#comment-15573867 ] Rohith Sharma K S commented on YARN-5699: - bq. the name DIAGNOSTICS_INFO_INFO doesn't sound quite right; just DIAGNOSTICS_INFO? make sense to me, I did not notice this. I will update this. bq. I'm a little confused by this test; it appears that we're preparing these entities as before (i.e. all this info is at the event level). Is that intended? Don't we want to reflect the same changes in this test too? I wonder how this test is passing then (or what it's testing even)? frankly, I haven not done any modification in this test classes *explicitly*. I did *MetricsConstants classes refactoring which resulted in changes in some of the test classes. I can look at these test class and if any minor modifications I can handle it in this patch itself, otherwise we might need to re visit whole test class again that could be done in separate JIRA. And also I notice that there is no specific test cases that are validating individual fields for entity that are published to ATSv2 in TestSystemMetricsPublisherForV2. Again it is bunch of test cases to be added to validate each fields. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch, 0002-YARN-5699.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[
https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573656#comment-15573656
]
Sangjin Lee commented on YARN-5699:
---
Thanks for the update patch [~rohithsharma]! I think it's almost there. A few
comments.
(AppAttemptMetricsConstants.java)
- l.55: the name {{DIAGNOSTICS_INFO_INFO}} doesn't sound quite right; just
{{DIAGNOSTICS_INFO}}?
(ContainerMetricsConstants.java)
- l.64: same as above
(TestApplicationHistoryManagerOnTimelineStore.java)
- I'm a little confused by this test; it appears that we're preparing these
entities as before (i.e. all this info is at the event level). Is that
intended? Don't we want to reflect the same changes in this test too? I wonder
how this test is passing then (or what it's testing even)?
(NMTimelinePublisher.java)
- l.207: I suspect it might be the same, but just to be explicit, should we do
{{toString()}}?
> Retrospect yarn entity fields which are publishing in events info fields.
> -
>
> Key: YARN-5699
> URL: https://issues.apache.org/jira/browse/YARN-5699
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch,
> 0002-YARN-5699.YARN-5355.patch, 0002-YARN-5699.patch
>
>
> Currently, all the container information are published at 2 places. Some of
> them are at entity info(top-level) and some are at event info.
> For containers, some of the event info should be published at container info
> level. For example : container exist status, container state, createdTime,
> finished time. These are general information to container required for
> container-report. So it is better to publish at top level info field.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572814#comment-15572814 ] Rohith Sharma K S commented on YARN-5699: - Test failure is not related to patch, i.e YARN-5679 handles it. Checkstyle is related to patch, I will fix it and upload a new patch. Before that I would like to hear review comments if any so that I can upload single patch. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch, 0002-YARN-5699.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[
https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571847#comment-15571847
]
Hadoop QA commented on YARN-5699:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
47s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 47s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 28s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The
patch generated 1 new + 102 unchanged - 11 fixed = 103 total (was 113) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 58s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 29s {color}
| {color:red} hadoop-yarn-server-applicationhistoryservice in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 38m 59s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 40s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12833093/0002-YARN-5699.patch |
| JIRA Issue | YARN-5699 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5a112343ca6d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 901eca0 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570899#comment-15570899 ] Sangjin Lee commented on YARN-5699: --- I'd like to know what would be made much easier with having the created time as entity info that is inconvenient or difficult with the explicit created time attribute. I get that it would be more symmetric with the finished time, but that alone is not a *strong* argument for replicating this info. Since we're talking about container entities, this is probably the most number of entities in the storage. If we can help not add a column *per object* unnecessarily, I think it would be a good thing. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570708#comment-15570708 ] Rohith Sharma K S commented on YARN-5699: - bq. So I'm not sure whether we need to store it again as part of the entity info. I agree with Li lu. Even I don't have strong opinion to add duplicate created time with different info key, since it is available in entity level. Though created time is same entity createdTime, adding duplicate createdTime with info key as YARN_CONTAINER_CREATED_TIME and YARN_CONTAINER_FINISHED_TIME makes more user facing to understand it. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570387#comment-15570387 ] Li Lu commented on YARN-5699: - OK, I don't have a strong preference on that. Once the data is available on entity level I think we're fine. The benefit for replicating it is not to make created time different to end time, but I'm fine with either ways. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570302#comment-15570302 ] Sangjin Lee commented on YARN-5699: --- The created time just seems redundant. Is there something that cannot be done by the explicit created time attribute? Rohith said bq. It is only for easier accessibility rather than going through event time stamps. We're already setting it to the entity created time, so you would *not* be going through event time stamps for the created time. So I'm not sure whether we need to store it again as part of the entity info. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570267#comment-15570267 ] Li Lu commented on YARN-5699: - Thanks [~sjlee0]. This said I'm generally fine with Rohith's planning. End time looks belong to an entity but not just an event, too. For start time, it looks natural to have some queries like "list all the apps start in a time window and finished in a time window"? So if it's not too expensive, I think it's fine to also put it in info field? > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570224#comment-15570224 ] Sangjin Lee commented on YARN-5699: --- Thanks for the clarification. Then what do you think of the currently proposed set of items in the patch? Do you think they all belong at the entity level? I think one thing we were debating was the diagnostic info. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569980#comment-15569980 ] Li Lu commented on YARN-5699: - bq. Are you proposing elevating all event info items to the entity info unless they really belong only to events? This is my intention. If the event carries information about the timeline entity, why not posting it as an entity level info? > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[
https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569971#comment-15569971
]
Sangjin Lee commented on YARN-5699:
---
bq. What about end time? End time has to be get from event time stamp. But
anyway, I have added a comment for streamlining YARN entities publishing, I
think we should decide for providing YARN entities converter rather than fixing
individual many bugs.
Created time is a first-class attribute of an entity, so it would be good to
use that instead of a generic entity info field. End time is not a first-class
attribute, so entity info would be a natural place.
{quote}
Actually let's take one step backward... What is the key benefit for keeping
most data in event info, rather than in entity info? To me, storing data in
entity info map looks a good representation that the data "belongs to" the
whole entity, but not just one event. Even though some data may come from a
certain type of timeline event, if it adds useful information to the whole
entity, we should put them in entity info field. I'm not sure if there are some
implementation side concerns that prevent us from doing this. However, even if
so, I believe we should fix those blockers but not solve the problem the other
way around (avoid putting data in entity info, but just put them in event
info). I would de-prioritize the work on making the v2 code path consistent
with v1's if this is a problem for our current design.
{quote}
I'm not so sure if I understand what you meant by this. Are you proposing
elevating all event info items to the entity info unless they really belong
only to events? Or are you proposing not to do this work? Bit confused...
> Retrospect yarn entity fields which are publishing in events info fields.
> -
>
> Key: YARN-5699
> URL: https://issues.apache.org/jira/browse/YARN-5699
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch,
> 0002-YARN-5699.YARN-5355.patch
>
>
> Currently, all the container information are published at 2 places. Some of
> them are at entity info(top-level) and some are at event info.
> For containers, some of the event info should be published at container info
> level. For example : container exist status, container state, createdTime,
> finished time. These are general information to container required for
> container-report. So it is better to publish at top level info field.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569771#comment-15569771 ] Li Lu commented on YARN-5699: - Actually let's take one step backward... What is the key benefit for keeping most data in event info, rather than in entity info? To me, storing data in entity info map looks a good representation that the data "belongs to" the whole entity, but not just one event. Even though some data may come from a certain type of timeline event, if it adds useful information to the whole entity, we should put them in entity info field. I'm not sure if there are some implementation side concerns that prevent us from doing this. However, even if so, I believe we should fix those blockers but not solve the problem the other way around (avoid putting data in entity info, but just put them in event info). I would de-prioritize the work on making the v2 code path consistent with v1's if this is a problem for our current design. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568675#comment-15568675 ] Rohith Sharma K S commented on YARN-5699: - bq. Created time is at the topmost level in an entity. We don't need to go through event timestamps for it. What about end time? End time has to be get from event time stamp. But anyway, I have added a [comment|https://issues.apache.org/jira/browse/YARN-5561?focusedCommentId=15568429=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15568429] for streamlining YARN entities publishing, I think we should decide for providing YARN entities converter rather than fixing individual many bugs. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568600#comment-15568600 ] Varun Saxena commented on YARN-5699: bq. It is only for easier accessibility rather than going through event time stamps. Created time is at the topmost level in an entity. We don't need to go through event timestamps for it. Other intention of having created time in info would be if we want to send a query like return me all entities which have a specific created time in milliseconds. Highly unlikely somebody would want this. Even this though, can be achieved by using createdTimeStart/end filters. To me, we are repeating this information by keeping it in info. I think it will serve your use case even otherwise. Thoughts ? cc [~rohithsharma] > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[
https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567797#comment-15567797
]
Rohith Sharma K S commented on YARN-5699:
-
To me, goal for moving event info to entity info is {{User should be able to
get application/application-attempt/container reports at entity-info rather
than entity-event-info.}} The main advantage is user need not iterate trough
every events to get the report details. Iterating is time consuming when
rendering data in web UI by deserializing the pay load.
[~gtCarrera9]
bq. A bigger concern is on the code itself. Right now we have a lot of info key
name constants ending as "EVENT_INFO",
True, I do a change removing *EVENT* in all the fields in next patches.
[~sjlee0]
bq. regarding the container state that was removed in YARN-5156; perhaps we
should add it as entity info here (with COMPLETED)?
I will make a change adding container_state in next patches
bq. we’re already setting the entity created time in l.190; do we need to add
it to the entity info? I would say, we don't need this
It is only for easier accessibility rather than going through event time stamps.
> Retrospect yarn entity fields which are publishing in events info fields.
> -
>
> Key: YARN-5699
> URL: https://issues.apache.org/jira/browse/YARN-5699
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch,
> 0002-YARN-5699.YARN-5355.patch
>
>
> Currently, all the container information are published at 2 places. Some of
> them are at entity info(top-level) and some are at event info.
> For containers, some of the event info should be published at container info
> level. For example : container exist status, container state, createdTime,
> finished time. These are general information to container required for
> container-report. So it is better to publish at top level info field.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563620#comment-15563620 ] Li Lu commented on YARN-5699: - Thanks for the work [~rohithsharma]! Generally fine but a few concerns: 1. I agree with Varun and Sangjin that we may need to think twice about diagnostic info. This field is way too cumbersome for users to directly query. I think it's fine to leave it in event info. 2. When we get app state updated event, we may want to post the event info as is, but at the same time update entity level information? 3. A bigger concern is on the code itself. Right now we have a lot of info key name constants ending as "EVENT_INFO", but actually we no longer use them as event info. Instead, we may post them as entity info, or event info, or both. We need to update or simplify those constants to make them less confusing. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557347#comment-15557347 ] Varun Saxena commented on YARN-5699: bq. regarding the container state that was removed in YARN-5156; perhaps we should add it as entity info here (with COMPLETED)? Agree, we can do so. That was one of the suggested options in YARN-5156, set it to RUNNING in container start and COMPLETED on container end. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[
https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556601#comment-15556601
]
Sangjin Lee commented on YARN-5699:
---
Thanks [~rohithsharma] for the patch! Sorry it took me a while to get to this.
I generally agree with pulling some info from events to the entity level,
especially if they can be part of queries. We do want to be careful not to put
everything at the entity level, however. We probably want to limit it to *only*
the things we will actually query.
In more detail,
(NMTimelinePublisher.java)
- l.182: we’re already setting the entity created time in l.190; do we need to
add it to the entity info? I would say, we don't need this
- l.203: do we need to add diagnostics to the entity level? Not sure if it is
useful. Can we keep it on the event?
- l.307: the same comment as Varun’s; should be the other way around?
- regarding the container state that was removed in YARN-5156; perhaps we
should add it as entity info here (with COMPLETED)?
(ContainerMetricsConstants.java)
- l.44: see {{NMTimelinePublisher.java}}
(TimelineServiceV2Publisher.java)
- l.158: same as above; is diagnostics needed at the entity level?
- l.190: this should be {{entity.setInfo()}} not {{tEvent.setInfo()}}
- l.289: same as above
It would be great if we can add some unit tests to check these. Thanks!
> Retrospect yarn entity fields which are publishing in events info fields.
> -
>
> Key: YARN-5699
> URL: https://issues.apache.org/jira/browse/YARN-5699
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch,
> 0002-YARN-5699.YARN-5355.patch
>
>
> Currently, all the container information are published at 2 places. Some of
> them are at entity info(top-level) and some are at event info.
> For containers, some of the event info should be published at container info
> level. For example : container exist status, container state, createdTime,
> finished time. These are general information to container required for
> container-report. So it is better to publish at top level info field.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552342#comment-15552342 ] Varun Saxena commented on YARN-5699: For tracking URL part though, AHS will not be enabled with ATSv2 and new Web UI. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552332#comment-15552332 ] Varun Saxena commented on YARN-5699: Thanks [~rohithsharma] for the latest patch. Few comments: # In NMTimelinePublisher#publishContainerEvent, shouldn't we check against httpAddress being null ? Instead of not being null. # I see that we are publishing tracking URL twice. If AHS is enabled, tracking URL is changed to AHS web endpoint when attempt finishes. So info field will have its latest value as AHS url. In that case it should be attached to event also so that we know the original AM tracking URL too (i.e. when attempt was registered). Thoughts ? # In TimelineServiceV2Publisher#appStateUpdated, changes are not required as we are effectively doing the same thing as before. We should definitely publish this info at event level because event is app state update event. Also should we update it at info level too so that we can filter apps while they are running based on their current state ? # Do we need to publish master container info at app attempt level twice ? bq. Updated patch also fixes a another bug i.e always NM published http address port used to come as zero. Thanks for the fix. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[
https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552237#comment-15552237
]
Hadoop QA commented on YARN-5699:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s {color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 38s
{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s {color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 34s
{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s {color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 11s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 36m 41s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 74m 35s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831965/0002-YARN-5699.YARN-5355.patch
|
| JIRA Issue | YARN-5699 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 35d5a49f12ea 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5355 / 5d7ad39 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/13307/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552187#comment-15552187 ] Rohith Sharma K S commented on YARN-5699: - Updated patch also fixes a another bug i.e always NM published http address port used to come as zero. This is because of when NMPublisher started, web server was not started which resulting in http port used be zero. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552048#comment-15552048 ] Rohith Sharma K S commented on YARN-5699: - Updated patch by reverting changes made in NMTimelinePublisher. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[
https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551926#comment-15551926
]
Hadoop QA commented on YARN-5699:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m 48s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s {color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 34s
{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 31s
{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
46s {color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s {color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 10s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 37m 4s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 1s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831943/0001-YARN-5699.YARN-5355.patch
|
| JIRA Issue | YARN-5699 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3c5553fe091d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5355 / 5d7ad39 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551908#comment-15551908 ] Varun Saxena commented on YARN-5699: bq. Otherwise we miss the this information else container_state info should be published from NM So the reason this was removed in YARN-5156 was that container state can either be RUNNING and COMPLETED and it becomes COMPLETED only on CONTAINER_FINISHED event. Which means it can be easily deduced from event. At the time container state was carried in event info so the majority opinion went with removing container state altogether. I have made a comment on YARN-5156 too. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551812#comment-15551812 ] Naganarasimha G R commented on YARN-5699: - Had missed YARN-5156 let me check how to include that scenario too > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551803#comment-15551803 ] Naganarasimha G R commented on YARN-5699: - There was a detailed discussion on this and then we concluded it to be in NM only and yes all the required information is getting published from NM right ? > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[
https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551786#comment-15551786
]
Rohith Sharma K S commented on YARN-5699:
-
bq. Is this intentional ?
yes, because the same information also published by RM. So I think we can keep
at one place.
{code}
[
{
"events": [
{
"id": "YARN_RM_CONTAINER_FINISHED",
"timestamp": 1475469588288,
"info": {
"YARN_CONTAINER_STATE": "COMPLETE",
"YARN_CONTAINER_EXIT_STATUS": 0,
"YARN_CONTAINER_DIAGNOSTICS_INFO": ""
}
},
{
"id": "YARN_RM_CONTAINER_CREATED",
"timestamp": 1475469570721,
"info": {
"YARN_CONTAINER_ALLOCATED_PORT": 25006,
"YARN_CONTAINER_ALLOCATED_MEMORY": 1024,
"YARN_CONTAINER_ALLOCATED_PRIORITY": 0,
"YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS":
"http://ctr-e29-1471411959733-0412-01-04.hwx.site:25008;,
"YARN_CONTAINER_ALLOCATED_HOST":
"ctr-e29-1471411959733-0412-01-04.hwx.site",
"YARN_CONTAINER_ALLOCATED_VCORE": 1
}
},
{
"id": "YARN_CONTAINER_CREATED",
"timestamp": -1,
"info": {}
},
{
"id": "YARN_CONTAINER_FINISHED",
"timestamp": -1,
"info": {
"YARN_CONTAINER_STATE": "RUNNING",
"YARN_CONTAINER_EXIT_STATUS": 0,
"YARN_CONTAINER_DIAGNOSTICS_INFO": ""
}
},
{
"id": "YARN_NM_CONTAINER_LOCALIZATION_FINISHED",
"timestamp": -1,
"info": {}
},
{
"id": "YARN_NM_CONTAINER_LOCALIZATION_STARTED",
"timestamp": -1,
"info": {}
}
],
"type": "YARN_CONTAINER",
"id": "container_e09_1475277121920_0010_01_01",
"createdtime": -1,
"info": {
"YARN_CONTAINER_ALLOCATED_PORT": 25006,
"UID":
"yarn-cluster!application_1475277121920_0010!YARN_CONTAINER!container_e09_1475277121920_0010_01_01",
"YARN_CONTAINER_ALLOCATED_MEMORY": 1024,
"SYSTEM_INFO_PARENT_ENTITY": {
"type": "YARN_APPLICATION_ATTEMPT",
"id": "appattempt_1475277121920_0010_01"
},
"YARN_CONTAINER_ALLOCATED_PRIORITY": "0",
"YARN_CONTAINER_ALLOCATED_HOST":
"ctr-e29-1471411959733-0412-01-04.hwx.site",
"YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS":
"ctr-e29-1471411959733-0412-01-04.hwx.site:0",
"YARN_CONTAINER_ALLOCATED_VCORE": 1
},
"configs": {},
"isrelatedto": {},
"relatesto": {}
}
]
{code}
bq. Container event publishing from RM is optional.
I see. But I think we should make it mandatory to publish from RM since some
information always get from RM like container_state. Otherwise we miss the this
information else container_state info should be published from NM. YARN-5156
removes from NM publisher.
> Retrospect yarn entity fields which are publishing in events info fields.
> -
>
> Key: YARN-5699
> URL: https://issues.apache.org/jira/browse/YARN-5699
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch
>
>
> Currently, all the container information are published at 2 places. Some of
> them are at entity info(top-level) and some are at event info.
> For containers, some of the event info should be published at container info
> level. For example : container exist status, container state, createdTime,
> finished time. These are general information to container required for
> container-report. So it is better to publish at top level info field.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551754#comment-15551754 ] Varun Saxena commented on YARN-5699: [~rohithsharma], thanks for the patch. In the patch I see you have removed container related info from NMTimelinePublisher. Is this intentional ? Container event publishing from RM is optional. And will be ideally be switched off due to the volume and impact on RM. > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[
https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551737#comment-15551737
]
Hadoop QA commented on YARN-5699:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} YARN-5699 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831941/0001-YARN-5699.patch |
| JIRA Issue | YARN-5699 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/13305/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Retrospect yarn entity fields which are publishing in events info fields.
> -
>
> Key: YARN-5699
> URL: https://issues.apache.org/jira/browse/YARN-5699
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-5699.patch
>
>
> Currently, all the container information are published at 2 places. Some of
> them are at entity info(top-level) and some are at event info.
> For containers, some of the event info should be published at container info
> level. For example : container exist status, container state, createdTime,
> finished time. These are general information to container required for
> container-report. So it is better to publish at top level info field.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
