[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785510#comment-16785510
]
Prabhu Joseph commented on YARN-8132:
-
[~bibinchundatt] RMAppImpl can publish the final application status based upon
the state it transitions to when app is finished. It publishes finishTime
similarly. This won't have the above issue. Can you review the v2 patch.
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch,
> YARN-8132-003.patch, YARN-8132-004.patch, YARN-8132-branch-3.1.001.patch,
> YARN-8132-branch-3.2.001.patch, YARN-8132-branch-3.2.002.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783297#comment-16783297
]
Hudson commented on YARN-8132:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16115 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16115/])
Revert "YARN-8132. Final Status of applications shown as UNDEFINED in
(bibinchundatt: rev 15098df744deb045c0a33b095d45728d5b2af454)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch,
> YARN-8132-003.patch, YARN-8132-004.patch, YARN-8132-branch-3.1.001.patch,
> YARN-8132-branch-3.2.001.patch, YARN-8132-branch-3.2.002.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783284#comment-16783284
]
Bibin A Chundatt commented on YARN-8132:
Reverted from trunk,branch-3.1,branch-3.2
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch,
> YARN-8132-003.patch, YARN-8132-004.patch, YARN-8132-branch-3.1.001.patch,
> YARN-8132-branch-3.2.001.patch, YARN-8132-branch-3.2.002.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783265#comment-16783265
]
Bibin A Chundatt commented on YARN-8132:
[~Prabhu Joseph]
Looking at the code again ..Current fix could give wrong FinalStatusReport for
RM Application State too..
Attempt1 -- FAILED
Attemtp2 -- RUNNING
During the new attempt 2 launch time ie, Attempt1 to Attempt2 FinalStatus of
application will be shown as FAILED.
I will reopen the jira and revert from all branches
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch,
> YARN-8132-003.patch, YARN-8132-004.patch, YARN-8132-branch-3.1.001.patch,
> YARN-8132-branch-3.2.001.patch, YARN-8132-branch-3.2.002.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782340#comment-16782340
]
Prabhu Joseph commented on YARN-8132:
-
[~Rakesh_Shah] It gets triggered at {{RMAppAttemptEventType.FAIL}} when yarn
client calls failApplicationAttempt (yarn application -fail). When job fails
due to tasks failing, the AM UnregisterEvent will set finalApplicationStatus to
FAILED. But have missed for failure cases like AM Crash, AM Expire where AM
UnregisterEvent won't be present.
[~bibinchundatt] The given fix works for Killed cases (including job timeout)
and failure cases like - Tasks failing, Client initiates failApplicationAttempt
but did not for AM Crash and AM Expire. Can we handle in a separate Jira or
continue with this one.
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch,
> YARN-8132-003.patch, YARN-8132-004.patch, YARN-8132-branch-3.1.001.patch,
> YARN-8132-branch-3.2.001.patch, YARN-8132-branch-3.2.002.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781599#comment-16781599
]
Rakesh Shah commented on YARN-8132:
---
Hi [~Prabhu Joseph] in which scenerio the below fail event gets triggered.
case FAIL:
status = FinalApplicationStatus.FAILED;
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch,
> YARN-8132-003.patch, YARN-8132-004.patch, YARN-8132-branch-3.1.001.patch,
> YARN-8132-branch-3.2.001.patch, YARN-8132-branch-3.2.002.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774113#comment-16774113
]
Prabhu Joseph commented on YARN-8132:
-
Branch-3.2 test case failures are not related, reported YARN-9325 and YARN-9324
to handle the same,
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch,
> YARN-8132-003.patch, YARN-8132-004.patch, YARN-8132-branch-3.1.001.patch,
> YARN-8132-branch-3.2.001.patch, YARN-8132-branch-3.2.002.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774036#comment-16774036
]
Hadoop QA commented on YARN-8132:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
51s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 226 unchanged - 2 fixed = 226 total (was 228)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 40s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}121m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestSchedulingRequestContainerAllocation
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestSchedulingRequestContainerAllocationAsync
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueManagementDynamicEditPolicy
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:63396be |
| JIRA Issue | YARN-8132 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959590/YARN-8132-branch-3.2.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 182648abc914 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.2 / c2ef443 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774008#comment-16774008
]
Hadoop QA commented on YARN-8132:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
3s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} branch-3.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 222 unchanged - 2 fixed = 222 total (was 224)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 50s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 65m
43s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:080e9d0 |
| JIRA Issue | YARN-8132 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959580/YARN-8132-branch-3.1.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux b94763e0cdd3 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.1 / 704330a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23470/testReport/ |
| Max. process+thread count | 904 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773924#comment-16773924
]
Hadoop QA commented on YARN-8132:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
39s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
26s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
25s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 25s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 226 unchanged - 2 fixed = 228 total (was 228)
{color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
28s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 4m
2s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 27s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:63396be |
| JIRA Issue | YARN-8132 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959577/YARN-8132-branch-3.2.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 58b18db995af 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.2 / ae8839e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-YARN-Build/23469/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| compile |
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773895#comment-16773895
]
Prabhu Joseph commented on YARN-8132:
-
[~bibinchundatt] Attached patch on top of branch-3.2 and branch-3.1.
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch,
> YARN-8132-003.patch, YARN-8132-004.patch, YARN-8132-branch-3.1.001.patch,
> YARN-8132-branch-3.2.001.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773834#comment-16773834
]
Bibin A Chundatt commented on YARN-8132:
[~Prabhu Joseph] Committed to trunk.
For branch-3.2 and branch 3.1 Could you please upload patch.
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch,
> YARN-8132-003.patch, YARN-8132-004.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773838#comment-16773838
]
Prabhu Joseph commented on YARN-8132:
-
Thanks [~bibinchundatt], will upload.
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch,
> YARN-8132-003.patch, YARN-8132-004.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773795#comment-16773795
]
Hudson commented on YARN-8132:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16015 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16015/])
YARN-8132. Final Status of applications shown as UNDEFINED in ATS app
(bibinchundatt: rev a63c358b78ad1d4765d7f0b2d595c49b693efb3f)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch,
> YARN-8132-003.patch, YARN-8132-004.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772933#comment-16772933
]
Prabhu Joseph commented on YARN-8132:
-
Thanks [~bibinchundatt] for the review. Have reported YARN-9315
({{TestCapacitySchedulerMetrics}}) and YARN-9316
({{TestPlacementConstraintsUtil}}) - both test cases are failing intermittent.
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch,
> YARN-8132-003.patch, YARN-8132-004.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772926#comment-16772926
]
Bibin A Chundatt commented on YARN-8132:
Over all patch looks good to me.
Will check in by tomorrow if no objections.
[~Prabhu Joseph] Any jira to track TestPlacementConstraintsUtil failure ?
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch,
> YARN-8132-003.patch, YARN-8132-004.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772861#comment-16772861
]
Prabhu Joseph commented on YARN-8132:
-
Test case failures are not related. {{TestCapacitySchedulerMetrics}} fails
intermittently as assert check happens before the allocate completes.
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch,
> YARN-8132-003.patch, YARN-8132-004.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772799#comment-16772799
]
Hadoop QA commented on YARN-8132:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 225 unchanged - 2 fixed = 225 total (was 227)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 43s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 59s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}145m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.constraint.TestPlacementConstraintsUtil
|
| | hadoop.yarn.server.resourcemanager.TestCapacitySchedulerMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8132 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959380/YARN-8132-004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e6d86bb5b20b 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1d30fd9 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/23449/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772665#comment-16772665
]
Prabhu Joseph commented on YARN-8132:
-
[~bibinchundatt] The existing test case TestRMAppTransitions#testAppNewKill
covers the scenario. The currentAttempt is not created (Null) and the
StateMachine currentState is transitioned properly to KILLED. The issue happens
only when the job is killed after attempt is created as the attempt finalStatus
is not updated.
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch,
> YARN-8132-003.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772635#comment-16772635
]
Prabhu Joseph commented on YARN-8132:
-
[~bibinchundatt] Yes, working on it, will update.
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch,
> YARN-8132-003.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772284#comment-16772284
]
Bibin A Chundatt commented on YARN-8132:
Thank you [~Prabhu Joseph] for patch.
Latest patch fixes issue when the attempt is available and application is
killed.
Could you add a test case to verify FINAL Status at TIMELINE when application
is KILLED before attempt is created.
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch,
> YARN-8132-003.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771372#comment-16771372
]
Hadoop QA commented on YARN-8132:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 147 unchanged - 2 fixed = 149 total (was 149)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 45s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 19s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}155m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.rmapp.attempt.TestRMAppAttemptTransitions |
| | hadoop.yarn.server.resourcemanager.TestApplicationMasterServiceCapacity |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8132 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959154/YARN-8132-003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 18be9c8c80a4 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 920a896 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/23439/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770804#comment-16770804
]
Rakesh Shah commented on YARN-8132:
---
Thanks for the reply,
Actually [~Prabhu Joseph] cuurentAttempt will not be null when job is
failed/killed. It is like when it tries to get the finalStatus using
currentAttempt.getFinalApplicationStatus() it returns null
because finalStatus will not be updated in case of job fail/kill
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770800#comment-16770800
]
Prabhu Joseph commented on YARN-8132:
-
[~Rakesh_Shah] Yes that looks fine but {{RMAppImpl}} does not have attempt
information i.e, {{currentAttempt}} is null when job is failed / killed and so
{{RMAppImpl}} relies on the current state to get Final Status.
{code}
@Override
public FinalApplicationStatus getFinalApplicationStatus() {
if (currentAttempt != null
&& currentAttempt.getFinalApplicationStatus() != null) {
return currentAttempt.getFinalApplicationStatus();
}
return createFinalApplicationStatus(this.stateMachine.getCurrentState());
}
{code}
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770794#comment-16770794
]
Rakesh Shah commented on YARN-8132:
---
Hi all,
Can we solve in these by setting the finalStatus variable as global in
RMAppAttemptImpl class which is already there.
That is by removing the local declaration of finalStatus in
*rememberTargetTransitionsAndStoreState*() method in RMAppAttemptImpl class.
and setting the finalStatus to failed and killed whenever they come into-
# KILL
# FAIL
# CONTAINER_FINISHED switch case. Since the value is only set when the
UNREGISTERED of APP Master happens.
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770042#comment-16770042
]
Hadoop QA commented on YARN-8132:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 39s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 0s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}146m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8132 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12958960/YARN-8132-002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 7056cc78daa9 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / dde0ab5 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/23422/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23422/testReport/ |
| Max. process+thread count | 882 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770017#comment-16770017
]
Prabhu Joseph commented on YARN-8132:
-
Hi [~haibochen], Can you review the patch for this jira when you get time.
Below is the analysis
When the Job is Finished, and if Suceeded - the current attempt will exist and
will provide the
right FinalApplicationStatus.
And if Failed / Killed, the current attempt does not exist and the
FinalApplicationStatus gets
from current state (not) (finished / failed / killed) will become UNDEFINED.
The fix gets the
FinalApplicationStatus from the final state it will transitionTo.
FinalState Finished will be either FinalApplicationStatus SUCCEEDED or FAILED.
FinalState Failed / Killed will be same for FinalApplicationStatus.
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8132-001.patch, YARN-8132-002.patch
>
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769684#comment-16769684
]
Hadoop QA commented on YARN-8132:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 148 unchanged - 0 fixed = 149 total (was 148)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 43s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 42s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8132 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12958916/YARN-8132-001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 829db1819c70 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d10444e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/23421/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/23421/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
[jira] [Commented] (YARN-8132) Final Status of applications shown as UNDEFINED in ATS app queries
[
https://issues.apache.org/jira/browse/YARN-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470462#comment-16470462
]
Haibo Chen commented on YARN-8132:
--
Both YARN_APPLICATION_STATE and YARN_APPLICATON_FINAL_STATUS are published in
RMAppImp.FinalTransition
The finanal status update only happens after the transition finishes. Hence, it
is UNDEFINED.
We could consider always keeping them the same to avoid confusion.
> Final Status of applications shown as UNDEFINED in ATS app queries
> --
>
> Key: YARN-8132
> URL: https://issues.apache.org/jira/browse/YARN-8132
> Project: Hadoop YARN
> Issue Type: Bug
> Components: ATSv2, timelineservice
>Reporter: Charan Hebri
>Priority: Major
>
> Final Status is shown as UNDEFINED for applications that are KILLED/FAILED. A
> sample request/response with INFO field for an application,
> {noformat}
> 2018-04-09 13:10:02,126 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1693)) - Received URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO from user
> hrt_qa
> 2018-04-09 13:10:02,156 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:getApp(1716)) - Processed URL
> /ws/v2/timeline/apps/application_1523259757659_0003?fields=INFO (Took 30
> ms.){noformat}
> {noformat}
> {
> "metrics": [],
> "events": [],
> "createdtime": 1523263360719,
> "idprefix": 0,
> "id": "application_1523259757659_0003",
> "type": "YARN_APPLICATION",
> "info": {
> "YARN_APPLICATION_CALLER_CONTEXT": "CLI",
> "YARN_APPLICATION_DIAGNOSTICS_INFO": "Application
> application_1523259757659_0003 was killed by user xxx_xx at XXX.XXX.XXX.XXX",
> "YARN_APPLICATION_FINAL_STATUS": "UNDEFINED",
> "YARN_APPLICATION_NAME": "Sleep job",
> "YARN_APPLICATION_USER": "hrt_qa",
> "YARN_APPLICATION_UNMANAGED_APPLICATION": false,
> "FROM_ID":
> "yarn-cluster!hrt_qa!test_flow!1523263360719!application_1523259757659_0003",
> "UID": "yarn-cluster!application_1523259757659_0003",
> "YARN_APPLICATION_VIEW_ACLS": " ",
> "YARN_APPLICATION_SUBMITTED_TIME": 1523263360718,
> "YARN_AM_CONTAINER_LAUNCH_COMMAND": [
> "$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir= -Dyarn.app.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
> -Dhdp.version=3.0.0.0-1163 -Xmx819m -Dhdp.version=3.0.0.0-1163
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/stdout
> 2>/stderr "
> ],
> "YARN_APPLICATION_QUEUE": "default",
> "YARN_APPLICATION_TYPE": "MAPREDUCE",
> "YARN_APPLICATION_PRIORITY": 0,
> "YARN_APPLICATION_LATEST_APP_ATTEMPT":
> "appattempt_1523259757659_0003_01",
> "YARN_APPLICATION_TAGS": [
> "timeline_flow_name_tag:test_flow"
> ],
> "YARN_APPLICATION_STATE": "KILLED"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{noformat}
> This is different to what the Resource Manager reports. For KILLED
> applications the final status is KILLED and for FAILED applications it is
> FAILED. This behavior is seen in ATSv2 as well as older versions of ATS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
