[jira] [Commented] (YARN-3143) RM Apps REST API can return NPE or entries missing id and other fields
[
https://issues.apache.org/jira/browse/YARN-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311403#comment-14311403
]
Hudson commented on YARN-3143:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #98 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/98/])
RM Apps REST API can return NPE or entries missing id and other fields
--
Key: YARN-3143
URL: https://issues.apache.org/jira/browse/YARN-3143
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Affects Versions: 2.5.2
Reporter: Kendall Thrapp
Assignee: Jason Lowe
Fix For: 2.7.0
Attachments: YARN-3143.001.patch
I'm seeing intermittent null pointer exceptions being returned by
the YARN Apps REST API.
For example:
{code}
http://{cluster}:{port}/ws/v1/cluster/apps?finalStatus=UNDEFINED
{code}
JSON Response was:
{code}
{RemoteException:{exception:NullPointerException,javaClassName:java.lang.NullPointerException}}
{code}
At a glance appears to be only when we query for unfinished apps (i.e.
finalStatus=UNDEFINED).
Possibly related, when I do get back a list of apps, sometimes one or more of
the apps will be missing most of the fields, like id, name, user, etc., and
the fields that are present all have zero for the value.
For example:
{code}
{progress:0.0,clusterId:0,applicationTags:,startedTime:0,finishedTime:0,elapsedTime:0,allocatedMB:0,allocatedVCores:0,runningContainers:0,preemptedResourceMB:0,preemptedResourceVCores:0,numNonAMContainerPreempted:0,numAMContainerPreempted:0}
{code}
Let me know if there's any other information I can provide to help debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3143) RM Apps REST API can return NPE or entries missing id and other fields
[
https://issues.apache.org/jira/browse/YARN-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311373#comment-14311373
]
Hudson commented on YARN-3143:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #832 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/832/])
RM Apps REST API can return NPE or entries missing id and other fields
--
Key: YARN-3143
URL: https://issues.apache.org/jira/browse/YARN-3143
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Affects Versions: 2.5.2
Reporter: Kendall Thrapp
Assignee: Jason Lowe
Fix For: 2.7.0
Attachments: YARN-3143.001.patch
I'm seeing intermittent null pointer exceptions being returned by
the YARN Apps REST API.
For example:
{code}
http://{cluster}:{port}/ws/v1/cluster/apps?finalStatus=UNDEFINED
{code}
JSON Response was:
{code}
{RemoteException:{exception:NullPointerException,javaClassName:java.lang.NullPointerException}}
{code}
At a glance appears to be only when we query for unfinished apps (i.e.
finalStatus=UNDEFINED).
Possibly related, when I do get back a list of apps, sometimes one or more of
the apps will be missing most of the fields, like id, name, user, etc., and
the fields that are present all have zero for the value.
For example:
{code}
{progress:0.0,clusterId:0,applicationTags:,startedTime:0,finishedTime:0,elapsedTime:0,allocatedMB:0,allocatedVCores:0,runningContainers:0,preemptedResourceMB:0,preemptedResourceVCores:0,numNonAMContainerPreempted:0,numAMContainerPreempted:0}
{code}
Let me know if there's any other information I can provide to help debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3143) RM Apps REST API can return NPE or entries missing id and other fields
[
https://issues.apache.org/jira/browse/YARN-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311578#comment-14311578
]
Hudson commented on YARN-3143:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #95 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/95/])
RM Apps REST API can return NPE or entries missing id and other fields
--
Key: YARN-3143
URL: https://issues.apache.org/jira/browse/YARN-3143
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Affects Versions: 2.5.2
Reporter: Kendall Thrapp
Assignee: Jason Lowe
Fix For: 2.7.0
Attachments: YARN-3143.001.patch
I'm seeing intermittent null pointer exceptions being returned by
the YARN Apps REST API.
For example:
{code}
http://{cluster}:{port}/ws/v1/cluster/apps?finalStatus=UNDEFINED
{code}
JSON Response was:
{code}
{RemoteException:{exception:NullPointerException,javaClassName:java.lang.NullPointerException}}
{code}
At a glance appears to be only when we query for unfinished apps (i.e.
finalStatus=UNDEFINED).
Possibly related, when I do get back a list of apps, sometimes one or more of
the apps will be missing most of the fields, like id, name, user, etc., and
the fields that are present all have zero for the value.
For example:
{code}
{progress:0.0,clusterId:0,applicationTags:,startedTime:0,finishedTime:0,elapsedTime:0,allocatedMB:0,allocatedVCores:0,runningContainers:0,preemptedResourceMB:0,preemptedResourceVCores:0,numNonAMContainerPreempted:0,numAMContainerPreempted:0}
{code}
Let me know if there's any other information I can provide to help debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3143) RM Apps REST API can return NPE or entries missing id and other fields
[
https://issues.apache.org/jira/browse/YARN-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311546#comment-14311546
]
Hudson commented on YARN-3143:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #2030 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2030/])
RM Apps REST API can return NPE or entries missing id and other fields
--
Key: YARN-3143
URL: https://issues.apache.org/jira/browse/YARN-3143
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Affects Versions: 2.5.2
Reporter: Kendall Thrapp
Assignee: Jason Lowe
Fix For: 2.7.0
Attachments: YARN-3143.001.patch
I'm seeing intermittent null pointer exceptions being returned by
the YARN Apps REST API.
For example:
{code}
http://{cluster}:{port}/ws/v1/cluster/apps?finalStatus=UNDEFINED
{code}
JSON Response was:
{code}
{RemoteException:{exception:NullPointerException,javaClassName:java.lang.NullPointerException}}
{code}
At a glance appears to be only when we query for unfinished apps (i.e.
finalStatus=UNDEFINED).
Possibly related, when I do get back a list of apps, sometimes one or more of
the apps will be missing most of the fields, like id, name, user, etc., and
the fields that are present all have zero for the value.
For example:
{code}
{progress:0.0,clusterId:0,applicationTags:,startedTime:0,finishedTime:0,elapsedTime:0,allocatedMB:0,allocatedVCores:0,runningContainers:0,preemptedResourceMB:0,preemptedResourceVCores:0,numNonAMContainerPreempted:0,numAMContainerPreempted:0}
{code}
Let me know if there's any other information I can provide to help debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3143) RM Apps REST API can return NPE or entries missing id and other fields
[
https://issues.apache.org/jira/browse/YARN-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311660#comment-14311660
]
Hudson commented on YARN-3143:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #99 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/99/])
RM Apps REST API can return NPE or entries missing id and other fields
--
Key: YARN-3143
URL: https://issues.apache.org/jira/browse/YARN-3143
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Affects Versions: 2.5.2
Reporter: Kendall Thrapp
Assignee: Jason Lowe
Fix For: 2.7.0
Attachments: YARN-3143.001.patch
I'm seeing intermittent null pointer exceptions being returned by
the YARN Apps REST API.
For example:
{code}
http://{cluster}:{port}/ws/v1/cluster/apps?finalStatus=UNDEFINED
{code}
JSON Response was:
{code}
{RemoteException:{exception:NullPointerException,javaClassName:java.lang.NullPointerException}}
{code}
At a glance appears to be only when we query for unfinished apps (i.e.
finalStatus=UNDEFINED).
Possibly related, when I do get back a list of apps, sometimes one or more of
the apps will be missing most of the fields, like id, name, user, etc., and
the fields that are present all have zero for the value.
For example:
{code}
{progress:0.0,clusterId:0,applicationTags:,startedTime:0,finishedTime:0,elapsedTime:0,allocatedMB:0,allocatedVCores:0,runningContainers:0,preemptedResourceMB:0,preemptedResourceVCores:0,numNonAMContainerPreempted:0,numAMContainerPreempted:0}
{code}
Let me know if there's any other information I can provide to help debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3143) RM Apps REST API can return NPE or entries missing id and other fields
[
https://issues.apache.org/jira/browse/YARN-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311646#comment-14311646
]
Hudson commented on YARN-3143:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2049 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2049/])
RM Apps REST API can return NPE or entries missing id and other fields
--
Key: YARN-3143
URL: https://issues.apache.org/jira/browse/YARN-3143
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Affects Versions: 2.5.2
Reporter: Kendall Thrapp
Assignee: Jason Lowe
Fix For: 2.7.0
Attachments: YARN-3143.001.patch
I'm seeing intermittent null pointer exceptions being returned by
the YARN Apps REST API.
For example:
{code}
http://{cluster}:{port}/ws/v1/cluster/apps?finalStatus=UNDEFINED
{code}
JSON Response was:
{code}
{RemoteException:{exception:NullPointerException,javaClassName:java.lang.NullPointerException}}
{code}
At a glance appears to be only when we query for unfinished apps (i.e.
finalStatus=UNDEFINED).
Possibly related, when I do get back a list of apps, sometimes one or more of
the apps will be missing most of the fields, like id, name, user, etc., and
the fields that are present all have zero for the value.
For example:
{code}
{progress:0.0,clusterId:0,applicationTags:,startedTime:0,finishedTime:0,elapsedTime:0,allocatedMB:0,allocatedVCores:0,runningContainers:0,preemptedResourceMB:0,preemptedResourceVCores:0,numNonAMContainerPreempted:0,numAMContainerPreempted:0}
{code}
Let me know if there's any other information I can provide to help debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3143) RM Apps REST API can return NPE or entries missing id and other fields
[
https://issues.apache.org/jira/browse/YARN-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309949#comment-14309949
]
Jason Lowe commented on YARN-3143:
--
Thanks for the review, Kihwal! Committing this.
RM Apps REST API can return NPE or entries missing id and other fields
--
Key: YARN-3143
URL: https://issues.apache.org/jira/browse/YARN-3143
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Affects Versions: 2.5.2
Reporter: Kendall Thrapp
Assignee: Jason Lowe
Attachments: YARN-3143.001.patch
I'm seeing intermittent null pointer exceptions being returned by
the YARN Apps REST API.
For example:
{code}
http://{cluster}:{port}/ws/v1/cluster/apps?finalStatus=UNDEFINED
{code}
JSON Response was:
{code}
{RemoteException:{exception:NullPointerException,javaClassName:java.lang.NullPointerException}}
{code}
At a glance appears to be only when we query for unfinished apps (i.e.
finalStatus=UNDEFINED).
Possibly related, when I do get back a list of apps, sometimes one or more of
the apps will be missing most of the fields, like id, name, user, etc., and
the fields that are present all have zero for the value.
For example:
{code}
{progress:0.0,clusterId:0,applicationTags:,startedTime:0,finishedTime:0,elapsedTime:0,allocatedMB:0,allocatedVCores:0,runningContainers:0,preemptedResourceMB:0,preemptedResourceVCores:0,numNonAMContainerPreempted:0,numAMContainerPreempted:0}
{code}
Let me know if there's any other information I can provide to help debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3143) RM Apps REST API can return NPE or entries missing id and other fields
[
https://issues.apache.org/jira/browse/YARN-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310055#comment-14310055
]
Hudson commented on YARN-3143:
--
FAILURE: Integrated in Hadoop-trunk-Commit #7045 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7045/])
YARN-3143. RM Apps REST API can return NPE or entries missing id and other
fields. Contributed by Jason Lowe (jlowe: rev
da2fb2bc46bddf42d79c6d7664cbf0311973709e)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java
RM Apps REST API can return NPE or entries missing id and other fields
--
Key: YARN-3143
URL: https://issues.apache.org/jira/browse/YARN-3143
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Affects Versions: 2.5.2
Reporter: Kendall Thrapp
Assignee: Jason Lowe
Fix For: 2.7.0
Attachments: YARN-3143.001.patch
I'm seeing intermittent null pointer exceptions being returned by
the YARN Apps REST API.
For example:
{code}
http://{cluster}:{port}/ws/v1/cluster/apps?finalStatus=UNDEFINED
{code}
JSON Response was:
{code}
{RemoteException:{exception:NullPointerException,javaClassName:java.lang.NullPointerException}}
{code}
At a glance appears to be only when we query for unfinished apps (i.e.
finalStatus=UNDEFINED).
Possibly related, when I do get back a list of apps, sometimes one or more of
the apps will be missing most of the fields, like id, name, user, etc., and
the fields that are present all have zero for the value.
For example:
{code}
{progress:0.0,clusterId:0,applicationTags:,startedTime:0,finishedTime:0,elapsedTime:0,allocatedMB:0,allocatedVCores:0,runningContainers:0,preemptedResourceMB:0,preemptedResourceVCores:0,numNonAMContainerPreempted:0,numAMContainerPreempted:0}
{code}
Let me know if there's any other information I can provide to help debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3143) RM Apps REST API can return NPE or entries missing id and other fields
[
https://issues.apache.org/jira/browse/YARN-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310136#comment-14310136
]
Kendall Thrapp commented on YARN-3143:
--
Thanks [~jlowe] for debugging and the super quick patch and thanks [~eepayne]
and [~kihwal] for reviewing.
RM Apps REST API can return NPE or entries missing id and other fields
--
Key: YARN-3143
URL: https://issues.apache.org/jira/browse/YARN-3143
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Affects Versions: 2.5.2
Reporter: Kendall Thrapp
Assignee: Jason Lowe
Fix For: 2.7.0
Attachments: YARN-3143.001.patch
I'm seeing intermittent null pointer exceptions being returned by
the YARN Apps REST API.
For example:
{code}
http://{cluster}:{port}/ws/v1/cluster/apps?finalStatus=UNDEFINED
{code}
JSON Response was:
{code}
{RemoteException:{exception:NullPointerException,javaClassName:java.lang.NullPointerException}}
{code}
At a glance appears to be only when we query for unfinished apps (i.e.
finalStatus=UNDEFINED).
Possibly related, when I do get back a list of apps, sometimes one or more of
the apps will be missing most of the fields, like id, name, user, etc., and
the fields that are present all have zero for the value.
For example:
{code}
{progress:0.0,clusterId:0,applicationTags:,startedTime:0,finishedTime:0,elapsedTime:0,allocatedMB:0,allocatedVCores:0,runningContainers:0,preemptedResourceMB:0,preemptedResourceVCores:0,numNonAMContainerPreempted:0,numAMContainerPreempted:0}
{code}
Let me know if there's any other information I can provide to help debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3143) RM Apps REST API can return NPE or entries missing id and other fields
[
https://issues.apache.org/jira/browse/YARN-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310004#comment-14310004
]
Jason Lowe commented on YARN-3143:
--
My apologies, I also meant to thank Eric for the original review!
RM Apps REST API can return NPE or entries missing id and other fields
--
Key: YARN-3143
URL: https://issues.apache.org/jira/browse/YARN-3143
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Affects Versions: 2.5.2
Reporter: Kendall Thrapp
Assignee: Jason Lowe
Fix For: 2.7.0
Attachments: YARN-3143.001.patch
I'm seeing intermittent null pointer exceptions being returned by
the YARN Apps REST API.
For example:
{code}
http://{cluster}:{port}/ws/v1/cluster/apps?finalStatus=UNDEFINED
{code}
JSON Response was:
{code}
{RemoteException:{exception:NullPointerException,javaClassName:java.lang.NullPointerException}}
{code}
At a glance appears to be only when we query for unfinished apps (i.e.
finalStatus=UNDEFINED).
Possibly related, when I do get back a list of apps, sometimes one or more of
the apps will be missing most of the fields, like id, name, user, etc., and
the fields that are present all have zero for the value.
For example:
{code}
{progress:0.0,clusterId:0,applicationTags:,startedTime:0,finishedTime:0,elapsedTime:0,allocatedMB:0,allocatedVCores:0,runningContainers:0,preemptedResourceMB:0,preemptedResourceVCores:0,numNonAMContainerPreempted:0,numAMContainerPreempted:0}
{code}
Let me know if there's any other information I can provide to help debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3143) RM Apps REST API can return NPE or entries missing id and other fields
[
https://issues.apache.org/jira/browse/YARN-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307315#comment-14307315
]
Eric Payne commented on YARN-3143:
--
Thank you for fixing this issue, [~jlowe]. The patch LGTM. +1
I verified that the failing test seems to be a manifestation of YARN-2666 and
is not related to this patch.
RM Apps REST API can return NPE or entries missing id and other fields
--
Key: YARN-3143
URL: https://issues.apache.org/jira/browse/YARN-3143
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Affects Versions: 2.5.2
Reporter: Kendall Thrapp
Assignee: Jason Lowe
Attachments: YARN-3143.001.patch
I'm seeing intermittent null pointer exceptions being returned by
the YARN Apps REST API.
For example:
{code}
http://{cluster}:{port}/ws/v1/cluster/apps?finalStatus=UNDEFINED
{code}
JSON Response was:
{code}
{RemoteException:{exception:NullPointerException,javaClassName:java.lang.NullPointerException}}
{code}
At a glance appears to be only when we query for unfinished apps (i.e.
finalStatus=UNDEFINED).
Possibly related, when I do get back a list of apps, sometimes one or more of
the apps will be missing most of the fields, like id, name, user, etc., and
the fields that are present all have zero for the value.
For example:
{code}
{progress:0.0,clusterId:0,applicationTags:,startedTime:0,finishedTime:0,elapsedTime:0,allocatedMB:0,allocatedVCores:0,runningContainers:0,preemptedResourceMB:0,preemptedResourceVCores:0,numNonAMContainerPreempted:0,numAMContainerPreempted:0}
{code}
Let me know if there's any other information I can provide to help debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3143) RM Apps REST API can return NPE or entries missing id and other fields
[
https://issues.apache.org/jira/browse/YARN-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307381#comment-14307381
]
Jason Lowe commented on YARN-3143:
--
bq. Could you provide the RM logs, please ? That will help debug.
Here's what the RM log says when the NPE occurs with a finalStatus query:
{noformat}
2015-02-05 15:18:09,744 [1124535424@qtp-165859665-85345] WARN
webapp.GenericExceptionHandler: INTERNAL_SERVER_ERROR
java.lang.NullPointerException
{noformat}
That's all there is. No stacktrace or anything else. Nothing else in the logs
looks out of place around the time of the call. We also saw nothing of note in
the logs when the web services returned apps with missing fields, which aligns
with what I'm pretty confident is happening. The RM is removing applications
from the RMApps map just as the web services are trying to walk it. Given how
expensive it is to grab all the scheduler lock for all those applications on
this busy cluster, I'm not surprised that by the time the web services receives
the full list of application reports at least one of the apps has retired from
the RMApps collection.
RM Apps REST API can return NPE or entries missing id and other fields
--
Key: YARN-3143
URL: https://issues.apache.org/jira/browse/YARN-3143
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Affects Versions: 2.5.2
Reporter: Kendall Thrapp
Assignee: Jason Lowe
Attachments: YARN-3143.001.patch
I'm seeing intermittent null pointer exceptions being returned by
the YARN Apps REST API.
For example:
{code}
http://{cluster}:{port}/ws/v1/cluster/apps?finalStatus=UNDEFINED
{code}
JSON Response was:
{code}
{RemoteException:{exception:NullPointerException,javaClassName:java.lang.NullPointerException}}
{code}
At a glance appears to be only when we query for unfinished apps (i.e.
finalStatus=UNDEFINED).
Possibly related, when I do get back a list of apps, sometimes one or more of
the apps will be missing most of the fields, like id, name, user, etc., and
the fields that are present all have zero for the value.
For example:
{code}
{progress:0.0,clusterId:0,applicationTags:,startedTime:0,finishedTime:0,elapsedTime:0,allocatedMB:0,allocatedVCores:0,runningContainers:0,preemptedResourceMB:0,preemptedResourceVCores:0,numNonAMContainerPreempted:0,numAMContainerPreempted:0}
{code}
Let me know if there's any other information I can provide to help debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3143) RM Apps REST API can return NPE or entries missing id and other fields
[
https://issues.apache.org/jira/browse/YARN-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306255#comment-14306255
]
Xuan Gong commented on YARN-3143:
-
bq. The problem here is that we're trying to look at a pool of applications
asynchronously, therefore it is possible that we could get an app report for an
application that is gone by the time we call apps.get(report.getApplicationId).
That means rmapp can be null, and note that AppInfo's constructor effectively
does nothing if the RMApp argument is null. That explains the lack of all sorts
of fields by the time it gets to JSON.
That could be the issue. But [~kthrapp] Could you provide the RM logs, please ?
That will help debug.
RM Apps REST API can return NPE or entries missing id and other fields
--
Key: YARN-3143
URL: https://issues.apache.org/jira/browse/YARN-3143
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Affects Versions: 2.5.2
Reporter: Kendall Thrapp
Assignee: Jason Lowe
Attachments: YARN-3143.001.patch
I'm seeing intermittent null pointer exceptions being returned by
the YARN Apps REST API.
For example:
{code}
http://{cluster}:{port}/ws/v1/cluster/apps?finalStatus=UNDEFINED
{code}
JSON Response was:
{code}
{RemoteException:{exception:NullPointerException,javaClassName:java.lang.NullPointerException}}
{code}
At a glance appears to be only when we query for unfinished apps (i.e.
finalStatus=UNDEFINED).
Possibly related, when I do get back a list of apps, sometimes one or more of
the apps will be missing most of the fields, like id, name, user, etc., and
the fields that are present all have zero for the value.
For example:
{code}
{progress:0.0,clusterId:0,applicationTags:,startedTime:0,finishedTime:0,elapsedTime:0,allocatedMB:0,allocatedVCores:0,runningContainers:0,preemptedResourceMB:0,preemptedResourceVCores:0,numNonAMContainerPreempted:0,numAMContainerPreempted:0}
{code}
Let me know if there's any other information I can provide to help debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3143) RM Apps REST API can return NPE or entries missing id and other fields
[
https://issues.apache.org/jira/browse/YARN-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306169#comment-14306169
]
Jason Lowe commented on YARN-3143:
--
I think this is caused by this code in the RMWebServices.java:
{code}
final ConcurrentMapApplicationId, RMApp apps =
rm.getRMContext().getRMApps();
AppsInfo allApps = new AppsInfo();
for (ApplicationReport report : appReports) {
RMApp rmapp = apps.get(report.getApplicationId());
if (finalStatusQuery != null !finalStatusQuery.isEmpty()) {
FinalApplicationStatus.valueOf(finalStatusQuery);
if (!rmapp.getFinalApplicationStatus().toString()
.equalsIgnoreCase(finalStatusQuery)) {
continue;
}
}
AppInfo app = new AppInfo(rmapp, hasAccess(rmapp, hsr),
WebAppUtils.getHttpSchemePrefix(conf));
allApps.add(app);
{code}
The problem here is that we're trying to look at a pool of applications
asynchronously, therefore it is possible that we could get an app report for an
application that is gone by the time we call apps.get(report.getApplicationId).
That means rmapp can be null, and note that AppInfo's constructor effectively
does nothing if the RMApp argument is null. That explains the lack of all
sorts of fields by the time it gets to JSON.
So there are a couple of problems here:
# We should not be adding AppInfos if rmapp is null
# We can blindly dereference rmapp and generate an NPE if there's a final
status query
RM Apps REST API can return NPE or entries missing id and other fields
--
Key: YARN-3143
URL: https://issues.apache.org/jira/browse/YARN-3143
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Affects Versions: 2.5.2
Reporter: Kendall Thrapp
Assignee: Jason Lowe
I'm seeing intermittent null pointer exceptions being returned by
the YARN Apps REST API.
For example:
{code}
http://{cluster}:{port}/ws/v1/cluster/apps?finalStatus=UNDEFINED
{code}
JSON Response was:
{code}
{RemoteException:{exception:NullPointerException,javaClassName:java.lang.NullPointerException}}
{code}
At a glance appears to be only when we query for unfinished apps (i.e.
finalStatus=UNDEFINED).
Possibly related, when I do get back a list of apps, sometimes one or more of
the apps will be missing most of the fields, like id, name, user, etc., and
the fields that are present all have zero for the value.
For example:
{code}
{progress:0.0,clusterId:0,applicationTags:,startedTime:0,finishedTime:0,elapsedTime:0,allocatedMB:0,allocatedVCores:0,runningContainers:0,preemptedResourceMB:0,preemptedResourceVCores:0,numNonAMContainerPreempted:0,numAMContainerPreempted:0}
{code}
Let me know if there's any other information I can provide to help debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3143) RM Apps REST API can return NPE or entries missing id and other fields
[
https://issues.apache.org/jira/browse/YARN-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306312#comment-14306312
]
Hadoop QA commented on YARN-3143:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696592/YARN-3143.001.patch
against trunk revision 5f4ef2d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6514//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6514//console
This message is automatically generated.
RM Apps REST API can return NPE or entries missing id and other fields
--
Key: YARN-3143
URL: https://issues.apache.org/jira/browse/YARN-3143
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Affects Versions: 2.5.2
Reporter: Kendall Thrapp
Assignee: Jason Lowe
Attachments: YARN-3143.001.patch
I'm seeing intermittent null pointer exceptions being returned by
the YARN Apps REST API.
For example:
{code}
http://{cluster}:{port}/ws/v1/cluster/apps?finalStatus=UNDEFINED
{code}
JSON Response was:
{code}
{RemoteException:{exception:NullPointerException,javaClassName:java.lang.NullPointerException}}
{code}
At a glance appears to be only when we query for unfinished apps (i.e.
finalStatus=UNDEFINED).
Possibly related, when I do get back a list of apps, sometimes one or more of
the apps will be missing most of the fields, like id, name, user, etc., and
the fields that are present all have zero for the value.
For example:
{code}
{progress:0.0,clusterId:0,applicationTags:,startedTime:0,finishedTime:0,elapsedTime:0,allocatedMB:0,allocatedVCores:0,runningContainers:0,preemptedResourceMB:0,preemptedResourceVCores:0,numNonAMContainerPreempted:0,numAMContainerPreempted:0}
{code}
Let me know if there's any other information I can provide to help debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
