[
https://issues.apache.org/jira/browse/SPARK-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359872#comment-15359872
]
Weiqing Yang commented on SPARK-15923:
--------------------------------------
Debugged in the cluster. Whether the cluster is secure or unsecure, this issue
happens. And only applications in yarn-client mode have this issue.
Detailed jira description:
1. yarn-client mode:
Applications in yarn-client mode, they donot have 'attemptId' in their records,
e.g:
"id": "application_1465778870517_0001",
"name": "Spark Pi",
"attempts": [
{"startTime": "2016-06-13T01:07:16.958GMT", "endTime" :
"2016-06-13T01:09:29.668GMT", "sparkUser" : "hrt_qa", "completed" : true }
]
So when checking the web UI for executors’ information, the link used is
http://<host>:18080/history/application_1465778870517_0001/executors/, which
shows all the executors’ information. Note: it does not have attemptId inside
the link. On the other hand, if calling the rest API:
http://<host>:18080/api/v1/applications/application_1465778870517_0001/1/executors,
it has attemptId "1" inside, and gets errors like "no such app" and "INFO
ApplicationCache: Failed to load application attempt
application_1465778870517_0001/Some(1)" . Instead, if you try the rest API:
"http://<hostid>:18080/api/v1/applications/application_1465778870517_0001/executors",
which has no attemptId inside, we can see all the executors’ information.
2. yarn-cluster mode:
Applications in yarn-cluster mode. They do have 'attemptId' in their record,
e.g.:
"id" : "application_1465778870517_0002",
"name" : "Spark Pi",
"attempts" : [
{"attemptId": "1", "startTime" : "2016-06-13T01:12:48.797GMT", "endTime" :
"2016-06-13T01:14:26.900GMT", "sparkUser" : "hrt_qa", "completed" : true }
]
We can check executor information by web UI and rest API since both of them
have attemptId “1”:
http://<hostid>:18080/history/application_1465778870517_0002/1/executors/
http://<hostid>:18080/api/v1/applications/application_1465778870517_0002/1/executors
Summary:
When checking job/executor information by rest APIs, the "attemptId" is
included inside. However, in yarn client mode, there will be no attempt ID.
I am going to make a pull request for review.
> Spark Application rest api returns "no such app: <appId>"
> ---------------------------------------------------------
>
> Key: SPARK-15923
> URL: https://issues.apache.org/jira/browse/SPARK-15923
> Project: Spark
> Issue Type: Bug
> Affects Versions: 1.6.1
> Reporter: Yesha Vora
>
> Env : secure cluster
> Scenario:
> * Run SparkPi application in yarn-client or yarn-cluster mode
> * After application finishes, check Spark HS rest api to get details like
> jobs / executor etc.
> {code}
> http://<host>:18080/api/v1/applications/application_1465778870517_0001/1/executors{code}
>
> Rest api return HTTP Code: 404 and prints "HTTP Data: no such app:
> application_1465778870517_0001"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
