[ https://issues.apache.org/jira/browse/SPARK-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359872#comment-15359872 ]
Weiqing Yang commented on SPARK-15923: -------------------------------------- Debugged in the cluster. Whether the cluster is secure or unsecure, this issue happens. And only applications in yarn-client mode have this issue. Detailed jira description: 1. yarn-client mode: Applications in yarn-client mode, they donot have 'attemptId' in their records, e.g: "id": "application_1465778870517_0001", "name": "Spark Pi", "attempts": [ {"startTime": "2016-06-13T01:07:16.958GMT", "endTime" : "2016-06-13T01:09:29.668GMT", "sparkUser" : "hrt_qa", "completed" : true } ] So when checking the web UI for executors’ information, the link used is http://<host>:18080/history/application_1465778870517_0001/executors/, which shows all the executors’ information. Note: it does not have attemptId inside the link. On the other hand, if calling the rest API: http://<host>:18080/api/v1/applications/application_1465778870517_0001/1/executors, it has attemptId "1" inside, and gets errors like "no such app" and "INFO ApplicationCache: Failed to load application attempt application_1465778870517_0001/Some(1)" . Instead, if you try the rest API: "http://<hostid>:18080/api/v1/applications/application_1465778870517_0001/executors", which has no attemptId inside, we can see all the executors’ information. 2. yarn-cluster mode: Applications in yarn-cluster mode. They do have 'attemptId' in their record, e.g.: "id" : "application_1465778870517_0002", "name" : "Spark Pi", "attempts" : [ {"attemptId": "1", "startTime" : "2016-06-13T01:12:48.797GMT", "endTime" : "2016-06-13T01:14:26.900GMT", "sparkUser" : "hrt_qa", "completed" : true } ] We can check executor information by web UI and rest API since both of them have attemptId “1”: http://<hostid>:18080/history/application_1465778870517_0002/1/executors/ http://<hostid>:18080/api/v1/applications/application_1465778870517_0002/1/executors Summary: When checking job/executor information by rest APIs, the "attemptId" is included inside. However, in yarn client mode, there will be no attempt ID. I am going to make a pull request for review. > Spark Application rest api returns "no such app: <appId>" > --------------------------------------------------------- > > Key: SPARK-15923 > URL: https://issues.apache.org/jira/browse/SPARK-15923 > Project: Spark > Issue Type: Bug > Affects Versions: 1.6.1 > Reporter: Yesha Vora > > Env : secure cluster > Scenario: > * Run SparkPi application in yarn-client or yarn-cluster mode > * After application finishes, check Spark HS rest api to get details like > jobs / executor etc. > {code} > http://<host>:18080/api/v1/applications/application_1465778870517_0001/1/executors{code} > > Rest api return HTTP Code: 404 and prints "HTTP Data: no such app: > application_1465778870517_0001" -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org