[
https://issues.apache.org/jira/browse/SPARK-10187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-10187.
----------------------------------
Resolution: Incomplete
> Sometimes Web UI reports Application history not found
> ------------------------------------------------------
>
> Key: SPARK-10187
> URL: https://issues.apache.org/jira/browse/SPARK-10187
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.4.1
> Environment: CentOS Linux release 7.1.1503 (Core)
> Linux 3.10.0-229.4.2.el7.x86_64 #1 SMP Wed May 13 10:06:09 UTC 2015 x86_64
> x86_64 x86_64 GNU/Linux
> Reporter: Rares Vernica
> Priority: Minor
> Labels: bulk-closed
>
> On the Web UI home page {{http://localhost:8080/}} in the list of Completed
> Applications clicking on the Name of a job shows the history of that job.
> This works for most of the time. For some jobs, Spark returns:
> {quote}
> Application history not found (app-20150824104953-0018)
> No event logs found for application bid-00001-ph-1 in
> file:///data/disk1/spark/events. Did you specify the correct logging
> directory?
> {quote}
> All these jobs have the FINISHED state. {{spark-defaults.conf}} contains
> {quote}
> spark.eventLog.dir file:///data/disk1/spark/events
> spark.history.fs.logDirectory file:///data/disk1/spark/events
> {quote}
> {{/data/disk1/spark/events}} is a normal directory on the disk. This works
> fine for some jobs but not others (even jobs on the same Spark session).
> The Spark History UI shows correctly the history of the jobs that the Web UI
> cannot find the logs for. So, I believe the log directory is fine.
> Here is the HTML snippet from the Web UI home page, under the Completed
> Applications section. Notice how the UI finds the history for some jobs, but
> not others:
> {quote}
> ...
> <a href="app?appId=app-20150824110804-0019">app-20150824110804-0019</a>
> ...
> <a href="/history/app-20150824110804-0019">bid-00001-ph-2</a>
> ...
> <a
> href="app?appId=app-20150824104953-0018">app-20150824104953-0018</a>
> ...
> <a
> href="/history/not-found?msg=No+event+logs+found+for+application+bid-00001-ph-1+in+file%3A%2F%2F%2Fdata%2Fdisk1%2Fspark%2Fevents.+Did+you+specify+the+correct+logging+directory%3F&title=Application
> history not found (app-20150824104953-0018)">bid-00001-ph-1</a>
> ...
> <a
> href="app?appId=app-20150824104907-0017">app-20150824104907-0017</a>
> ...
> <a
> href="/history/not-found?msg=No+event+logs+found+for+application+bid-00001-ph-2+in+file%3A%2F%2F%2Fdata%2Fdisk1%2Fspark%2Fevents.+Did+you+specify+the+correct+logging+directory%3F&title=Application
> history not found (app-20150824104907-0017)">bid-00001-ph-2</a>
> ...
> <a
> href="app?appId=app-20150824103131-0016">app-20150824103131-0016</a>
> ...
> <a href="/history/app-20150824103131-0016">bid-00001-ph-1</a>
> {quote}
> These are sequential jobs and all the necessary files and directories in
> {{/data/disk1/spark/events}} and present and accessible and Spark History UI
> shows the history for all these jobs.
> I looked around JIRA for similar issues, I think this is related to
> SPARK-6107 and SPARK-6950 but not a duplicate. The jobs for which these
> manifests are all FINISHED for some time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
