[
https://issues.apache.org/jira/browse/SPARK-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15026849#comment-15026849
]
Steve Loughran commented on SPARK-6951:
---------------------------------------
Thomas, I think you are right w.r.t simple fixes: add a metadata document to
summarise the ongoing app. Essentially the same stuff that I am pushing out to
the timeline server as summary info
# timestamp of start
# yarn attempt ID (allows for comparision with running apps and their state)
# timestamp of finished (if set). Unset ==> app is not finished.
# updated time
# spark version (I hope to use this in better incompatibility warnings)
There's a risk of inconsistency with the real log: what if the log saved as
finished but the metadata isn't in sync and tagged as open? Maybe when an app
is viewed and the full log pulled in any discrepancies could be corrected there
by saving a new metadata file.
> History server slow startup if the event log directory is large
> ---------------------------------------------------------------
>
> Key: SPARK-6951
> URL: https://issues.apache.org/jira/browse/SPARK-6951
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.3.0
> Reporter: Matt Cheah
>
> I started my history server, then navigated to the web UI where I expected to
> be able to view some completed applications, but the webpage was not
> available. It turned out that the History Server was not finished parsing all
> of the event logs in the event log directory that I had specified. I had
> accumulated a lot of event logs from months of running Spark, so it would
> have taken a very long time for the History Server to crunch through them
> all. I purged the event log directory and started from scratch, and the UI
> loaded immediately.
> We should have a pagination strategy or parse the directory lazily to avoid
> needing to wait after starting the history server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
