[
https://issues.apache.org/jira/browse/SPARK-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711226#comment-14711226
]
Thomas Graves commented on SPARK-6951:
--------------------------------------
I'm running Spark 1.4.1 history server in production and its still very slow to
start up and pick up new versions, especially when there are very large history
files. I haven't looked at the implementation but it seems like it would be
better for the history server to just read some metadata and display it in the
list and then when the user clicks on it load the entire thing. Especially for
very large ones. Or keep a certain number loaded in the background.
> History server slow startup if the event log directory is large
> ---------------------------------------------------------------
>
> Key: SPARK-6951
> URL: https://issues.apache.org/jira/browse/SPARK-6951
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.3.0
> Reporter: Matt Cheah
>
> I started my history server, then navigated to the web UI where I expected to
> be able to view some completed applications, but the webpage was not
> available. It turned out that the History Server was not finished parsing all
> of the event logs in the event log directory that I had specified. I had
> accumulated a lot of event logs from months of running Spark, so it would
> have taken a very long time for the History Server to crunch through them
> all. I purged the event log directory and started from scratch, and the UI
> loaded immediately.
> We should have a pagination strategy or parse the directory lazily to avoid
> needing to wait after starting the history server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
