[
https://issues.apache.org/jira/browse/SPARK-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15026858#comment-15026858
]
Steve Loughran commented on SPARK-6951:
---------------------------------------
SPARK-11373 proposes adding metrics to the history server; if the time to list
and load files is included then it'll help track this problem. my latest
SPARK-7889 adds timings & counts of the getAppUI() invocations in preparation
for the monitoring, but does nothing about listings
> History server slow startup if the event log directory is large
> ---------------------------------------------------------------
>
> Key: SPARK-6951
> URL: https://issues.apache.org/jira/browse/SPARK-6951
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.3.0
> Reporter: Matt Cheah
>
> I started my history server, then navigated to the web UI where I expected to
> be able to view some completed applications, but the webpage was not
> available. It turned out that the History Server was not finished parsing all
> of the event logs in the event log directory that I had specified. I had
> accumulated a lot of event logs from months of running Spark, so it would
> have taken a very long time for the History Server to crunch through them
> all. I purged the event log directory and started from scratch, and the UI
> loaded immediately.
> We should have a pagination strategy or parse the directory lazily to avoid
> needing to wait after starting the history server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
