[
https://issues.apache.org/jira/browse/SPARK-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15900871#comment-15900871
]
Cui Xixin commented on SPARK-6951:
----------------------------------
In my case,the inprogress file is the main reason, so if the inprogress file is
necessary for the history server, or if the inprogress file can be cleaned up
with the 'spark.history.fs.cleaner.maxAge'
> History server slow startup if the event log directory is large
> ---------------------------------------------------------------
>
> Key: SPARK-6951
> URL: https://issues.apache.org/jira/browse/SPARK-6951
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.3.0
> Reporter: Matt Cheah
>
> I started my history server, then navigated to the web UI where I expected to
> be able to view some completed applications, but the webpage was not
> available. It turned out that the History Server was not finished parsing all
> of the event logs in the event log directory that I had specified. I had
> accumulated a lot of event logs from months of running Spark, so it would
> have taken a very long time for the History Server to crunch through them
> all. I purged the event log directory and started from scratch, and the UI
> loaded immediately.
> We should have a pagination strategy or parse the directory lazily to avoid
> needing to wait after starting the history server.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
