[
https://issues.apache.org/jira/browse/SPARK-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15889588#comment-15889588
]
Zheng Shao commented on SPARK-6951:
-----------------------------------
Did we consider using a distributed store to solve the scalability issue?
Single-node LevelDB will hit scalability issue soon.
> History server slow startup if the event log directory is large
> ---------------------------------------------------------------
>
> Key: SPARK-6951
> URL: https://issues.apache.org/jira/browse/SPARK-6951
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.3.0
> Reporter: Matt Cheah
>
> I started my history server, then navigated to the web UI where I expected to
> be able to view some completed applications, but the webpage was not
> available. It turned out that the History Server was not finished parsing all
> of the event logs in the event log directory that I had specified. I had
> accumulated a lot of event logs from months of running Spark, so it would
> have taken a very long time for the History Server to crunch through them
> all. I purged the event log directory and started from scratch, and the UI
> loaded immediately.
> We should have a pagination strategy or parse the directory lazily to avoid
> needing to wait after starting the history server.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
