Matt Cheah created SPARK-6951:
---------------------------------
Summary: History server slow startup if the event log directory is
large
Key: SPARK-6951
URL: https://issues.apache.org/jira/browse/SPARK-6951
Project: Spark
Issue Type: Bug
Components: Web UI
Affects Versions: 1.3.0
Reporter: Matt Cheah
I started my history server, then navigated to the web UI where I expected to
be able to view some completed applications, but the webpage was not available.
It turned out that the History Server was not finished parsing all of the event
logs in the event log directory that I had specified. I had accumulated a lot
of event logs from months of running Spark, so it would have taken a very long
time for the History Server to crunch through them all. I purged the event log
directory and started from scratch, and the UI loaded immediately.
We should have a pagination strategy or parse the directory lazily to avoid
needing to wait after starting the history server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
