Imran Rashid reassigned SPARK-6951:

    Assignee: Marcelo Vanzin

> History server slow startup if the event log directory is large
> ---------------------------------------------------------------
>                 Key: SPARK-6951
>                 URL: https://issues.apache.org/jira/browse/SPARK-6951
>             Project: Spark
>          Issue Type: Bug
>          Components: Web UI
>    Affects Versions: 1.3.0
>            Reporter: Matt Cheah
>            Assignee: Marcelo Vanzin
>            Priority: Major
>             Fix For: 2.4.0
> I started my history server, then navigated to the web UI where I expected to 
> be able to view some completed applications, but the webpage was not 
> available. It turned out that the History Server was not finished parsing all 
> of the event logs in the event log directory that I had specified. I had 
> accumulated a lot of event logs from months of running Spark, so it would 
> have taken a very long time for the History Server to crunch through them 
> all. I purged the event log directory and started from scratch, and the UI 
> loaded immediately.
> We should have a pagination strategy or parse the directory lazily to avoid 
> needing to wait after starting the history server.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to