[
https://issues.apache.org/jira/browse/SPARK-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-5522.
------------------------------
Resolution: Fixed
Looks resolved by https://github.com/apache/spark/pull/4525 but just never got
marked as such.
> Accelerate the History Server start
> -----------------------------------
>
> Key: SPARK-5522
> URL: https://issues.apache.org/jira/browse/SPARK-5522
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, Web UI
> Affects Versions: 1.0.0
> Reporter: Liangliang Gu
> Assignee: Liangliang Gu
> Fix For: 1.4.0
>
>
> When starting the history server, all the log files will be fetched and
> parsed in order to get the applications' meta data e.g. App Name, Start Time,
> Duration, etc. In our production cluster, there exist 2600 log files (160G)
> in HDFS and it costs 3 hours to restart the history server, which is a little
> bit too long for us.
> It would be better, if the history server can show logs with missing
> information during start-up and fill the missing information after fetching
> and parsing a log file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
