Mars Gu created SPARK-5522:
------------------------------
Summary: Accelerate the Histroty Server start
Key: SPARK-5522
URL: https://issues.apache.org/jira/browse/SPARK-5522
Project: Spark
Issue Type: Improvement
Components: Spark Core
Reporter: Mars Gu
When starting the history server, all the log files will be fetched and parsed
in order to get the applications' meta data e.g. App Name, Start Time,
Duration, etc. In our production cluster, there exist 2600 log files (160G) in
HDFS and it costs 3 hours to restart the history server, which is a little bit
too long for us.
It would be better, if the history server does not fetch all the log files but
only the meta data during start-up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
