Thomas Graves created SPARK-6751:
------------------------------------
Summary: Spark History Server support multiple application attempts
Key: SPARK-6751
URL: https://issues.apache.org/jira/browse/SPARK-6751
Project: Spark
Issue Type: Improvement
Components: Spark Core, YARN
Affects Versions: 1.3.0
Reporter: Thomas Graves
Spark on Yarn supports running multiple application attempts (configurable
number) in case the first (or second..) attempts fail. The Spark History
server only supports one history file though. Under the default configs it
keeps the first attempts history file. You can set the undocumented config
spark.eventLog.overwrite to allow the follow on attempts to overwrite the first
attempts history file.
Note that in spark 1.2 not having the overwrite config set causes any following
attempts to actually fail to run, in spark 1.3 they run and you just see a
warning at the end of the attempts.
It would be really nice to have an option that keeps all the attempts history
files. This way a user can go back and look at each one individually.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
