Parag Chaudhari created SPARK-13914:
---------------------------------------
Summary: Add functionality to back up spark event logs
Key: SPARK-13914
URL: https://issues.apache.org/jira/browse/SPARK-13914
Project: Spark
Issue Type: Improvement
Components: Scheduler
Affects Versions: 1.6.0, 1.6.2, 2.0.0
Reporter: Parag Chaudhari
Spark event logs are usually stored in HDFS when running Spark on YARN. In a
cloud environment, these HDFS files are often stored on the disks of ephemeral
instances that could go away once the instances are terminated. Users may want
to persist the event logs as the event happens for issue investigation and
performance analysis before and after the cluster is terminated. The backup
path can be managed by the spark users based on their needs. For example, some
users may copy the event logs to a cloud storage service directly and keep them
there forever. While some other users may want to store the event logs on local
disks and back them up to a cloud storage service from time to time. Other
users will not want to use the feature, so this feature should be off by
default; users enable the feature when and only when they need it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
