[
https://issues.apache.org/jira/browse/SPARK-13914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15195932#comment-15195932
]
Sean Owen commented on SPARK-13914:
-----------------------------------
I don't think this is functionality Spark should implement. This is really just
an issue because your permanent storage is permanent and you want permanent
storage. There are a bunch of places to copy the files, and easy ways to do it,
and Spark wouldn't want to support them all.
> Add functionality to back up spark event logs
> ---------------------------------------------
>
> Key: SPARK-13914
> URL: https://issues.apache.org/jira/browse/SPARK-13914
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler
> Affects Versions: 1.6.0, 1.6.2, 2.0.0
> Reporter: Parag Chaudhari
>
> Spark event logs are usually stored in HDFS when running Spark on YARN. In a
> cloud environment, these HDFS files are often stored on the disks of
> ephemeral instances that could go away once the instances are terminated.
> Users may want to persist the event logs as the event happens for issue
> investigation and performance analysis before and after the cluster is
> terminated. The backup path can be managed by the spark users based on their
> needs. For example, some users may copy the event logs to a cloud storage
> service directly and keep them there forever. While some other users may want
> to store the event logs on local disks and back them up to a cloud storage
> service from time to time. Other users will not want to use the feature, so
> this feature should be off by default; users enable the feature when and only
> when they need it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
