[
https://issues.apache.org/jira/browse/SPARK-13914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196117#comment-15196117
]
Sean Owen commented on SPARK-13914:
-----------------------------------
This is just a problem because you have an ephemeral FS, but you want it to be
durable.
If you want to do this anyway you can and should accomplish copying with
standard tools for the job -- cron + rsync?
I do not support this, no.
> Add functionality to back up spark event logs
> ---------------------------------------------
>
> Key: SPARK-13914
> URL: https://issues.apache.org/jira/browse/SPARK-13914
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler
> Affects Versions: 1.6.0, 1.6.2, 2.0.0
> Reporter: Parag Chaudhari
>
> Spark event logs are usually stored in HDFS when running Spark on YARN. In a
> cloud environment, these HDFS files are often stored on the disks of
> ephemeral instances that could go away once the instances are terminated.
> Users may want to persist the event logs as the event happens for issue
> investigation and performance analysis before and after the cluster is
> terminated. The backup path can be managed by the spark users based on their
> needs. For example, some users may copy the event logs to a cloud storage
> service directly and keep them there forever. While some other users may want
> to store the event logs on local disks and back them up to a cloud storage
> service from time to time. Other users will not want to use the feature, so
> this feature should be off by default; users enable the feature when and only
> when they need it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
