[
https://issues.apache.org/jira/browse/SPARK-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14728864#comment-14728864
]
Chris Bannister commented on SPARK-9708:
----------------------------------------
When the executor is killed by Mesos it does not immediately clean up the
sandbox dir, it waits for a GC time period based on total disk usage in the
work_dir, im not entirely sure what will happen if the executor is stopped, if
the data is still readable by external applications.
Regarding spark.local.dir, as far as I understand it when running in YARN this
is overridden by the YARN config, I intended to do something similar here.
Would it be better to add a config option to explicitly enable this behaviour?
> Spark should create local temporary directories in Mesos sandbox when
> launched with Mesos
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-9708
> URL: https://issues.apache.org/jira/browse/SPARK-9708
> Project: Spark
> Issue Type: Bug
> Components: Mesos
> Reporter: Timothy Chen
>
> Currently Spark creates temporary directories with
> Utils.getConfiguredLocalDirs, and it writes to YARN directories if YARN is
> detected, otherwise just writes in a temporary directory in the host.
> However, Mesos does create a directory per task and ideally Spark should use
> that directory to create its local temporary directories since it then can be
> cleaned up when the task is gone and not left on the host or cleaned until
> reboot.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
