[
https://issues.apache.org/jira/browse/SPARK-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152758#comment-14152758
]
Matt Cheah edited comment on SPARK-1860 at 9/30/14 3:14 AM:
------------------------------------------------------------
I agree we should focus the scope on cleaning up things that have successfully
finished. Preserving state is beneficial in erroneous cases.
However, should it not be the case that when an Executor shuts down, it cleans
up all of the files it created? As you stated, the Worker doesn't know where a
particular Executor is storing its data, but the Executor should know where it
is storing its own data, and be managing it and cleaning up when completed.
This is regardless of the distinction between application data and shuffle data.
The Executor class has a record of the files and jars added through the
SparkContext (currentFiles and currentJars fields) for that Executor's use, and
these should naturally expire and be cleaned up when the Executor terminates.
was (Author: mcheah):
I agree we should focus the scope on cleaning up things that have successfully
finished.
However, should it not be the case that when an Executor shuts down, it cleans
up all of the files it created? As you stated, the Worker doesn't know where a
particular Executor is storing its data, but the Executor should know where it
is storing its own data, and be managing it and cleaning up when completed.
This is regardless of the distinction between application data and shuffle data.
The Executor class has a record of the files and jars added through the
SparkContext (currentFiles and currentJars fields) for that Executor's use, and
these should naturally expire and be cleaned up when the Executor terminates.
> Standalone Worker cleanup should not clean up running executors
> ---------------------------------------------------------------
>
> Key: SPARK-1860
> URL: https://issues.apache.org/jira/browse/SPARK-1860
> Project: Spark
> Issue Type: Bug
> Components: Deploy
> Affects Versions: 1.0.0
> Reporter: Aaron Davidson
> Priority: Blocker
>
> The default values of the standalone worker cleanup code cleanup all
> application data every 7 days. This includes jars that were added to any
> executors that happen to be running for longer than 7 days, hitting streaming
> jobs especially hard.
> Executor's log/data folders should not be cleaned up if they're still
> running. Until then, this behavior should not be enabled by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
