[
https://issues.apache.org/jira/browse/SPARK-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14001151#comment-14001151
]
Andrew Ash commented on SPARK-1860:
-----------------------------------
[~mkim] is going to take a look at this after discussion at
https://issues.apache.org/jira/browse/SPARK-1154
I think the correct fix as Patrick outlines would be:
{code}
// pseudocode
for folder in onDiskFolders:
if folder is owned by a running application:
continue
if folder contains any folder/file (recursively) that is more recently
touched (mtime) than the TTS:
continue
cleanUp(folder)
{code}
Schedule that to run periodically (interval configured by setting) and this
should be all fixed up.
Is that right?
An alternative approach could be to have executor clean up the application's
work directory when the application terminates, but un-clean executor shutdown
could still leave work directories around so a TTL approach still needs to be
included as well.
> Standalone Worker cleanup should not clean up running applications
> ------------------------------------------------------------------
>
> Key: SPARK-1860
> URL: https://issues.apache.org/jira/browse/SPARK-1860
> Project: Spark
> Issue Type: Bug
> Components: Deploy
> Affects Versions: 1.0.0
> Reporter: Aaron Davidson
> Priority: Critical
> Fix For: 1.1.0
>
>
> The default values of the standalone worker cleanup code cleanup all
> application data every 7 days. This includes jars that were added to any
> applications that happen to be running for longer than 7 days, hitting
> streaming jobs especially hard.
> Applications should not be cleaned up if they're still running. Until then,
> this behavior should not be enabled by default.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
