[
https://issues.apache.org/jira/browse/SPARK-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-6033.
------------------------------
Resolution: Fixed
Fix Version/s: 1.2.2
1.1.2
1.3.0
1.0.3
Assignee: pengxu
> the description abou the "spark.worker.cleanup.enabled" is not matched with
> the code
> ------------------------------------------------------------------------------------
>
> Key: SPARK-6033
> URL: https://issues.apache.org/jira/browse/SPARK-6033
> Project: Spark
> Issue Type: Documentation
> Components: Documentation
> Affects Versions: 1.2.0, 1.2.1
> Reporter: pengxu
> Assignee: pengxu
> Priority: Minor
> Fix For: 1.0.3, 1.3.0, 1.1.2, 1.2.2
>
>
> Some error about the section _Cluster Launch Scripts_ in the
> http://spark.apache.org/docs/latest/spark-standalone.html
> In the description about the property spark.worker.cleanup.enabled, it states
> that *all the directory* under the work dir will be removed whether the
> application is running or not.
> After checking the implementation in the code level, I found that +only the
> stopped application+ dirs would be removed. So the description in the
> document is incorrect.
> the code implementation in worker.scala
> {code: title=WorkDirCleanup}
> case WorkDirCleanup =>
> // Spin up a separate thread (in a future) to do the dir cleanup; don't
> tie up worker actor
> val cleanupFuture = concurrent.future {
> val appDirs = workDir.listFiles()
> if (appDirs == null) {
> throw new IOException("ERROR: Failed to list files in " + appDirs)
> }
> appDirs.filter { dir =>
> // the directory is used by an application - check that the
> application is not running
> // when cleaning up
> val appIdFromDir = dir.getName
> val isAppStillRunning =
> executors.values.map(_.appId).contains(appIdFromDir)
> dir.isDirectory && !isAppStillRunning &&
> !Utils.doesDirectoryContainAnyNewFiles(dir, APP_DATA_RETENTION_SECS)
> }.foreach { dir =>
> logInfo(s"Removing directory: ${dir.getPath}")
> Utils.deleteRecursively(dir)
> }
> }
> cleanupFuture onFailure {
> case e: Throwable =>
> logError("App dir cleanup failed: " + e.getMessage, e)
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
