David Shar created SPARK-18027:
----------------------------------
Summary: .sparkStaging not clean on error
Key: SPARK-18027
URL: https://issues.apache.org/jira/browse/SPARK-18027
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 1.6.0
Reporter: David Shar
Hi,
It seems that SPARK-7705 didn't fix all issues with .sparkStaging folder
cleanup.
in Client.scala:monitorApplication
{code}
val report: ApplicationReport =
try {
getApplicationReport(appId)
} catch {
case e: ApplicationNotFoundException =>
logError(s"Application $appId not found.")
return (YarnApplicationState.KILLED, FinalApplicationStatus.KILLED)
case NonFatal(e) =>
logError(s"Failed to contact YARN for application $appId.", e)
return (YarnApplicationState.FAILED, FinalApplicationStatus.FAILED)
}
....
if (state == YarnApplicationState.FINISHED ||
state == YarnApplicationState.FAILED ||
state == YarnApplicationState.KILLED) {
cleanupStagingDir(appId)
return (state, report.getFinalApplicationStatus)
}
{code}
In case of ApplicationNotFoundException, we don't cleanup the sparkStaging
folder.
I believe we call cleanupStagingDir(appId) on the catch clause above.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
