[
https://issues.apache.org/jira/browse/SPARK-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Shar updated SPARK-18027:
-------------------------------
Description:
Hi,
It seems that SPARK-7705 didn't fix all issues with .sparkStaging folder
cleanup.
in Client.scala:monitorApplication
{code}
val report: ApplicationReport =
try {
getApplicationReport(appId)
} catch {
case e: ApplicationNotFoundException =>
logError(s"Application $appId not found.")
return (YarnApplicationState.KILLED, FinalApplicationStatus.KILLED)
case NonFatal(e) =>
logError(s"Failed to contact YARN for application $appId.", e)
return (YarnApplicationState.FAILED, FinalApplicationStatus.FAILED)
}
....
if (state == YarnApplicationState.FINISHED ||
state == YarnApplicationState.FAILED ||
state == YarnApplicationState.KILLED) {
cleanupStagingDir(appId)
return (state, report.getFinalApplicationStatus)
}
{code}
In case of ApplicationNotFoundException, we don't cleanup the sparkStaging
folder.
I believe we should call cleanupStagingDir(appId) on the catch clause above.
was:
Hi,
It seems that SPARK-7705 didn't fix all issues with .sparkStaging folder
cleanup.
in Client.scala:monitorApplication
{code}
val report: ApplicationReport =
try {
getApplicationReport(appId)
} catch {
case e: ApplicationNotFoundException =>
logError(s"Application $appId not found.")
return (YarnApplicationState.KILLED, FinalApplicationStatus.KILLED)
case NonFatal(e) =>
logError(s"Failed to contact YARN for application $appId.", e)
return (YarnApplicationState.FAILED, FinalApplicationStatus.FAILED)
}
....
if (state == YarnApplicationState.FINISHED ||
state == YarnApplicationState.FAILED ||
state == YarnApplicationState.KILLED) {
cleanupStagingDir(appId)
return (state, report.getFinalApplicationStatus)
}
{code}
In case of ApplicationNotFoundException, we don't cleanup the sparkStaging
folder.
I believe we call cleanupStagingDir(appId) on the catch clause above.
> .sparkStaging not clean on error
> --------------------------------
>
> Key: SPARK-18027
> URL: https://issues.apache.org/jira/browse/SPARK-18027
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.6.0
> Reporter: David Shar
>
> Hi,
> It seems that SPARK-7705 didn't fix all issues with .sparkStaging folder
> cleanup.
> in Client.scala:monitorApplication
> {code}
> val report: ApplicationReport =
> try {
> getApplicationReport(appId)
> } catch {
> case e: ApplicationNotFoundException =>
> logError(s"Application $appId not found.")
> return (YarnApplicationState.KILLED,
> FinalApplicationStatus.KILLED)
> case NonFatal(e) =>
> logError(s"Failed to contact YARN for application $appId.", e)
> return (YarnApplicationState.FAILED,
> FinalApplicationStatus.FAILED)
> }
> ....
> if (state == YarnApplicationState.FINISHED ||
> state == YarnApplicationState.FAILED ||
> state == YarnApplicationState.KILLED) {
> cleanupStagingDir(appId)
> return (state, report.getFinalApplicationStatus)
> }
> {code}
> In case of ApplicationNotFoundException, we don't cleanup the sparkStaging
> folder.
> I believe we should call cleanupStagingDir(appId) on the catch clause above.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
