[
https://issues.apache.org/jira/browse/SPARK-7705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-7705.
------------------------------
Resolution: Fixed
Fix Version/s: 1.5.0
1.4.1
Resolved by https://github.com/apache/spark/pull/6409
> Cleanup of .sparkStaging directory fails if application is killed
> -----------------------------------------------------------------
>
> Key: SPARK-7705
> URL: https://issues.apache.org/jira/browse/SPARK-7705
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 1.3.0
> Reporter: Wilfred Spiegelenburg
> Priority: Minor
> Fix For: 1.4.1, 1.5.0
>
>
> When a streaming application is killed while running on YARN the
> .sparkStaging directory is not cleaned up. Setting
> spark.yarn.preserve.staging.files=false does not help and still leaves the
> files around.
> The changes in SPARK-7503 do not catch this case since there is no exception
> in the shutdown. When the application gets killed the AM gets told to
> shutdown and the shutdown hook is run but the clean up is not triggered.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
