[
https://issues.apache.org/jira/browse/SPARK-7705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-7705:
-----------------------------------
Assignee: Apache Spark
> Cleanup of .sparkStaging directory fails if application is killed
> -----------------------------------------------------------------
>
> Key: SPARK-7705
> URL: https://issues.apache.org/jira/browse/SPARK-7705
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 1.3.0
> Reporter: Wilfred Spiegelenburg
> Assignee: Apache Spark
> Priority: Minor
>
> When a streaming application is killed while running on YARN the
> .sparkStaging directory is not cleaned up. Setting
> spark.yarn.preserve.staging.files=false does not help and still leaves the
> files around.
> The changes in SPARK-7503 do not catch this case since there is no exception
> in the shutdown. When the application gets killed the AM gets told to
> shutdown and the shutdown hook is run but the clean up is not triggered.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
