Wilfred Spiegelenburg created SPARK-7705:
--------------------------------------------
Summary: Cleanup of .sparkStaging directory fails if application
is killed
Key: SPARK-7705
URL: https://issues.apache.org/jira/browse/SPARK-7705
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 1.3.0
Reporter: Wilfred Spiegelenburg
When a streaming application is killed while running on YARN the .sparkStaging
directory is not cleaned up. Setting spark.yarn.preserve.staging.files=false
does not help and still leaves the files around.
The changes in SPARK-7503 do not catch this case since there is no exception in
the shutdown. When the application gets killed the AM gets told to shutdown and
the shutdown hook is run but the clean up is not triggered.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
