Thomas Graves created SPARK-8617:
------------------------------------
Summary: Handle history files better
Key: SPARK-8617
URL: https://issues.apache.org/jira/browse/SPARK-8617
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.4.0
Reporter: Thomas Graves
Right now there are many conditions in which history files get left around with
the .inprogress extension. The cleaner doesn't remove these because it can't
distinguish between something running and left over abandoned files.
There are cases its really hard to not leave these around (application
killed/crashes/etc). But we should still try to handle the history files
better.
For instance we could first write them to the .sparkStaging directory and then
moving to the history file location before the application finishes. That way
you only get completed history files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
