[
https://issues.apache.org/jira/browse/SPARK-8617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin updated SPARK-8617:
----------------------------------
Target Version/s: (was: 1.5.0)
> Handle history files better
> ---------------------------
>
> Key: SPARK-8617
> URL: https://issues.apache.org/jira/browse/SPARK-8617
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.4.0
> Reporter: Thomas Graves
>
> Right now there are many conditions in which history files get left around
> with the .inprogress extension. The cleaner doesn't remove these because it
> can't distinguish between something running and left over abandoned files.
> There are cases its really hard to not leave these around (application
> killed/crashes/etc). But we should still try to handle the history files
> better.
>
> For instance we could first write them to the .sparkStaging directory and
> then moving to the history file location before the application finishes.
> That way you only get completed history files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
