[
https://issues.apache.org/jira/browse/SPARK-8617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726761#comment-15726761
]
Apache Spark commented on SPARK-8617:
-------------------------------------
User 'seyfe' has created a pull request for this issue:
https://github.com/apache/spark/pull/16165
> Handle history files better
> ---------------------------
>
> Key: SPARK-8617
> URL: https://issues.apache.org/jira/browse/SPARK-8617
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.4.0
> Reporter: Thomas Graves
>
> Right now there are many conditions in which history files get left around
> with the .inprogress extension. The cleaner doesn't remove these because it
> can't distinguish between something running and left over abandoned files.
> There are cases its really hard to not leave these around (application
> killed/crashes/etc). But we should still try to handle the history files
> better.
>
> For instance we could first write them to the .sparkStaging directory and
> then moving to the history file location before the application finishes.
> That way you only get completed history files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
