[
https://issues.apache.org/jira/browse/SPARK-17547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Rosen resolved SPARK-17547.
--------------------------------
Resolution: Fixed
Fix Version/s: 2.1.0
2.0.1
1.6.3
Issue resolved by pull request 15104
[https://github.com/apache/spark/pull/15104]
> Temporary shuffle data files may be leaked following exception in write
> -----------------------------------------------------------------------
>
> Key: SPARK-17547
> URL: https://issues.apache.org/jira/browse/SPARK-17547
> Project: Spark
> Issue Type: Bug
> Components: Shuffle
> Affects Versions: 1.5.3, 1.6.0, 2.0.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Fix For: 1.6.3, 2.0.1, 2.1.0
>
>
> SPARK-8029 modified shuffle writers to first stage their data to a temporary
> file in the same directory as the final destination file and then to
> atomically rename the file at the end of the write job. However, this change
> introduced the potential for the temporary output file to be leaked if an
> exception occurs during the write because the shuffle writers' existing error
> cleanup code doesn't handle this new temp file.
> This is easy to fix: we just need to add a {{finally}} block to ensure that
> the temporary file is guaranteed to be either moved or deleted before
> existing the shuffle write method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
