[
https://issues.apache.org/jira/browse/FLINK-1268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-1268:
----------------------------------
Labels: stale-minor (was: )
> FileOutputFormat with overwrite does not clear local output directories
> -----------------------------------------------------------------------
>
> Key: FLINK-1268
> URL: https://issues.apache.org/jira/browse/FLINK-1268
> Project: Flink
> Issue Type: Bug
> Components: API / DataSet
> Reporter: Till Rohrmann
> Priority: Minor
> Labels: stale-minor
>
> I noticed that the FileOutputFormat does not clear the output directories if
> it writes to local disk. This has the consequence that previous partitions
> are still contained in the directory if one decreases the DOP between
> subsequent runs. If one reads the data from this directory, then more
> partitions will be read in than were actually written. This can lead to a
> wrong user code behaviour which is hard to debug. I'm aware that in case of a
> distributed execution the TaskManagers or the Tasks have to be responsible
> for the cleanup and if multiple Tasks are running on a TaskManager, then the
> cleanup has to be coordinated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
