[
https://issues.apache.org/jira/browse/SPARK-19779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Feng Gui updated SPARK-19779:
-----------------------------
Description: The PR (https://github.com/apache/spark/pull/17012) can to fix
restart a Structured Streaming application using hdfs as fileSystem, but also
exist a problem that a tmp file of delta file is still reserved in hdfs. And
Structured Streaming don't delete the tmp file generated when restart streaming
job in future, so we need to delete the tmp file after restart streaming job.
(was: The PR (https://github.com/apache/spark/pull/17012) can to fix restart a
Structured Streaming application using hdfs as fileSystem, but that exist an
problem that an tmp file of delta file is still reserved in hdfs. And
Structured Streaming don't delete the tmp file which generated when restart
streaming job.)
> structured streaming exist residual tmp file
> ---------------------------------------------
>
> Key: SPARK-19779
> URL: https://issues.apache.org/jira/browse/SPARK-19779
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.1.0
> Reporter: Feng Gui
> Priority: Minor
>
> The PR (https://github.com/apache/spark/pull/17012) can to fix restart a
> Structured Streaming application using hdfs as fileSystem, but also exist a
> problem that a tmp file of delta file is still reserved in hdfs. And
> Structured Streaming don't delete the tmp file generated when restart
> streaming job in future, so we need to delete the tmp file after restart
> streaming job.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
