Burak Yavuz created SPARK-18342:
-----------------------------------
Summary: HDFSBackedStateStore can fail to rename files causing
snapshotting and recovery to fail
Key: SPARK-18342
URL: https://issues.apache.org/jira/browse/SPARK-18342
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 2.0.1
Reporter: Burak Yavuz
Priority: Critical
The HDFSBackedStateStore renames temporary files to delta files as it commits
new versions. It however doesn't check whether the rename succeeded. If the
rename fails, then recovery will not be possible. It should fail during the
rename stage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
