Burak Yavuz created SPARK-21370:
-----------------------------------
Summary: Clarify In-Memory State Store purpose (read-only,
read-write) with an additional state
Key: SPARK-21370
URL: https://issues.apache.org/jira/browse/SPARK-21370
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 2.1.1
Reporter: Burak Yavuz
Assignee: Burak Yavuz
Currently the HDFSBackedStateStore sets it's state as UPDATING as it is
initialized.
For every trigger, we create two state stores, one used during "Restore" and
one during "Save". The "Restore" StateStore is read-only. This state store gets
"aborted" after a task is completed, which results in a file being created and
immediately deleted.
This can be avoided if there is an INITIALIZED state and abort deletes files
only when there is an update to the state store using "put" or "remove".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
