[
https://issues.apache.org/jira/browse/SPARK-21370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Burak Yavuz updated SPARK-21370:
--------------------------------
Description:
During Streaming Aggregation, we have two StateStores per task, one used as
read-only in StateStoreRestoreExec, and one read-write used in
`StateStoreSaveExec`. `StateStore.abort` will be called for these StateStores
if they haven't committed their results. We need to make sure that abort in
read-only store after a commit in the read-write store doesn't
accidentally lead to the deletion of state.
This JIRA proposes a test for this case. StateStore implementations should
successfully handle this use case.
was:
Currently the HDFSBackedStateStore sets it's state as UPDATING as it is
initialized.
For every trigger, we create two state stores, one used by "StateStoreRestore"
operator to only read data and one by "StateStoreSave" operator to write
updates. So, the "Restore" StateStore is read-only. This state store gets
"aborted" after a task is completed, and this abort attempts to delete files
This can be avoided if there is an INITIALIZED state and abort deletes files
only when there is an update to the state store using "put" or "remove".
> Avoid doing anything on HDFSBackedStateStore.abort() when there are no
> updates to commit
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-21370
> URL: https://issues.apache.org/jira/browse/SPARK-21370
> Project: Spark
> Issue Type: Test
> Components: Structured Streaming
> Affects Versions: 2.1.1
> Reporter: Burak Yavuz
> Assignee: Burak Yavuz
> Priority: Minor
>
> During Streaming Aggregation, we have two StateStores per task, one used as
> read-only in StateStoreRestoreExec, and one read-write used in
> `StateStoreSaveExec`. `StateStore.abort` will be called for these StateStores
> if they haven't committed their results. We need to make sure that abort in
> read-only store after a commit in the read-write store doesn't
> accidentally lead to the deletion of state.
> This JIRA proposes a test for this case. StateStore implementations should
> successfully handle this use case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
