[
https://issues.apache.org/jira/browse/SPARK-49411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved SPARK-49411.
----------------------------------
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 47895
[https://github.com/apache/spark/pull/47895]
> Communicate RocksDB State Store CheckpointID Between Driver and Executor
> ------------------------------------------------------------------------
>
> Key: SPARK-49411
> URL: https://issues.apache.org/jira/browse/SPARK-49411
> Project: Spark
> Issue Type: Task
> Components: Structured Streaming
> Affects Versions: 4.0.0
> Reporter: Siying Dong
> Assignee: Siying Dong
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> A incremental step to implement RocksDB state store checkpoint format V2.
> Once conf STATE_STORE_CHECKPOINT_FORMAT_VERSION is set to be higher than
> version 2, the executor returns checkpointID to the driver (only done for
> RocksDB). The driver stores is locally. For the next batch, the checkpointID
> is sent to the executor to be used to load the state store. If the local
> version of the executor doesn't match the uniqueID, it will reload from the
> checkpoint.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
