[
https://issues.apache.org/jira/browse/SPARK-52188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-52188:
----------------------------------
Fix Version/s: (was: 4.1.0)
> State store source can't read from RocksDB
> ------------------------------------------
>
> Key: SPARK-52188
> URL: https://issues.apache.org/jira/browse/SPARK-52188
> Project: Spark
> Issue Type: Improvement
> Components: SQL, Structured Streaming
> Affects Versions: 4.0.0
> Reporter: Adam Binford
> Assignee: Eric Marnadi
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> Trying to use the new statestore source to read a checkpoint using RocksDB
> fails with "assert failed: Failed to find query id/batch Id in task context".
> This happens here:
> https://github.com/apache/spark/blob/v4.0.0-rc6/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala#L801
> This doesn't fail in tests since the assertion specifically allows this case
> in tests. The RUN_ID_KEY never gets set on the path of loading the state
> store for the statestore source.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
