[
https://issues.apache.org/jira/browse/SPARK-38522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved SPARK-38522.
----------------------------------
Fix Version/s: 3.4.0
Resolution: Fixed
Issue resolved by pull request 35816
[https://github.com/apache/spark/pull/35816]
> Strengthen the contract on iterator method in StateStore
> --------------------------------------------------------
>
> Key: SPARK-38522
> URL: https://issues.apache.org/jira/browse/SPARK-38522
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.3.0
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
> Priority: Major
> Fix For: 3.4.0
>
>
> The root cause of SPARK-38320 was that the logic initialized the iterator
> first, and performed some updates against state store, and iterated through
> iterator expecting that all updates in between should be visible in iterator.
> That is not guaranteed in RocksDB state store, and the contract of Java
> ConcurrentHashMap which is used in HDFSBackedStateStore does not also
> guarantee it.
> It would be clearer if we update the contract to draw a line on behavioral
> guarantee to callers so that callers don't get such expectation.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
