Jungtaek Lim created SPARK-36619:
------------------------------------
Summary: Iterators on prefix scan in RocksDB are not cleaned up
unless aborting
Key: SPARK-36619
URL: https://issues.apache.org/jira/browse/SPARK-36619
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 3.2.0
Reporter: Jungtaek Lim
In RocksDB state store provider implementation, we leverage iterators on prefix
scan, which are being closed in rollback() method.
While this works now for session window since state store instance in read
physical plan will always call abort, it could bring correctness issue for
stateful operator which doesn't instantiate two different physical plans on
read and write.
We should make sure these iterators get closed to let these iterators don't
affect multiple micro-batches (plays as side-effect).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
