HeartSaVioR opened a new pull request #33870: URL: https://github.com/apache/spark/pull/33870
### What changes were proposed in this pull request? This PR ensures that resetting prefix-scan iterators is done on loading RocksDB, which was only done in rollback. ### Why are the changes needed? Prefix-scan iterators are only closed on RocksDB.rollback(), which is only called in RocksDBStateStore.abort(). While `RocksDBStateStore.abort()` method will be called for streaming session window (since it has two physical plans for read and write), other stateful operators which only have read-write physical plan will call either commit or abort, and don't close the iterators on committing. These unclosed iterators can be "reused" and produce incorrect outputs. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Modified UT which failed without this PR and passes with this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
