anishshri-db opened a new pull request, #40696:
URL: https://github.com/apache/spark/pull/40696
### What changes were proposed in this pull request?
RocksDB state store commit should continue background work in finally only
if its paused
### Why are the changes needed?
If an exception is thrown earlier in the commit sequence before background
work is paused, we fail with an exception in the finally clause.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Ran unit tests
```
StateStoreIntegrationSuite
[info] Run completed in 16 seconds, 131 milliseconds.
[info] Total number of tests run: 5
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 5, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```
```
StateStoreSuite
[info] Run completed in 1 minute, 18 seconds.
[info] Total number of tests run:
73
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 73, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```
With simulated exception and without the fix, the code crashes with stack
trace below. With the fix, we don't see the issue in the finally block.
```
org.rocksdb.RocksDBException: InvalidArgument
at org.rocksdb.RocksDB.continueBackgroundWork(Native Method)
at org.rocksdb.RocksDB.continueBackgroundWork(RocksDB.java:3611)
at
org.apache.spark.sql.execution.streaming.state.RocksDB.commit(RocksDB.scala:421)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
