anishshri-db opened a new pull request, #48292:
URL: https://github.com/apache/spark/pull/48292
### What changes were proposed in this pull request?
Avoid flush during shutdown in rocksdb close path
### Why are the changes needed?
Without this change, we see sometimes that `cancelAllBackgroundWork` gets
hung if there are memtables that need to be flushed. We also don't need to
flush in this path, because we only assume that sync flush is required in the
commit path.
```
at app//org.rocksdb.RocksDB.cancelAllBackgroundWork(Native Method)
at app//org.rocksdb.RocksDB.cancelAllBackgroundWork(RocksDB.java:4053)
at
app//org.apache.spark.sql.execution.streaming.state.RocksDB.closeDB(RocksDB.scala:1406)
at
app//org.apache.spark.sql.execution.streaming.state.RocksDB.load(RocksDB.scala:383)
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Verified the config is passed manually in the logs and existing unit tests.
Before:
```
sql/core/target/unit-tests.log:141:18:20:06.223
pool-1-thread-1-ScalaTest-running-RocksDBSuite INFO RocksDB [Thread-17]:
[NativeRocksDB-1] Options.avoid_flush_during_shutdown: 0
sql/core/target/unit-tests.log:776:18:20:06.871
pool-1-thread-1-ScalaTest-running-RocksDBSuite INFO RocksDB [Thread-17]:
[NativeRocksDB-1] Options.avoid_flush_during_shutdown: 0
sql/core/target/unit-tests.log:1096:18:20:07.129
pool-1-thread-1-ScalaTest-running-RocksDBSuite INFO RocksDB [Thread-17]:
[NativeRocksDB-1] Options.avoid_flush_during_shutdown: 0
```
After:
```
sql/core/target/unit-tests.log:6561:18:17:42.723
pool-1-thread-1-ScalaTest-running-RocksDBSuite INFO RocksDB [Thread-17]:
[NativeRocksDB-1] Options.avoid_flush_during_shutdown: 1
sql/core/target/unit-tests.log:6947:18:17:43.035
pool-1-thread-1-ScalaTest-running-RocksDBSuite INFO RocksDB [Thread-17]:
[NativeRocksDB-1] Options.avoid_flush_during_shutdown: 1
sql/core/target/unit-tests.log:7344:18:17:43.313
pool-1-thread-1-ScalaTest-running-RocksDBSuite INFO RocksDB [Thread-17]:
[NativeRocksDB-1] Options.avoid_flush_during_shutdown: 1
```
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
