anishshri-db commented on code in PR #44712:
URL: https://github.com/apache/spark/pull/44712#discussion_r1454498382
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -671,6 +682,21 @@ class RocksDB(
override protected def logName: String = s"${super.logName} $loggingId"
}
+object RocksDB {
+ // Increase background thread pool size to 2 flush threads and 2 compaction
threads.
+ // Snapshot checkpoint requires a flush, so more threads will reduce the
blocking time. More
+ // compaction threads will reduce the chance that compaction is backlogged,
causing online
+ // traffic to slowdown.
+ if (RocksDBEnv.getDefault().getBackgroundThreads(Priority.HIGH) < 2) {
Review Comment:
How would this interact with the `setMaxgroundJobs` setting though ?
if user sets maxBackgroundJobs=2, then we are explicitly overriding to 4 ?
should we change minimum allowed for maxBackgroundJobs to 4 then ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
