[
https://issues.apache.org/jira/browse/SPARK-44480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17749433#comment-17749433
]
Snoot.io commented on SPARK-44480:
----------------------------------
User 'ericm-db' has created a pull request for this issue:
https://github.com/apache/spark/pull/42066
> Add option for thread pool to perform maintenance for RocksDB/HDFS State
> Store Providers
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-44480
> URL: https://issues.apache.org/jira/browse/SPARK-44480
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.5.0
> Reporter: Eric Marnadi
> Priority: Major
>
> Maintenance tasks on StateStore was being done by a single background thread,
> which is prone to straggling. In this change, the single background thread
> would instead schedule maintenance tasks to a thread pool.
> Introduce
> {{spark.sql.streaming.stateStore.enableStateStoreMaintenanceThreadPool}}
> config so that the user can enable a thread pool for maintenance manually.
> Introduce {{spark.sql.streaming.stateStore.numStateStoreMaintenanceThreads}}
> config so the thread pool size is configurable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
