Siying Dong created SPARK-45504:
-----------------------------------
Summary: RocksDB State Store Should Lower RocksDB Background
Thread CPU Priority
Key: SPARK-45504
URL: https://issues.apache.org/jira/browse/SPARK-45504
Project: Spark
Issue Type: Task
Components: Structured Streaming
Affects Versions: 3.5.1
Reporter: Siying Dong
We can move RocksDB flush and compaction to lower CPU priority. They usually
are background tasks and don't need to compete with task execution. For the
case where a task may wait for some RocksDB background task to finish, such as
checkpointing, or waiting async checkpointing to finish, the task slot is
waiting so we are likely to have enough CPU for low pri CPU anyway.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
