Riya Verma created SPARK-48586:
----------------------------------
Summary: Remove lock contention between maintenance and task
threads
Key: SPARK-48586
URL: https://issues.apache.org/jira/browse/SPARK-48586
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 3.4.3
Reporter: Riya Verma
Currently the lock of the RocksDB state store is acquired when uploading the
snapshot inside maintenance tasks when change log checkpointing is enabled,
which causes lock contention between query processing tasks and state
maintenance thread. To eliminate the lock contention, lock acquisition inside
maintenance tasks should be avoided. To prevent race conditions between task
and maintenance threads, we can ensure that RocksDBFileManager has a linear
history by ensuring a deep copy of RocksDBFileManager every time a previous
version is loaded. The original file manager is not affected by future state
update. The new file manager is not affected by background snapshot uploading
tasks that attempt to upload a snapshot.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
