L. C. Hsieh created SPARK-33783:
-----------------------------------
Summary: Unload State Store Provider after configured keep alive
time
Key: SPARK-33783
URL: https://issues.apache.org/jira/browse/SPARK-33783
Project: Spark
Issue Type: New Feature
Components: Structured Streaming
Affects Versions: 3.2.0
Reporter: L. C. Hsieh
Assignee: L. C. Hsieh
Currently Spark unloads an inactive state store provider in an maintenance task
which is run periodically. So it is said one state store provider might be
unloaded immediately after it becomes inactive because the maintenance task is
asynchronous. It is inefficient as the state store provider could be reused in
next batches and we should be able to have more control of unloading behavior.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
