Daniel Haviv created SPARK-18563:
------------------------------------
Summary: mapWithState: initialState should have a timeout setting
per record
Key: SPARK-18563
URL: https://issues.apache.org/jira/browse/SPARK-18563
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Reporter: Daniel Haviv
when passing an initialState for mapWithState there should a possibility to set
a timeout at the record level.
If for example mapWithState is configured with a 48H timeout, loading an
initialState will cause the state to bloat and hold 96H of data and then
release 48H of data at once.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
