Daniel Haviv created SPARK-18564:
------------------------------------
Summary: mapWithState: add configuration for
DEFAULT_CHECKPOINT_DURATION_MULTIPLIER
Key: SPARK-18564
URL: https://issues.apache.org/jira/browse/SPARK-18564
Project: Spark
Issue Type: Improvement
Reporter: Daniel Haviv
Currently mapWithState checkpoints the whole state every 10 batches.
Large state checkpointing can cause huge delays. exposing
DEFAULT_CHECKPOINT_DURATION_MULTIPLIER as a configuration parameter can the
user mitigate these delays.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
