[
https://issues.apache.org/jira/browse/SPARK-13222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mao, Wei updated SPARK-13222:
-----------------------------
Description: The CheckpointInterval of DStream could be customized to
multiple of BatchInteveal. So when user call shutdown it could be in the middle
of two RDD checkpoint. In case the input source is not repeatable and user
don't want to enable WAL because of its extra cost, the application could be
unrecoverable. (was: In case the input source is not repeatable, and user
don't want to enable WAL because of the overhead costing, we need to make sure
the latest status of stateful RDD is checkpointed.)
> checkpoint latest stateful RDD on graceful shutdown
> ---------------------------------------------------
>
> Key: SPARK-13222
> URL: https://issues.apache.org/jira/browse/SPARK-13222
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Reporter: Mao, Wei
>
> The CheckpointInterval of DStream could be customized to multiple of
> BatchInteveal. So when user call shutdown it could be in the middle of two
> RDD checkpoint. In case the input source is not repeatable and user don't
> want to enable WAL because of its extra cost, the application could be
> unrecoverable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
