[ https://issues.apache.org/jira/browse/SPARK-13222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mao, Wei updated SPARK-13222: ----------------------------- Description: The CheckpointInterval of DStream could be customized to multiple of BatchInteveal. So when user call shutdown it could be in the middle of two RDD checkpoint. In case the input source is not repeatable and user don't want to enable WAL because of its extra cost, the application could be unrecoverable. (was: In case the input source is not repeatable, and user don't want to enable WAL because of the overhead costing, we need to make sure the latest status of stateful RDD is checkpointed.) > checkpoint latest stateful RDD on graceful shutdown > --------------------------------------------------- > > Key: SPARK-13222 > URL: https://issues.apache.org/jira/browse/SPARK-13222 > Project: Spark > Issue Type: Bug > Components: Streaming > Reporter: Mao, Wei > > The CheckpointInterval of DStream could be customized to multiple of > BatchInteveal. So when user call shutdown it could be in the middle of two > RDD checkpoint. In case the input source is not repeatable and user don't > want to enable WAL because of its extra cost, the application could be > unrecoverable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org