[
https://issues.apache.org/jira/browse/SPARK-8384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14588363#comment-14588363
]
Norman He commented on SPARK-8384:
----------------------------------
It seems that if checkpoint interval is the same as batchDuration, you will
have a lot of checkpoint saving. ( old documentation talk about set checkpoint
interval 5 ~10 times batch duration ).
We are pushing batch duration to 200ms, if checkpoint duration is the same. I
am not sure whether checkpoint saving to hdfs or disk will impact the streaming
processing.
Is this a limitation now or will be improved upon in the future?
> Can not set checkpointDuration or Interval in spark 1.3 and later
> -----------------------------------------------------------------
>
> Key: SPARK-8384
> URL: https://issues.apache.org/jira/browse/SPARK-8384
> Project: Spark
> Issue Type: Bug
> Reporter: Norman He
> Priority: Critical
>
> StreamingContext missing setCheckpointDuration().
> No way around for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
