[
https://issues.apache.org/jira/browse/SPARK-20209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15954665#comment-15954665
]
Apache Spark commented on SPARK-20209:
--------------------------------------
User 'tdas' has created a pull request for this issue:
https://github.com/apache/spark/pull/17525
> Execute next trigger immediately if previous batch took longer than trigger
> interval
> ------------------------------------------------------------------------------------
>
> Key: SPARK-20209
> URL: https://issues.apache.org/jira/browse/SPARK-20209
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.1.0
> Reporter: Tathagata Das
> Assignee: Tathagata Das
>
> For large trigger intervals (e.g. 10 minutes), if a batch takes 11 minutes,
> then it will wait for 9 mins before starting the next batch. This does not
> make sense. The processing time based trigger policy should be to do process
> batches as fast as possible, but no faster than 1 in every trigger interval.
> If batches are taking longer than trigger interval anyways, then no point
> waiting extra trigger interval.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
