Tathagata Das created SPARK-20209:
-------------------------------------
Summary: Execute next trigger immediately if previous batch took
longer than trigger interval
Key: SPARK-20209
URL: https://issues.apache.org/jira/browse/SPARK-20209
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 2.1.0
Reporter: Tathagata Das
Assignee: Tathagata Das
For large trigger intervals (e.g. 10 minutes), if a batch takes 11 minutes,
then it will wait for 9 mins before starting the next batch. This does not make
sense. The processing time based trigger policy should be to do process batches
as fast as possible, but no faster than 1 in every trigger interval. If batches
are taking longer than trigger interval anyways, then no point waiting extra
trigger interval.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
