GitHub user efimpoberezkin opened a pull request:
https://github.com/apache/spark/pull/21392
[SPARK-24063][SS] Control maximum epoch backlog for ContinuousExecution
## What changes were proposed in this pull request?
This pull request adds maxEpochBacklog SQL configuration option.
EpochCoordinator tracks if the length of the queue of waiting epochs has
exceeded it. If so, stream is stopped with an error indicating too many epochs
stacked up
## How was this patch tested?
Existing unit tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/efimpoberezkin/spark pr/control-epoch-backlog
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21392.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21392
----
commit 1d2fc298284f6d553d78035f3095e5d2abe2a8a8
Author: Efim Poberezkin <efim@...>
Date: 2018-04-25T13:01:29Z
Add max epoch backlog option to SQLConf
commit 0919b3f7542aa0a807b0ac56e0da1366f347bb54
Author: Efim Poberezkin <efim@...>
Date: 2018-05-07T10:11:21Z
Replace logging an error with throwing an exception
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
