GitHub user gaborgsomogyi opened a pull request:
https://github.com/apache/spark/pull/23156
[SPARK-24063][SS] Add maximum epoch queue threshold for ContinuousExecution
## What changes were proposed in this pull request?
Continuous processing is waiting on epochs which are not yet complete (for
example one partition is not making progress) and stores pending items in
queues. These queues are unbounded and can consume up all the memory easily. In
this PR I've added `spark.sql.streaming.continuous.epochBacklogQueueSize`
configuration possibility to make them bounded. If the related threshold
reached then the query will stop with `IllegalStateException`.
## How was this patch tested?
Existing + additional unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gaborgsomogyi/spark SPARK-24063
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23156.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23156
----
commit 72733c5068d85e70e0a65b05a593c82120277622
Author: Gabor Somogyi <gabor.g.somogyi@...>
Date: 2018-11-22T20:45:37Z
[SPARK-24063][SS] Add maximum epoch queue threshold for ContinuousExecution.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
