GitHub user koeninger opened a pull request:
https://github.com/apache/spark/pull/21300
[SPARK-24067][BACKPORT-2.3][STREAMING][KAFKA] Allow non-consecutive offsets
## What changes were proposed in this pull request?
Backport of the bugfix in SPARK-17147
Add a configuration spark.streaming.kafka.allowNonConsecutiveOffsets to
allow streaming jobs to proceed on compacted topics (or other situations
involving gaps between offsets in the log).
## How was this patch tested?
Added new unit test
justinrmiller has been testing this branch in production for a few weeks
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/daten-kieker/spark branch-2.3_kafkafix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21300.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21300
----
commit 84648e0222330fd3bc43ce214689b66795efdffe
Author: cody koeninger <cody@...>
Date: 2018-02-27T14:21:11Z
[SPARK-17147][STREAMING][KAFKA] Allow non-consecutive offsets
## What changes were proposed in this pull request?
Add a configuration spark.streaming.kafka.allowNonConsecutiveOffsets to
allow streaming jobs to proceed on compacted topics (or other situations
involving gaps between offsets in the log).
## How was this patch tested?
Added new unit test
justinrmiller has been testing this branch in production for a few weeks
Author: cody koeninger <c...@koeninger.org>
Closes #20572 from koeninger/SPARK-17147.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
