GitHub user koeninger opened a pull request:
https://github.com/apache/spark/pull/20572
[SPARK-17147][STREAMING][KAFKA] Allow non-consecutive offsets
## What changes were proposed in this pull request?
Add a configuration spark.streaming.kafka.allowNonConsecutiveOffsets to
allow streaming jobs to proceed on compacted topics (or other situations
involving gaps between offsets in the log).
## How was this patch tested?
Added new unit test
@justinrmiller has been testing this branch in production for a few weeks
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/koeninger/spark-1 SPARK-17147
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20572.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20572
----
commit 3082de7e43e8c381dc2227005d1e0fc5bd2c3d29
Author: cody koeninger <cody@...>
Date: 2016-10-08T21:21:48Z
[SPARK-17147][STREAMING][KAFKA] failing test for compacted topics
commit e8ea89ea10527c6723df4af2685004ea67d872cd
Author: cody koeninger <cody@...>
Date: 2016-10-09T04:59:39Z
[SPARK-17147][STREAMING][KAFKA] test passing for compacted topics
commit 182943e36f596d0cb5841a9c63471bea1dd9047b
Author: cody koeninger <cody@...>
Date: 2018-02-11T04:09:38Z
spark.streaming.kafka.allowNonConsecutiveOffsets
commit 89f4bc5f4de78cdcc22b5c9b26a27ee9263048c8
Author: cody koeninger <cody@...>
Date: 2018-02-11T04:13:49Z
[SPARK-17147][STREAMING][KAFKA] remove stray param doc
commit 12e65bedddbcd2407598e69fa3c6fcbcdfc67e5d
Author: cody koeninger <cody@...>
Date: 2018-02-11T04:28:22Z
[SPARK-17147][STREAMING][KAFKA] prepare for merge of master
commit 2ed51f1f73ee75ffd08355265a72e68e83ef592d
Author: cody koeninger <cody@...>
Date: 2018-02-11T05:19:31Z
Merge branch 'master' into SPARK-17147
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
