[GitHub] spark pull request #21300: [SPARK-24067][BACKPORT-2.3][STREAMING][KAFKA] All...

[koeninger]

Github user koeninger closed the pull request at:

https://github.com/apache/spark/pull/21300


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21300: [SPARK-24067][BACKPORT-2.3][STREAMING][KAFKA] All...

[koeninger]

GitHub user koeninger opened a pull request:

https://github.com/apache/spark/pull/21300

[SPARK-24067][BACKPORT-2.3][STREAMING][KAFKA] Allow non-consecutive offsets

## What changes were proposed in this pull request?

Backport of the bugfix in SPARK-17147

Add a configuration spark.streaming.kafka.allowNonConsecutiveOffsets to 
allow streaming jobs to proceed on compacted topics (or other situations 
involving gaps between offsets in the log).

## How was this patch tested?

Added new unit test

justinrmiller has been testing this branch in production for a few weeks



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/daten-kieker/spark branch-2.3_kafkafix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/21300.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #21300


commit 84648e0222330fd3bc43ce214689b66795efdffe
Author: cody koeninger 
Date:   2018-02-27T14:21:11Z

[SPARK-17147][STREAMING][KAFKA] Allow non-consecutive offsets

## What changes were proposed in this pull request?

Add a configuration spark.streaming.kafka.allowNonConsecutiveOffsets to 
allow streaming jobs to proceed on compacted topics (or other situations 
involving gaps between offsets in the log).

## How was this patch tested?

Added new unit test

justinrmiller has been testing this branch in production for a few weeks

Author: cody koeninger 

Closes #20572 from koeninger/SPARK-17147.




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org