[GitHub] spark issue #20572: [SPARK-17147][STREAMING][KAFKA] Allow non-consecutive of...
Github user justinrmiller commented on the issue: https://github.com/apache/spark/pull/20572 Otherwise the job crashes, even for gaps of one or two messages on a couple of partitions (and while fail hard is a good philosophy, in this case it's not really practical for us) so it's pretty significant to us. That said, I've got the code copied into my project so when we go to 2.3.0 we could probably just keep using that. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20572: [SPARK-17147][STREAMING][KAFKA] Allow non-consecutive of...
Github user justinrmiller commented on the issue: https://github.com/apache/spark/pull/20572 Do you think it would be possible for this to go into 2.3.0? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20572: [SPARK-17147][STREAMING][KAFKA] Allow non-consecutive of...
Github user justinrmiller commented on the issue: https://github.com/apache/spark/pull/20572 Not to pile on, but I was encountering non-consecutive offsets on non-compacted topics once or twice a day at our volume. This patch fixes this (haven't had a problem with it since) and I'm sure there will be others encountering this if they stream continuously tens of billions of messages a day on kafka. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20572: [SPARK-17147][STREAMING][KAFKA] Allow non-consecutive of...
Github user justinrmiller commented on the issue: https://github.com/apache/spark/pull/20572 This looks good to me. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
