GitHub user rangadi opened a pull request:
https://github.com/apache/beam/pull/3461
[BEAM-2534] Handle offset gaps in Kafka messages.
KafkaIO logged a warning when there is a gap in offstes for messages.
Kafka also supports log compaction for topics some of the messages
are deleted leading to gaps in offsets. Log compaction removes earlier
messages for key. This PR removes the log and accounts for offset gaps in
backlog estimate.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rangadi/incubator-beam handle_offset_gaps
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3461.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3461
----
commit 3874bbaa6c88ace25adc132a9f785fb8ace4b68a
Author: Raghu Angadi <[email protected]>
Date: 2017-06-28T19:07:06Z
Handle offset gaps in Kafka messages.
KafkaIO logged a warning when there is a gap in offstes for messages.
Kafka also support 'KV' store style topics where some of the messages
are deleted leading gaps in offsets. This PR removes the log and
accounts for offset gaps in backlog estimate.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
