Ewen Cheslack-Postava created KAFKA-3820:
--------------------------------------------
Summary: Provide utilities for tracking source offsets
Key: KAFKA-3820
URL: https://issues.apache.org/jira/browse/KAFKA-3820
Project: Kafka
Issue Type: New Feature
Components: KafkaConnect
Reporter: Ewen Cheslack-Postava
Assignee: Ewen Cheslack-Postava
OffsetStorageReader does not (and is not expected to) be immediately updated
when a SourceRecord is returned from poll(). However, this can be a bit
confusing to connector developers as they may return that data, then expect a
subsequent read from OffsetStorageReader should match that. In other words,
rather than tracking which offset they are at themselves in variables
maintained by the task implementation, the connector developer expected
OffsetStorageReader to do this for them.
Part of the confusion comes from the fact that data is sent asynchronously
after returned from poll(), which explains the semantics we have. However, it
does also mean many connectors have similarly structured code where they keep
track of the current offset themselves. It might be nice to provide some
utilities, probably through the Context object, to get the last returned offset
for each source partition being processed by a task.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
