[
https://issues.apache.org/jira/browse/KAFKA-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973506#comment-15973506
]
Randall Hauch edited comment on KAFKA-3821 at 4/18/17 9:17 PM:
---------------------------------------------------------------
[~criccomini] logged KAFKA-5084, but that is really another situation where the
connector would like to provide new offsets without having to produce
{{SourceRecord}}. As he says
[here|https://confluentcommunity.slack.com/archives/C49L0V3L7/p1492547735471505],
the connector sometimes need to tell Kafka Connect:
{quote}
i made some progress in my source, but i don’t want to send any of the data
{quote}
This becomes an issue when stopping and restarting the connector. In cases when
it's been a long time since the connector produced source records with an
updated offset, the connector upon startup will receive this old (but still the
latest) offset. But the source has long purged that portion of the log, so the
connector is unable to start up. In reality, the connector had processed much
farther than was described by the last offset, and if it had been able to
submit that offset to Kafka Connect, upon restart it would have been able to
successfully start at a point in the log that still exists in the source.
was (Author: rhauch):
[~criccomini] logged KAFKA-5084, but that is really another situation where the
connector would like to provide new offsets without having to produce
{{SourceRecord}}. As he says
[here|https://confluentcommunity.slack.com/archives/C49L0V3L7/p1492547735471505],
the connector sometimes need to tell Kafka Connect:
{quote}
i made some progress in my source, but i don’t want to send any of the data
{quote}
> Allow Kafka Connect source tasks to produce offset without writing to topics
> ----------------------------------------------------------------------------
>
> Key: KAFKA-3821
> URL: https://issues.apache.org/jira/browse/KAFKA-3821
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Affects Versions: 0.9.0.1
> Reporter: Randall Hauch
> Labels: needs-kip
>
> Provide a way for a {{SourceTask}} implementation to record a new offset for
> a given partition without necessarily writing a source record to a topic.
> Consider a connector task that uses the same offset when producing an unknown
> number of {{SourceRecord}} objects (e.g., it is taking a snapshot of a
> database). Once the task completes those records, the connector wants to
> update the offsets (e.g., the snapshot is complete) but has no more records
> to be written to a topic. With this change, the task could simply supply an
> updated offset.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
