[
https://issues.apache.org/jira/browse/KAFKA-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16588619#comment-16588619
]
ASF GitHub Bot commented on KAFKA-3821:
---------------------------------------
gunnarmorling opened a new pull request #5553: WIP - KAFKA-3821 Allowing source
tasks to produce offset without emitting r…
URL: https://github.com/apache/kafka/pull/5553
Hi @rhauch, @ewencp et al., here's a proposal for
[KAFKA-3821](https://issues.apache.org/jira/browse/KAFKA-3821) based on my
latest comment on the JIRA issue. It'd need more polishing and testing, but I
wanted to demonstrate the idea and see whether you agree with it before
spending more time on it.
I think it's the simplest way to let source connectors produce offsets
without emitting messages and without creating an awkward inheritance
relationship of `SourceRecord` and a sub-class just carrying offset
information. Please see [this
comment](https://issues.apache.org/jira/browse/KAFKA-3821?focusedCommentId=16564900&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16564900)
on the JIRA issue for the two use cases of this new functionality.
Looking forward to learn about your toughts. Thanks!
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Allow Kafka Connect source tasks to produce offset without writing to topics
> ----------------------------------------------------------------------------
>
> Key: KAFKA-3821
> URL: https://issues.apache.org/jira/browse/KAFKA-3821
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Affects Versions: 0.9.0.1
> Reporter: Randall Hauch
> Priority: Major
> Labels: needs-kip
> Fix For: 2.1.0
>
>
> Provide a way for a {{SourceTask}} implementation to record a new offset for
> a given partition without necessarily writing a source record to a topic.
> Consider a connector task that uses the same offset when producing an unknown
> number of {{SourceRecord}} objects (e.g., it is taking a snapshot of a
> database). Once the task completes those records, the connector wants to
> update the offsets (e.g., the snapshot is complete) but has no more records
> to be written to a topic. With this change, the task could simply supply an
> updated offset.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
