[
https://issues.apache.org/jira/browse/KAFKA-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952915#comment-16952915
]
Randall Hauch commented on KAFKA-6080:
--------------------------------------
My previous comment suggests combining a few different feature requests:
allowing tasks to optionally demarcate source-level transaction boundaries,
allowing tasks to optionally update source offsets without writing new source
records, and enabling Kafka EOS for the producer (primarily to eliminate the
risk of writing duplicate records). I think all are important and all are
related to each other, so let's make sure that if we only tackle a subset of
these that we don't prevent the addition of the remaining features.
The first two are pretty straightforward, and I thought EOS might be
straightforward -- so much so that I actually had a prototype at one point to
enable Kafka EOS in Connect, but ran into a snag: Connect doesn't have affinity
when it comes to task assignments for each worker, yet Kafka's EOS requires
that each producer using AK transactions needs a unique ID and, upon a failure,
*the same producer with the same ID needs to abort the previously incomplete
transaction*. That means that when a task is restarted, it's likely it won't be
able to abort the previous transaction and we'd have to wait for the
transaction to time out. IIUC we're trying to minimize or eliminate that
constraint. As I learned with my prototype, changing the code to use Kafka EOS
is actually easy -- it's the recovery semantics correct that's the hard part.
> Transactional EoS for source connectors
> ---------------------------------------
>
> Key: KAFKA-6080
> URL: https://issues.apache.org/jira/browse/KAFKA-6080
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect
> Reporter: Antony Stubbs
> Assignee: Ryanne Dolan
> Priority: Major
> Labels: needs-kip
>
> Exactly once (eos) message production for source connectors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
