[
https://issues.apache.org/jira/browse/KAFKA-9982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118586#comment-17118586
]
Qinghui Xu commented on KAFKA-9982:
-----------------------------------
Hi, [~ChrisEgerton] Sorry for the delay.
> As of the fix for https://issues.apache.org/jira/browse/KAFKA-8586, the
>framework will cease processing records from a source task if it fails to send
>a record to Kafka.
This indeed addresses my concern about failure retries.
> The framework does use an entirely different producer to write source offsets
>to Kafka, but no offsets are written to Kafka unless the record they
>correspond to has been ack'd by the broker and safely made it to Kafka.
I don't see how this is guaranteed, though. As offset commit and producer
sending records are asynchronous, the two can happen in any order, and if the
task is lost/restarted in the middle, there's a chance that offset is committed
while records are not yet sent.
> [kafka-connect] Source connector does not guarantee at least once delivery
> --------------------------------------------------------------------------
>
> Key: KAFKA-9982
> URL: https://issues.apache.org/jira/browse/KAFKA-9982
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 2.5.0
> Reporter: Qinghui Xu
> Priority: Major
>
> In kafka-connect runtime, the WorkerSourceTask is responsible for sending
> records to the destination topics and managing the source offset commit.
> Committed offsets are then used later for recovery of tasks during rebalance
> or restart.
> But there are two concerns when looking into the WorkerSourceTask
> implementation:
> * When producer fail to send records, there's no retry but just skipping
> offset commit and then execute next loop (poll for new records)
> * The offset commit and effectively sending records over network are in fact
> asynchronous, which means the offset commit could happen before records are
> received by brokers, and a rebalance/restart in this gap could lead to
> message loss.
> The conclusion is thus that the source connector does not support at least
> once semantics by default (without the plugin implementation making extra
> effort itself). I consider this as a bug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
