[
https://issues.apache.org/jira/browse/KAFKA-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton reassigned KAFKA-8586:
------------------------------------
Assignee: Chris Egerton
> Source task producers silently fail to send records
> ---------------------------------------------------
>
> Key: KAFKA-8586
> URL: https://issues.apache.org/jira/browse/KAFKA-8586
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 2.3.0
> Reporter: Chris Egerton
> Assignee: Chris Egerton
> Priority: Major
>
> The Connect framework marks source records as successfully sent when they are
> dispatched to the producer, instead of when they are actually sent to Kafka.
> [This is assumed to be good
> enough|https://github.com/apache/kafka/blob/3e9d1c1411c5268de382f9dfcc95bdf66d0063a0/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java#L324-L331]
> since the Connect framework sets up its producer to use infinite retries on
> retriable errors, but in the case of an authorization or authentication
> failure with a secured Kafka broker, the errors aren't retriable and cause
> the producer to invoke its send callback with an exception and then give up
> on sending the message. This is a problem since the callback currently used
> by the WorkerSourceTask class when it invokes Producer.send(...) logs the
> exception and does nothing else. This leads to data loss since the source
> offsets for those failed records are committed, and the status of the task is
> never affected so users may not even know that something is wrong unless they
> check the worker log files or notice that data isn't flowing into Kafka.
> Until and unless someone does notice that something's wrong, the task will
> continue processing records and committing offsets, even though nothing is
> making it into Kafka.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
