[
https://issues.apache.org/jira/browse/KAFKA-13601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509319#comment-17509319
]
Anil Dasari commented on KAFKA-13601:
-------------------------------------
Hello [~ChrisEgerton] , thanks for the response. new proposed option provides
exactly once guarantees in case of field partitioner and size or time rotation
based flush strategy considering no duplicates in kafka topic.
S3 sink connector can provide exactly once guarantee with async offset commit
only in size/count based flush strategy and not in time rotation based flush
strategy.
Agree with you on " I think we'd need to do more than enable synchronous
offset commits in the Kafka Connect framework". Offset commit interval on
cluster to 1 is too aggressive and have performance impacts. At least, this new
option commits offsets of sink connectors of the cluster sync'ly. This is user
choice, by default it will be false. I am not sure how this effects source
connectors as source connectors not require offset commit.
Sync offset commit can't be added to sink connectors as offset commit is
initiated from kafka connect only.
> Add option to support sync offset commit in Kafka Connect Sink
> --------------------------------------------------------------
>
> Key: KAFKA-13601
> URL: https://issues.apache.org/jira/browse/KAFKA-13601
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect
> Reporter: Anil Dasari
> Priority: Major
>
> Exactly once in s3 connector with scheduled rotation and field partitioner
> can be achieved with consumer offset sync' commit after message batch flushed
> to sink successfully
> Currently, WorkerSinkTask committing the consumer offsets asynchronously and
> at regular intervals of WorkerConfig.OFFSET_COMMIT_INTERVAL_MS_CONFIG
> [https://github.com/apache/kafka/blob/371f14c3c12d2e341ac96bd52393b43a10acfa84/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L203]
> [https://github.com/apache/kafka/blob/371f14c3c12d2e341ac96bd52393b43a10acfa84/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L196]
> [https://github.com/apache/kafka/blob/371f14c3c12d2e341ac96bd52393b43a10acfa84/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L354]
>
> Add config to allow user to select synchronous commit over
> WorkerConfig.OFFSET_COMMIT_INTERVAL_MS_CONFIG
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
