Greg Harris created KAFKA-14727:
-----------------------------------
Summary: Connect EOS mode should periodically call task commit
Key: KAFKA-14727
URL: https://issues.apache.org/jira/browse/KAFKA-14727
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Affects Versions: 3.4.0
Reporter: Greg Harris
Assignee: Greg Harris
In non-EOS mode, there is a background thread which periodically commits
offsets for a task. If this thread does not have resources to flush on the
framework side (records, or offsets) it still calls the task's commit() method
to update the internal state of the task.
In EOS mode, there is no background thread, and all offset commits are
performed on the main task thread in response to sending records to Kafka. This
has the effect of only triggering the task's commit() method when there are
records to send to Kafka, which is different than non-EOS mode.
In order to bring the two modes into better alignment, and allow tasks reliant
on the non-EOS empty commit() behavior to work in EOS mode out-of-the-box, EOS
mode should provide offset commits periodically for tasks which do not produce
records.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
