Pavel Kuznetsov created KAFKA-10327:
---------------------------------------
Summary: Make flush after some count of putted records in SinkTask
Key: KAFKA-10327
URL: https://issues.apache.org/jira/browse/KAFKA-10327
Project: Kafka
Issue Type: Improvement
Components: KafkaConnect
Affects Versions: 2.5.0
Reporter: Pavel Kuznetsov
In current version of kafka connect all records accumulated with SinkTask.put
method are flushed to target system on a time-based manner. So data is flushed
and offsets are committed every offset.flush.timeout.ms (default is 60000) ms.
But you can't control the number of messages you receive from Kafka between two
flushes. It may cause out of memory errors, because in-memory buffer may grow a
lot.
I suggest to add out of box support of count-based flush to kafka connect. It
requires new configuration parameter (offset.flush.count, for example). Number
of records sent to SinkTask.put should be counted, and if these amount is
greater than offset.flush.count's value, SinkTask.flush is called and offsets
are committed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
