[
https://issues.apache.org/jira/browse/KAFKA-10327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantine Karantasis updated KAFKA-10327:
-------------------------------------------
Labels: kip-required needs-kip (was: )
> Make flush after some count of putted records in SinkTask
> ---------------------------------------------------------
>
> Key: KAFKA-10327
> URL: https://issues.apache.org/jira/browse/KAFKA-10327
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Affects Versions: 2.5.0
> Reporter: Pavel Kuznetsov
> Priority: Major
> Labels: kip-required, needs-kip
>
> In current version of kafka connect all records accumulated with SinkTask.put
> method are flushed to target system on a time-based manner. So data is
> flushed and offsets are committed every offset.flush.timeout.ms (default is
> 60000) ms.
> But you can't control the number of messages you receive from Kafka between
> two flushes. It may cause out of memory errors, because in-memory buffer may
> grow a lot.
> I suggest to add out of box support of count-based flush to kafka connect. It
> requires new configuration parameter (offset.flush.count, for example).
> Number of records sent to SinkTask.put should be counted, and if these amount
> is greater than offset.flush.count's value, SinkTask.flush is called and
> offsets are committed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
