Matthias J. Sax created KAFKA-13549:
---------------------------------------

             Summary: Add "delete interval" config
                 Key: KAFKA-13549
                 URL: https://issues.apache.org/jira/browse/KAFKA-13549
             Project: Kafka
          Issue Type: Improvement
          Components: streams
            Reporter: Matthias J. Sax


Kafka Streams uses "delete record" requests to aggressively purge data from 
repartition topics. Those request are sent each time we commit.

For at-least-once with a default commit interval of 30 seconds, this works 
fine. However, for exactly-once with a default commit interval of 100ms, it's 
very aggressive. The main issue is broker side, because the broker logs every 
"delete record" request, and thus broker logs are spammed if EOS is enabled.

We should consider to add a new config (eg `delete.record.interval.ms` or 
similar) to have a dedicated config for "delete record" requests, to decouple 
it from the commit interval config and allow to purge data less aggressively, 
even if the commit interval is small to avoid the broker side log spamming.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Reply via email to