Hey y'all, I've expanded the scope of this KIP slightly to include a pluggable interface, RateLimiter.
After implementing this a few different ways, it's clear that the configuration story is actually simpler with a pluggable model. Out-of-the-box, we have just two configuration properties to tweak: record.rate.limit and record.batch.rate.limit (subj to change ofc). These are provided by built-in RecordRateLimiter and RecordBatchRateLimiter impls. >From there, additional custom RateLimiters can be enabled with whatever configuration they need. This is essentially the same pattern taken with MetricsReporters and others. I had originally envisioned that the set of built-in limits would expand over time, eg individual put/poll/commit/flush limits. However, these can all be throttled adequately with the proposed API by limiting overall record and batch thruput. Please let me know what you think. The voting thread is open. Ryanne On Fri, Apr 9, 2021, 1:41 PM Ryanne Dolan <ryannedo...@gmail.com> wrote: > Hey y'all, I'd like to draw you attention to a new KIP: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-731%3A+Record+Rate+Limiting+for+Kafka+Connect > > Lemme know what you think. Thanks! > > Ryanne >
