Have you looked at Kafka quotas( https://kafka.apache.org/0110/documentation.html#design_quotas) to achieve rate limiting on the consumer side? In your flink app, you should be able to set the client.id and configure kafka to rate limit you.
Regards, Nagarjun *Success is not final, failure is not fatal: it is the courage to continue that counts. * *- Winston Churchill - * On Fri, Jan 25, 2019 at 5:39 PM Lakshmi Gururaja Rao <l...@lyft.com.invalid> wrote: > Hey all, > > We have a Flink job reading from Kafka (specifically it uses > FlinkKafkaConsumer011). There are instances when the job is processing a > backlog and it ends up reading at a significantly high throughput and > degrades the underlying Kafka cluster. If there was a way to rate limit the > calls to Kafka (by controlling how often the *consumer*.poll() is called), > it would be a useful feature for our use case. > > Has anyone has run into a similar issue? Are there are any efforts/thoughts > on implementing a rate-limiting feature in the Flink Kafka connector? > > Thanks > Lakshmi >
