Ewen Cheslack-Postava commented on KAFKA-5828:

I don't think this is best solved at the Connect level. Could you just use 
[quotas|http://kafka.apache.org/documentation.html#design_quotas] to rate limit 
the connector?

> Allow explicitly setting polling interval for Kafka connectors
> --------------------------------------------------------------
>                 Key: KAFKA-5828
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5828
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>            Reporter: Behrang Saeedzadeh
>            Priority: Major
> I have a Kafka cluster deployed in our internal data center and a Kafka 
> Connect server deployed in AWS that gets data records from a topic on the 
> Kafka cluster and writes them into a Kinesis stream.
> We want to ensure that our Kafka connect server does not saturate the 
> available bandwidth between our internal data center and AWS.
> Using {{max.partition.fetch.bytes}} we can limit the upper bound of data that 
> can be fetched in each poll call. If we can also configure the poll interval, 
> then we can limit how much data is transferred per partition per second.

This message was sent by Atlassian JIRA

Reply via email to