Hi, I would also go for Niels approach. If the mapper has the same parallelism as the source and its right after it, it'll be chained to the source. The throttling then happens with almost no overhead.
Regarding the ThrottledIterator: Afaik there is no iterator involved when reading data out of the Kafka connector, so there is no way to plug it in anywhere. On Sun, Apr 17, 2016 at 8:53 AM, Márton Balassi <[email protected]> wrote: > There is a utility in flink-streaming-examples that might be useful, but > is generally the same idea that Niels suggests. [1] > > [1] > https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/utils/ThrottledIterator.java > > On Sun, Apr 17, 2016 at 8:42 AM, Niels Basjes <[email protected]> wrote: > >> Simple idea: create a map function that only does "sleep 1/5 second" and >> put that in your pipeline somewhere. >> >> Niels >> On 16 Apr 2016 22:38, "Chen Bekor" <[email protected]> wrote: >> >>> is there a way to consume a kafka stream using flink with a predefined >>> rate limit (eg 5 events per second) >>> >>> we need this because we need to control some 3rd party api rate >>> limitations so, even if we have a much larger throughput potential, we >>> must control the consumption rate in order not to overflow the API channel. >>> >> >
