Thanks Rong for your references.
>From what I can see, the rate limiter is initialized statically. But if the
>load on downstream services varies, is there a way to update the rater limiter
>at runtime ? Please let me know if I missed anything.
Thanks again for your advice. On Wednesday, November 27, 2019, 11:32:06 AM
EST, Rong Rong <[email protected]> wrote:
Hi Mans,
is this what you are looking for [1][2]?
--Rong
[1] https://issues.apache.org/jira/browse/FLINK-11501[2]
https://github.com/apache/flink/pull/7679
On Mon, Nov 25, 2019 at 3:29 AM M Singh <[email protected]> wrote:
Thanks Ciazhi & Thomas for your responses.
I read the throttling example but want to see if that work with a distributed
broker like Kinesis and how to have throttling feedback to the Kinesis source
so that it can vary the rate without interfering with watermarks, etc.
Thanks again
Mans
On Monday, November 25, 2019, 05:55:21 AM EST, Thomas Julian
<[email protected]> wrote:
related
https://issues.apache.org/jira/browse/FLINK-13792
Regards,
Julian.
---- On Mon, 25 Nov 2019 15:25:14 +0530 Caizhi Weng <[email protected]>
wrote ----
Hi,
As far as I know, Flink currently doesn't have a built-in throttling function.
You can write your own user-defined function to achieve this. Your function
just gives out what it reads in and limits the speed it gives out records at
the same time.
If you're not familiar with user-defined functions, see
https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/udfs.html
Here is a throttling iterator example:
https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/utils/ThrottledIterator.java
M Singh <[email protected]> 于2019年11月25日周一 上午5:50写道:
Hi:
I have an Flink streaming application that invokes some other web services.
However the webservices have limited throughput. So I wanted to find out if
there any recommendations on how to throttle the Flink datastream so that they
don't overload the downstrream services. I am using Kinesis as source and sink
in my application.
Please let me know if there any hooks available in Flink, what are the patterns
that can be used and what are the best practices/pitfalls for using them.
Thanks
Mans