Re: RateLimit for Kinesis Producer

2018-04-28 Thread Lasse Nedergaard
Hi I encounter the same problem for the Kinesis producer and will try to play around with the config setting and look into the code base. If I figure it out l let you know and please do the same if you figure it out before me Med venlig hilsen / Best regards Lasse Nedergaard > Den 27. apr.

Re: RateLimit for Kinesis Producer

2018-04-27 Thread Tao Xia
Are you sure rate limit is coming from KinesisProducer? If yes, Kinesis support 1000 record write per sec per shard. if you hit the limit, just increase your shard. On Fri, Apr 27, 2018 at 8:58 AM, Urs Schoenenberger < urs.schoenenber...@tngtech.com> wrote: > Hi all, > > we are struggling with

RateLimit for Kinesis Producer

2018-04-27 Thread Urs Schoenenberger
Hi all, we are struggling with RateLimitExceededExceptions with the Kinesis Producer. The Flink documentation claims that the Flink Producer overrides the RateLimit setting from Amazon's default of 150 to 100. I am wondering whether we'd need 100/($sink_parallelism) in order for it to work