Good for AWS has increased the limit from 5txn/s last year to 10txn/s On Mon, Mar 5, 2018 at 8:32 AM, Thomas Weise <t...@apache.org> wrote:
> On Sun, Mar 4, 2018 at 11:38 PM, Bowen Li <bowenl...@gmail.com> wrote: > > > If ListShards() gives all the info that Flink needs, +1 on switching. > > DescribeStreams() has a limitation of 5 requests/sec, which is pretty > > bad.... > > > > But, I believe the goal of switching APIs should be *making Flink jobs > that > > read from Kinesis more stable*, rather than having faster shard discovery > > rate. The default shard discovery rate is every 10s, which is already > very > > very fast and can satisfy most Kinesis users, we shouldn't shorten the > > default value anymore. Developers who want faster discovery rate than 10s > > should overwrite the default value themselves. > > > > > https://docs.aws.amazon.com/kinesis/latest/APIReference/ > API_DescribeStream.html > > "This operation has a limit of 10 transactions per second per account." > > 10s interval would be sufficient, but with 5 requests per second and a > larger number of subtasks and multiple applications in the account, we are > looking at discovery rates in double digit minutes. > > That's where ListShards will help. Of course, the consumer should still be > redesigned to centralize the discovery. > > Thanks, > Thomas >
