It will be good to be able to use the ListShards API. Are there any concerns bumping up the AWS SDK dependency? I see it was last done in https://issues.apache.org/jira/browse/FLINK-7422
Thanks On Wed, Feb 28, 2018 at 10:38 PM, Kailash Dayanand <kdayan...@lyft.com> wrote: > Based on the discussion at here > <https://lists.apache.org/thread.html/7b5f8034127c303414927232835990c2573ea63abda9dd9b8f2a261f@%3Cdev.flink.apache.org%3E>, > I want to propose using the latest ListShards API instead of the > DescribeStreams on AWS to overcome the rate limits currently imposed on > DescribeStream. The new List Shards have a much higher rate limits (a > limit of 100 transactions per second per data stream link > <https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ListShards.html>). > This was recently introduced in the aws-sdk-java release of 1.11.272 > <https://github.com/aws/aws-sdk-java/releases/tag/1.11.272>. I propose > bumping up the aws-sdk-java used in flink-kinesis connector and replace the > DescribeStream calls with ListShards in the KinesisProxy class here > <https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxy.java> > allowing > for faster shard discovery rate. > > Thanks > Kailash >
