Based on the discussion at here <https://lists.apache.org/thread.html/7b5f8034127c303414927232835990c2573ea63abda9dd9b8f2a261f@%3Cdev.flink.apache.org%3E>, I want to propose using the latest ListShards API instead of the DescribeStreams on AWS to overcome the rate limits currently imposed on DescribeStream. The new List Shards have a much higher rate limits (a limit of 100 transactions per second per data stream link <https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ListShards.html>). This was recently introduced in the aws-sdk-java release of 1.11.272 <https://github.com/aws/aws-sdk-java/releases/tag/1.11.272>. I propose bumping up the aws-sdk-java used in flink-kinesis connector and replace the DescribeStream calls with ListShards in the KinesisProxy class here <https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxy.java> allowing for faster shard discovery rate.
Thanks Kailash
