If ListShards() gives all the info that Flink needs, +1 on switching.
DescribeStreams() has a limitation of 5 requests/sec, which is pretty
bad....
But, I believe the goal of switching APIs should be *making Flink jobs that
read from Kinesis more stable*, rather than having faster shard discovery
rate. The default shard discovery rate is every 10s, which is already very
very fast and can satisfy most Kinesis users, we shouldn't shorten the
default value anymore. Developers who want faster discovery rate than 10s
should overwrite the default value themselves.
I last bumped up AWS SDK version in flink-connector-kinesis. So from my
experience, I'd recommend checking:
1. make sure the new SDK works with both KCL and KPL in Flink.
Unfortunately, AWS SDK versions in KCL and KPL are not aligned well...
1. 1 If necessary, you need to bump their versions too
2. might need to test it on AWS EMR. AWS SDK is of 1.11.267 in latest
EMR (5.12.0), need to test compatibility.
Bowen
On Fri, Mar 2, 2018 at 8:47 AM, Thomas Weise <t...@apache.org> wrote:
> It will be good to be able to use the ListShards API. Are there any
> concerns bumping up the AWS SDK dependency? I see it was last done in
> https://issues.apache.org/jira/browse/FLINK-7422
>
> Thanks
>
> On Wed, Feb 28, 2018 at 10:38 PM, Kailash Dayanand <kdayan...@lyft.com>
> wrote:
>
> > Based on the discussion at here
> > <https://lists.apache.org/thread.html/7b5f8034127c303414927232835990
> c2573ea63abda9dd9b8f2a261f@%3Cdev.flink.apache.org%3E>,
> > I want to propose using the latest ListShards API instead of the
> > DescribeStreams on AWS to overcome the rate limits currently imposed on
> > DescribeStream. The new List Shards have a much higher rate limits (a
> > limit of 100 transactions per second per data stream link
> > <https://docs.aws.amazon.com/kinesis/latest/APIReference/
> API_ListShards.html>).
> > This was recently introduced in the aws-sdk-java release of 1.11.272
> > <https://github.com/aws/aws-sdk-java/releases/tag/1.11.272>. I propose
> > bumping up the aws-sdk-java used in flink-kinesis connector and replace
> the
> > DescribeStream calls with ListShards in the KinesisProxy class here
> > <https://github.com/apache/flink/blob/master/flink-
> connectors/flink-connector-kinesis/src/main/java/org/
> apache/flink/streaming/connectors/kinesis/proxy/KinesisProxy.java>
> allowing
> > for faster shard discovery rate.
> >
> > Thanks
> > Kailash
> >
>
