Re: Proposal - Change shard discovery in Flink Kinesis Connector to use ListShards

Hello Bowen, Thank you very much for your feedback. I had a question with regard to your concern on compatibilities issues I notice that aws-java-sdk imported in the flink connector is shaded. So it is not likely to cause conflicts in classpath due to AWS EMR depending on a older version of the S

Re: Proposal - Change shard discovery in Flink Kinesis Connector to use ListShards

Good for AWS has increased the limit from 5txn/s last year to 10txn/s On Mon, Mar 5, 2018 at 8:32 AM, Thomas Weise wrote: > On Sun, Mar 4, 2018 at 11:38 PM, Bowen Li wrote: > > > If ListShards() gives all the info that Flink needs, +1 on switching. > > DescribeStreams() has a limitation of 5 re

Re: Proposal - Change shard discovery in Flink Kinesis Connector to use ListShards

On Sun, Mar 4, 2018 at 11:38 PM, Bowen Li wrote: > If ListShards() gives all the info that Flink needs, +1 on switching. > DescribeStreams() has a limitation of 5 requests/sec, which is pretty > bad > > But, I believe the goal of switching APIs should be *making Flink jobs that > read from Ki

Re: Proposal - Change shard discovery in Flink Kinesis Connector to use ListShards

Hi, +1, I think the new ListShards API should provide all the information we need. In the past, we were restricted to update the SDK version because it would break state compatibility (the consumer used to write AWS classes inside state). I think this was fixed since Flink 1.3, so it should be s

Re: Proposal - Change shard discovery in Flink Kinesis Connector to use ListShards

Btw, could you also open a JIRA to track this improvement? Thanks! On 5 March 2018 at 5:47:09 PM, Tzu-Li (Gordon) Tai (tzuli...@apache.org) wrote: Hi, +1, I think the new ListShards API should provide all the information we need. In the past, we were restricted to update the SDK version becaus

Re: Proposal - Change shard discovery in Flink Kinesis Connector to use ListShards

If ListShards() gives all the info that Flink needs, +1 on switching. DescribeStreams() has a limitation of 5 requests/sec, which is pretty bad But, I believe the goal of switching APIs should be *making Flink jobs that read from Kinesis more stable*, rather than having faster shard discovery

Re: Proposal - Change shard discovery in Flink Kinesis Connector to use ListShards

It will be good to be able to use the ListShards API. Are there any concerns bumping up the AWS SDK dependency? I see it was last done in https://issues.apache.org/jira/browse/FLINK-7422 Thanks On Wed, Feb 28, 2018 at 10:38 PM, Kailash Dayanand wrote: > Based on the discussion at here >