Hello Bowen, Thank you very much for your feedback. I had a question with regard to your concern on compatibilities issues
I notice that aws-java-sdk imported in the flink connector is shaded. So it is not likely to cause conflicts in classpath due to AWS EMR depending on a older version of the SDK. Also KCL and KPL are depending on the aws-java-sdk. Since minimum required aws-java-sdk version for KCL and KPL is earlier than the the proposed version, it should not be a problem. I checked for if there are any changes breaking the backward compatibility between the new version and minimum version required by KCL and KPL. Looks like there are not any such changes. Do you think there is something else which I am missing here? Thanks Kailash On Mon, Mar 5, 2018 at 3:20 PM, Bowen Li <bowenl...@gmail.com> wrote: > Good for AWS has increased the limit from 5txn/s last year to 10txn/s > > On Mon, Mar 5, 2018 at 8:32 AM, Thomas Weise <t...@apache.org> wrote: > > > On Sun, Mar 4, 2018 at 11:38 PM, Bowen Li <bowenl...@gmail.com> wrote: > > > > > If ListShards() gives all the info that Flink needs, +1 on switching. > > > DescribeStreams() has a limitation of 5 requests/sec, which is pretty > > > bad.... > > > > > > But, I believe the goal of switching APIs should be *making Flink jobs > > that > > > read from Kinesis more stable*, rather than having faster shard > discovery > > > rate. The default shard discovery rate is every 10s, which is already > > very > > > very fast and can satisfy most Kinesis users, we shouldn't shorten the > > > default value anymore. Developers who want faster discovery rate than > 10s > > > should overwrite the default value themselves. > > > > > > > > https://docs.aws.amazon.com/kinesis/latest/APIReference/ > > API_DescribeStream.html > > > > "This operation has a limit of 10 transactions per second per account." > > > > 10s interval would be sufficient, but with 5 requests per second and a > > larger number of subtasks and multiple applications in the account, we > are > > looking at discovery rates in double digit minutes. > > > > That's where ListShards will help. Of course, the consumer should still > be > > redesigned to centralize the discovery. > > > > Thanks, > > Thomas > > >
