[
https://issues.apache.org/jira/browse/FLINK-6365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094159#comment-16094159
]
Tzu-Li (Gordon) Tai commented on FLINK-6365:
--------------------------------------------
Alright, I think [~sthm] definitely makes sense here. I'll also agree to
increasing {{SHARD_GETRECORDS_INTERVAL}} to 200ms.
One correction to your description [~phoenixjiangnan]: each shard will only
ever be picked up by a single parallel source instance across instances of a
single {{FlinkKinesisConsumer}} (unless you're referring to the case where you
have 2 jobs both consuming the same Kinesis stream).
Regarding the fact that {{SHARD_GETRECORDS_INTERVAL}} is currently not a strict
interval: there's actually a JIRA and PR for that - FLINK-4574. It's a more
critical change that needs more time for careful reviewing, so I haven't really
got back to reviewing that yet. Would be very helpful if you're interested in
digging into that and reviewing it also!
> Adapt default values of the Kinesis connector
> ---------------------------------------------
>
> Key: FLINK-6365
> URL: https://issues.apache.org/jira/browse/FLINK-6365
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
> Affects Versions: 1.2.0
> Reporter: Steffen Hausmann
> Assignee: Bowen Li
> Priority: Minor
> Fix For: 1.4.0, 1.3.2
>
>
> As discussed in
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-connector-SHARD-GETRECORDS-MAX-default-value-td12332.html,
> it seems reasonable to change the default values of the Kinesis connector to
> follow KCL’s default settings. I suggest to adapt at least the values for
> SHARD_GETRECORDS_MAX and SHARD_GETRECORDS_INTERVAL_MILLIS.
> As a Kinesis shard is currently limited to 5 get operations per second, you
> can observe high ReadProvisionedThroughputExceeded rates with the current
> default value for SHARD_GETRECORDS_INTERVAL_MILLIS of 0; it seem reasonable
> to increase it to 200. As it's described in the email thread, it seems
> furthermore desirable to increase the default value for SHARD_GETRECORDS_MAX
> to 10000.
> The values that are used by the KCL can be found here:
> https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java
> Thanks for looking into this!
> Steffen
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
