[
https://issues.apache.org/jira/browse/FLINK-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107888#comment-16107888
]
Stephan Ewen commented on FLINK-7223:
-------------------------------------
Hmmm... Is there a way to auto-configure this value as in the following way:
- The Flink job would theoretically do discovery once per 5 seconds (this
means running up to 50 Flink jobs er account)
- The interval in which the TaskManagers can discover is than {{5 seconds x
source parallelism}}
> Increase DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS for Flink-kinesis-connector
> ----------------------------------------------------------------------------
>
> Key: FLINK-7223
> URL: https://issues.apache.org/jira/browse/FLINK-7223
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
> Affects Versions: 1.3.0
> Reporter: Bowen Li
> Assignee: Bowen Li
> Priority: Minor
> Fix For: 1.4.0
>
>
> Background: {{DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS}} in
> {{org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants}}
> is the default value for Flink to call Kinesis's {{describeStream()}} API.
> Problem: Right now, its value is 10,000millis (10sec), which is too short. We
> ran into problems that Flink-kinesis-connector's call of {{describeStream()}}
> exceeds Kinesis rate limit, and broken Flink taskmanager.
> According to
> http://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStream.html,
>
> "This operation has a limit of 10 transactions per second per account.". What
> it means is that the 10transaction/account is a limit on a single
> organization's AWS account......:( We contacted AWS Support, and confirmed
> this. If you have more applications (either other Flink apps or non-Flink
> apps) competing aggressively with your Flink app on this API, your Flink app
> breaks.
> I propose increasing the value DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS from
> 10,000millis(10sec) to preferably 300,000 (5min). Or at least 60,000 (1min)
> if anyone has a solid reason arguing that 5min is too long,
> This is also related to https://issues.apache.org/jira/browse/FLINK-6365
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
