[
https://issues.apache.org/jira/browse/FLINK-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093498#comment-16093498
]
Bowen Li edited comment on FLINK-7223 at 7/20/17 4:24 AM:
----------------------------------------------------------
I believe what Gordon and I proposed are complementary. We need to do both. We
can create another ticket for what Gordon proposed for code change (simply
because I don't have time to do that now), and link that to this ticket.
10sec is simply too short and intensive. What do you think about 3min or 1min?
AWS takes about 5 minutes to update kinesics shard itself, so users are of
right expectation that Flink takes the same amount of time.
was (Author: phoenixjiangnan):
I believe what Gordon and I proposed are complementary. We need to do both. We
can create another ticket for what Gordon proposed for code change, and link it
to this ticket.
10sec is simply too short and intensive. What do you think about 3min or 1min?
AWS takes about 5 minutes to update kinesics shard itself, so users are of
right expectation that Flink takes the same amount of time.
> Increase DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS for Flink-kinesis-connector
> ----------------------------------------------------------------------------
>
> Key: FLINK-7223
> URL: https://issues.apache.org/jira/browse/FLINK-7223
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
> Affects Versions: 1.3.0
> Reporter: Bowen Li
> Assignee: Bowen Li
> Priority: Minor
> Fix For: 1.4.0
>
>
> Background: {{DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS}} in
> {{org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants}}
> is the default value for Flink to call Kinesis's {{describeStream()}} API.
> Problem: Right now, its value is 10,000millis (10sec), which is too short. We
> ran into problems that Flink-kinesis-connector's call of {{describeStream()}}
> exceeds Kinesis rate limit, and broken Flink taskmanager.
> According to
> http://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStream.html,
>
> "This operation has a limit of 10 transactions per second per account.". What
> it means is that the 10transaction/account is a limit on a single
> organization's AWS account......:( We contacted AWS Support, and confirmed
> this. If you have more applications (either other Flink apps or non-Flink
> apps) competing aggressively with your Flink app on this API, your Flink app
> breaks.
> I propose increasing the value DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS from
> 10,000millis(10sec) to preferably 300,000 (5min). Or at least 60,000 (1min)
> if anyone has a solid reason arguing that 5min is too long,
> This is also related to https://issues.apache.org/jira/browse/FLINK-6365
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
