Hong Liang Teoh created FLINK-19226:
---------------------------------------
Summary: [Kinesis] [EFO] Connector reaches default max attempts
for describeStream and describeStreamConsumer when parallelism is high
Key: FLINK-19226
URL: https://issues.apache.org/jira/browse/FLINK-19226
Project: Flink
Issue Type: Improvement
Components: Connectors / Kinesis
Reporter: Hong Liang Teoh
Fix For: 1.12.0
*Background*
When lazily registering the stream consumer on apps with high parallelism, EFO
connector hits default maximum number of attempts when calling describeStream
and describeStreamConsumer on the Kinesis Streams API.
The default FullJitterBackoff constants are tuned to prevent this when
parallelism of 1024 is used.
*Scope*
* See
[FLIP|https://cwiki.apache.org/confluence/display/FLINK/FLIP-128%3A+Enhanced+Fan+Out+for+AWS+Kinesis+Consumers]
for full list of configuration options
* Suggested changes:
|flink.stream.describe.maxretries|50|
|flink.stream.describe.backoff.base|2000L|
|flink.stream.describe.backoff.max|5000L|
|flink.stream.describestreamconsumer.maxretries|50|
|flink.stream.describestreamconsumer.backoff.base|2000L|
|flink.stream.describestreamconsumer.backoff.max|5000L|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
