Aaron Meihm created BEAM-9476:
---------------------------------
Summary: KinesisIO DescribeStream transient errors are not retried
Key: BEAM-9476
URL: https://issues.apache.org/jira/browse/BEAM-9476
Project: Beam
Issue Type: Bug
Components: io-java-kinesis
Reporter: Aaron Meihm
During Kinesis stream setup {{DescribeStream}} is used in {{listShards}}. This
API call has quota limits that can become problematic when attempting to
configure multiple Kinesis streams in the same AWS account. AWS currently
limits this call by default to 10 times per second for a given account. With
multiple streams, this can be hit and results in a {{RuntimeException}} being
thrown immediately upon the first error.
Ideally the rate limited call can be retried a number of times before giving up
instead of failing immediately.
I have a work in progress PR that resolves this issue at
https://github.com/apache/beam/pull/10973.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
