Aaron Meihm created BEAM-9476: --------------------------------- Summary: KinesisIO DescribeStream transient errors are not retried Key: BEAM-9476 URL: https://issues.apache.org/jira/browse/BEAM-9476 Project: Beam Issue Type: Bug Components: io-java-kinesis Reporter: Aaron Meihm
During Kinesis stream setup {{DescribeStream}} is used in {{listShards}}. This API call has quota limits that can become problematic when attempting to configure multiple Kinesis streams in the same AWS account. AWS currently limits this call by default to 10 times per second for a given account. With multiple streams, this can be hit and results in a {{RuntimeException}} being thrown immediately upon the first error. Ideally the rate limited call can be retried a number of times before giving up instead of failing immediately. I have a work in progress PR that resolves this issue at https://github.com/apache/beam/pull/10973. -- This message was sent by Atlassian Jira (v8.3.4#803005)