ameihm0912 commented on issue #10973: KinesisIO retry LimitExceededException URL: https://github.com/apache/beam/pull/10973#issuecomment-591642429 > Thank you for contribution! Have you seen this PR #9765 and a discussion over this topic there? I think it tries to solve the similar problem, if I'm not mistaken. @aromanenko-dev thank you for the response. I had a quick look at the referenced PR and I suspect that may be solving a different issue, that being transient errors associated with hitting read throughput rate limiting. Some additional context on the issue we are running into: in our case, we have a large number of Kinesis streams in a single account. By default, Kinesis `DescribeStream` calls are limited to 10 per second per account. So we easily run into this during graph construction as we have a large number of streams that need to be setup in the same pipeline. This results in a `RuntimeException` being thrown by the pipeline. So, our issue is happening during stream initialization right off the bat, where as the referenced PR I believe is more to address issues that are occurring while reading from an already initialized stream. It is possible I am misinterpreting something in the other PR however.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
