tannnmayy opened a new pull request, #41359: URL: https://github.com/apache/beam/pull/41359
`AnthropicModelHandler` creates the client with its default `max_retries=2`, so the client retries inside each of `RemoteModelHandler`'s attempts. A request that keeps failing with 429 or 500 is sent 18 times instead of 6, and Beam's throttler only sees 6 of those failures. `OpenAIModelHandler` already sets `max_retries=0` for this reason. This PR: - creates the client with `max_retries=0`; - retries `APIConnectionError` in the retry filter. Until now only the client retried connection errors, so without this they would no longer be retried. Testing: - Updated and new tests in `anthropic_inference_test.py`. - HTTP requests counted through the real client with a mock transport (anthropic 2.4.0). With 429 or 500 on every request: 18 requests on master, 6 with this change. With a connection error on the first 3 requests: the batch fails on master and succeeds on the 4th request with this change. This continues #40914 and #40929: I've been going through how the remote LLM handlers behave when requests fail. If there's something else in this area you'd rather I work on, I'm happy to do that instead. R: @jrmccluskey -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
