tannnmayy opened a new pull request, #41359:
URL: https://github.com/apache/beam/pull/41359

   `AnthropicModelHandler` creates the client with its default `max_retries=2`, 
so the client retries inside each of `RemoteModelHandler`'s attempts. A request 
that keeps failing with 429 or 500 is sent 18 times instead of 6, and Beam's 
throttler only sees 6 of those failures. `OpenAIModelHandler` already sets 
`max_retries=0` for this reason.
   
   This PR:
   - creates the client with `max_retries=0`;
   - retries `APIConnectionError` in the retry filter. Until now only the 
client retried connection errors, so without this they would no longer be 
retried.
   
   Testing:
   - Updated and new tests in `anthropic_inference_test.py`.
   - HTTP requests counted through the real client with a mock transport 
(anthropic 2.4.0). With 429 or 500 on every request: 18 requests on master, 6 
with this change. With a connection error on the first 3 requests: the batch 
fails on master and succeeds on the 4th request with this change.
   
   This continues #40914 and #40929: I've been going through how the remote LLM 
handlers behave when requests fail. If there's something else in this area 
you'd rather I work on, I'm happy to do that instead.
   
   R: @jrmccluskey
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] [Python] Make Beam the on... tannnmayy (via GitHub) via github

Reply via email to