tannnmayy opened a new pull request, #41361:
URL: https://github.com/apache/beam/pull/41361

   Fixes #41360
   
   `GeminiModelHandler` and `OpenAIModelHandler` set no default 
`max_batch_size`, so default batches grow to about 1,000 prompts, and to 10,000 
inputs with `embedding_from_string`, which is over the 2,048 inputs the 
embeddings endpoint accepts. Per-prompt requests are only sent 32 at a time, 
and a failed request retries the whole batch.
   
   This PR defaults `max_batch_size` to 32, or to 2,048 with 
`embedding_from_string`, and never below `min_batch_size`. An explicit 
`max_batch_size` behaves as before.
   
   Testing:
   - New tests in `gemini_inference_test.py` and `openai_inference_test.py`. 
They fail on master.
   - Pipelines with fake clients and default settings: embedding requests went 
from up to 10,000 inputs (7 of 20 over the limit) to at most 2,048 (0 of 43). 
Gemini batches went from about 1,000 prompts to 32; 3,000 prompts at 0.5 s per 
call took 51 s before and 53 s after.
   
   This is from the same pass over the remote LLM handlers as #40914 and 
#41359. I'm happy to change the default values if you'd pick different ones, 
and if something else in this area would be more useful for me to work on, I'd 
be glad to.
   
   R: @jrmccluskey
   cc: @claudevdm
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to