tannnmayy commented on PR #41361:
URL: https://github.com/apache/beam/pull/41361#issuecomment-7022292112

   Thanks @claudevdm, without `max_batch_weight`, `max_batch_size` is counted 
in `element_size_fn` units, so the default put every prompt in its own batch.
   
   I've added your change to both handlers, along with the docstring note and 
your tests. With your setup it now gives 35 batches in 1.2 s, the same as 
master (the previous commit gave 3,000 batches of one in 33 s). Without 
`element_size_fn`, the default is unchanged. All 66 tests in both files pass.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to