tannnmayy commented on PR #41361: URL: https://github.com/apache/beam/pull/41361#issuecomment-7022292112
Thanks @claudevdm, without `max_batch_weight`, `max_batch_size` is counted in `element_size_fn` units, so the default put every prompt in its own batch. I've added your change to both handlers, along with the docstring note and your tests. With your setup it now gives 35 batches in 1.2 s, the same as master (the previous commit gave 3,000 batches of one in 33 s). Without `element_size_fn`, the default is unchanged. All 66 tests in both files pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
