AMOOOMA commented on code in PR #40908:
URL: https://github.com/apache/beam/pull/40908#discussion_r4566491194
##########
sdks/python/apache_beam/ml/inference/vllm_inference.py:
##########
@@ -370,11 +406,14 @@ def __init__(
dynamo_frontend_kwargs: Optional[dict[str, Optional[str]]] = None,
min_batch_size: Optional[int] = None,
max_batch_size: Optional[int] = None,
- max_batch_duration_secs: Optional[int] = None,
+ max_batch_duration_secs: Optional[float] = None,
max_batch_weight: Optional[int] = None,
element_size_fn: Optional[Callable[[Any], int]] = None,
batch_length_fn: Optional[Callable[[Any], int]] = None,
- batch_bucket_boundaries: Optional[list[int]] = None):
+ batch_bucket_boundaries: Optional[list[int]] = None,
+ prefix_batching_config: Optional[PrefixBatchingConfig] = None,
+ enable_autoscaling_metric: bool = True,
Review Comment:
good point yeah, I was just under the habit of turning everything into a
knob but i agree this one i think we can just make it default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
