tvalentyn commented on code in PR #35234:
URL: https://github.com/apache/beam/pull/35234#discussion_r2139191584
##########
sdks/python/apache_beam/ml/inference/vllm_inference.py:
##########
@@ -114,28 +114,30 @@ def __init__(self, model_name: str, vllm_server_kwargs:
dict[str, str]):
self._server_started = False
self._server_process = None
self._server_port: int = -1
+ self._server_process_lock = threading.RLock()
Review Comment:
When large_model=true, multiple processes share the server through a single
server handle shared across processes, and only one process is responsible for
managing the handle.
The problem is that right now we call `check_connectivity` from an async
method:
https://github.com/apache/beam/blob/de76cfa16d36e2cb8dd1a1092710f0cd477e0f8d/sdks/python/apache_beam/ml/inference/vllm_inference.py#L219
and `check_connectivity` happens to also start the server if the process
already terminated, so as a result we create many processes at once (see
internal: b/422577332), and overwrite `self._server_process`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
