Zhuoxi2000 opened a new pull request, #945: URL: https://github.com/apache/flink-agents/pull/945
Linked issue: #928 ### Purpose of change Add a dedicated vLLM chat model connection out of the box, in both Java and Python, per the direction agreed on #928 (dedicated connection over alias-only, for discoverability). vLLM exposes an OpenAI-compatible API, so both connections reuse the existing OpenAI integrations with vLLM-friendly defaults: - `api_base_url` defaults to `http://localhost:8000/v1` (the default address of `vllm serve`). - `api_key` defaults to a placeholder: vLLM servers started without `--api-key` require no credential, while both OpenAI SDKs require a non-empty key — an alias-only route would have forced users to pass a dummy key by hand. One deliberate difference from the OpenAI setups: `model` is **required** and has no default, because a vLLM server only serves the model(s) it was started with. **Implementation description** - Java: `VLLMChatModelConnection` / `VLLMChatModelSetup` live in the existing `integrations/chat-models/openai` module (same packaging as `AzureOpenAIChatModelConnection`, so no new Maven module or distribution change). The connection injects defaults by rewriting the `ResourceDescriptor` before delegating to `OpenAICompletionsConnection`; the caller's descriptor is not mutated. The setup delegates to `OpenAICompletionsSetup` and only adds the model-required validation. - Python: `flink_agents/integrations/chat_models/vllm/vllm_chat_model.py`, subclassing the OpenAI connection/setup the same way, mirroring the `chat_models/azure` package layout. - Registered in `ResourceName.ChatModel` (Java + `Python` inner class) and in the Python `ResourceName`, plus the `vllm` YAML alias for both languages in `Aliases.java` and `aliases.py`. - Failure behavior: a missing/blank `model` on the setup raises `IllegalArgumentException` / `ValueError` at construction with a message pointing at `vllm serve <model>` / `GET /v1/models`. Everything else (timeouts, retries, error wrapping) is inherited unchanged. - Compatibility: additive only; no existing behavior changes. ### Tests - Java: `VLLMChatModelConnectionTest` (defaults injected — construction succeeds with no arguments where the parent would throw on missing `api_key`; blank args defaulted; explicit args honored; caller descriptor not mutated) and `VLLMChatModelSetupTest` (model required/blank rejected; `getParameters` carries model + inherited fields). Azure/OpenAI module tests pass unchanged. - Python: `test_vllm_chat_model.py` (connection defaults + overrides, setup model-required, `model_kwargs` passthrough). ### API New public classes/constants (`VLLM_CONNECTION` / `VLLM_SETUP`, `vllm` alias); no changes to existing APIs. ### Documentation - [x] `doc-included` — new "vLLM" provider section in `chat_models.md` and a `vllm` row in the YAML alias table. --- Generative-AI disclosure (per the direction in #932): this change was developed with AI assistance; I reviewed the design, code and tests, and ran the test suites locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
