ambiguous-pointer opened a new issue, #1574:
URL: https://github.com/apache/dubbo-admin/issues/1574
<!-- Please only use this template for submitting feature requests -->
**What would you like to be added**:
A vendor-neutral **protocol and capability abstraction** for the AI module's
LLM layer, so that supporting a new model vendor or a new wire protocol is a
matter of configuration plus one adapter, instead of an intrusive refactor.
Concretely, I would like the model layer to gain four things:
1. **An explicit `protocol` dimension in the provider registry.** Today a
provider entry implies exactly one wire protocol. I would like to declare it,
for example:
```yaml
providers:
deepseek:
base_url: "https://api.deepseek.com/v1";
protocol: chat_completions # chat_completions | messages |
responses | <custom>
```
Adding a protocol should mean writing one adapter and registering it —
not touching the agent, tools, or server packages.
2. **A declared and validated capability set per provider/model.** Something
like `capabilities: { tools: true, media: false, reasoning: true, system_role:
true }`, validated at startup and used to gate agent behaviour.
This one matters a lot for this project: the agent is a ReAct loop whose
control flow is entirely built on tool calling. An adapter that cannot do tool
calling must **fail fast at startup**, not silently produce a degraded agent.
(As a data point, the Anthropic adapter currently available to us is
OpenAI-compatible and declares `Tools: false`, so it cannot back the ReAct loop
at all — the failure mode to avoid is discovering that at request time.)
3. **Provider- and model-specific parameters that are actually honoured.**
The configuration schema already accepts a free-form `config` object at
provider, model and embedder level, but nothing consumes it. I would like those
values to reach model registration and per-request generation options —
sampling defaults, extra request-body fields (for example `enable_thinking`,
`reasoning_effort`), provider-specific headers, timeouts, and so on — so that
vendor-specific behaviour is configuration rather than code.
4. **Reasoning / thinking content surfaced end-to-end.** When a provider
returns a structured reasoning block (DeepSeek's thinking output, Anthropic's
thinking blocks, etc.), I would like it to be opt-in configurable per provider
and carried all the way through: adapter → reasoning content part → SSE event →
UI rendering. The transport layer already defines a `thinking_delta` delta type
that nothing currently emits, so part of this plumbing appears to be
anticipated already.
**Optional / later, but worth designing for now:** a control-plane API (and
eventually UI) to manage the registry — list providers and models, validate a
candidate configuration, and apply changes with explicitly defined reload
semantics. I am happy for this to be a separate, later issue; I mention it only
because it constrains how the registry should be modelled.
I would also suggest **extending the existing `models.yaml` registry and its
JSON Schema rather than introducing a second configuration format** (e.g.
`models.json`). This project already has a single consistent convention —
`config.yaml` referencing per-component YAML files validated against
`ai/schema/json/*.schema.json` — and adding another file format for one
component would make that convention harder to reason about without adding any
capability.
### Current behaviour (for reference / correction)
I want to be accurate about what already exists, because part of what I
originally assumed turned out to be wrong when I checked the code, and I do not
want to ask for work that is already done:
- A **multi-provider registry already exists**:
`ai/component/models/models.yaml`, `ai/component/models/config.go`,
`ai/schema/json/models.schema.json`. `providers` is an arbitrary map of `{
api_key, base_url, models[], embedders[], config }`, `api_key` supports
`${ENV}` expansion (`ai/config/loader.go` uses `os.ExpandEnv`), and any
OpenAI-compatible endpoint can be registered with configuration only — I
verified that adding new providers (for example DeepSeek and a local Ollama
endpoint) registers working models without touching any Go code.
- Requests already go over **`chat/completions`**, not the legacy OpenAI
`completions` endpoint.
- What is genuinely missing is the **protocol/capability dimension**,
consumption of the existing **`config`** fields, and the **reasoning**
pipeline. Two smaller issues are worth fixing in the same pass, because both
fail silently today: the `gemini` provider is hard-coded to be skipped
(`ai/component/models/component.go`, `createModelPlugin` returns `nil` for it)
even though the shipped `models.yaml` configures it, and providers without an
`api_key` are skipped without any startup diagnostic.
**Why is this needed**:
1. **Protocol rigidity is real, but it is a capability problem rather than a
dialect problem.** Because `provider` currently conflates "namespace",
"endpoint" and "wire protocol", the model layer can only ever talk to
OpenAI-compatible endpoints. Any vendor whose native API differs — Anthropic's
`messages`, or a custom enterprise streaming schema — requires code changes
across the model layer, and the only ready-made alternative available to us
loses tool calling, which makes it unusable for the agent. Declaring `protocol`
and `capabilities` explicitly turns that into "add one adapter, reuse
everything else".
2. **Vendor-specific features are lost at the model boundary.** Advanced
provider behaviour (streaming raw reasoning, provider-specific system-prompt
handling, vendor request flags) has nowhere to be expressed today: the
configuration slots exist but are ignored, so the data is dropped before it can
reach the UI. For a governance toolchain that is expected to front different
models, this is a concrete functional limitation, not just a cosmetic one.
3. **Enterprise deployments need to pivot between vendors and private
endpoints without code changes.** Cost, compliance, data residency and local
deployment (private gateways, Ollama, on-prem inference) all require switching
providers. Configuration-only switching is already the norm for endpoints; what
is missing is doing the same for protocol and capability differences, with
validation up front instead of silent degradation at request time.
4. **Silent misconfiguration is a maintenance hazard.** Right now a provider
that is skipped (unsupported name, missing key) produces no error and no
warning — a misconfigured deployment looks healthy until a user reports that a
model is unavailable. Startup validation with clear diagnostics should be part
of this work.
5. **Consistency with the rest of the module.** The agent, hook, tooling and
tracing layers have been progressively componentized with schema-validated
configuration and declared contracts. The model layer is the remaining place
where behaviour is implied by an implicit assumption ("everything is
OpenAI-compatible"), and it should be brought in line.
### Acceptance criteria (suggested)
- [ ] A provider entry can declare `protocol`; an unknown protocol is
rejected at startup with a clear error.
- [ ] A provider/model can declare `capabilities`; a protocol adapter that
cannot satisfy the capabilities required by the configured agent (at minimum
`tools`) fails validation, rather than degrading silently.
- [ ] `config` at provider, model and embedder level reaches the
corresponding registration/request options, verified by a test asserting the
value is present in the outgoing request.
- [ ] Disabling a provider by removing its key, or configuring an
unsupported provider, produces a startup warning or error — never a silent skip.
- [ ] Reasoning content can be enabled per provider and is observable
end-to-end (SSE event emitted, rendered in the chat UI), verified by an
integration test against at least one provider that returns reasoning.
- [ ] At least one non-OpenAI protocol adapter exists **with tool calling**,
covered by a tool-calling round-trip test.
- [ ] Existing `models.yaml` configurations remain valid (no breaking change
to the current schema).
<details>
<summary><b>🇨🇳 中文版 (Chinese version)</b></summary>
**希望新增的功能**：
为 AI 模块的 LLM 层引入一套**厂商中立的协议与能力抽象**，让"接入新的模型厂商"或"支持新的传输协议"变成"配置 +
一个适配器"的事情，而不是一次侵入式重构。
具体希望模型层具备四点能力：
1. **在 provider 注册表中显式声明 `protocol`**。现在一个 provider 条目隐含了唯一一种传输协议，我希望能够声明：
```yaml
providers:
deepseek:
base_url: "https://api.deepseek.com/v1";
protocol: chat_completions # chat_completions | messages |
responses | <custom>
```
新增一种协议应当意味着"写一个适配器并注册"，而不是改动 agent、tools、server 等包。
2. **按 provider/model 声明并校验能力集**，例如 `capabilities: { tools: true, media:
false, reasoning: true, system_role: true }`，在启动时校验，并据此约束 agent 行为。
这一条对本项目尤其重要：agent 是 ReAct
循环，整个控制流建立在工具调用之上。**无法进行工具调用的适配器必须在启动时快速失败**，而不是悄悄产出一个退化的
agent。（作为一个现实数据点：目前我们能拿到的 Anthropic 适配器是 OpenAI 兼容实现且声明 `Tools: false`，它根本无法支撑
ReAct 循环；要避免的正是"到请求时才暴露"这种失败方式。）
3. **让 provider/model 专属参数真正生效**。配置 schema 已经允许在 provider、model、embedder
三个层级写自由格式的 `config`，但没有任何代码消费它。我希望这些值能够传递到模型注册与按请求的生成参数上——采样默认值、额外请求体字段（如
`enable_thinking`、`reasoning_effort`）、provider 专属
header、超时等——从而让厂商特性由配置驱动而不是写死在代码里。
4. **打通 reasoning / thinking 内容的端到端链路**。当某个 provider 返回结构化推理内容（DeepSeek
的思考输出、Anthropic 的 thinking 块等）时，我希望它能够按 provider 显式开关，并完整贯穿：适配层 → 推理内容 part →
SSE 事件 → UI 渲染。传输层其实已经定义了一个当前无人使用的 `thinking_delta` 事件类型，说明这部分管道可能早有预留。
**可选/后续（但建议现在就纳入设计）**：提供控制面 API（后续再做 UI）来管理注册表——列出 provider
与模型、校验候选配置、并以明确的重载语义应用变更。这一项完全可以拆成后续 issue，我在这里提及它只是因为会影响注册表该如何建模。
同时我建议**扩展现有的 `models.yaml` 注册表及其 JSON Schema，而不是引入第二种配置格式**（例如
`models.json`）。本项目已经有一套一致的约定：`config.yaml` 引用各组件 YAML，并用
`ai/schema/json/*.schema.json` 校验；为单个组件再加一种文件格式，会降低这套约定的一致性与可推理程度，却不会带来任何额外能力。
### 当前行为（供参考与修正）
我希望对"已经存在什么"保持准确——因为我最初的一些假设在核对代码后发现是错的，不想把已经做完的事情再提成需求：
- **多 provider
注册表已经存在**：`ai/component/models/models.yaml`、`ai/component/models/config.go`、`ai/schema/json/models.schema.json`。`providers`
是任意 map，条目包含 `{ api_key, base_url, models[], embedders[], config }`；`api_key`
支持 `${ENV}` 展开（`ai/config/loader.go` 使用 `os.ExpandEnv`）；任意 OpenAI
兼容端点只需配置即可注册——我验证过新增 provider（例如 DeepSeek 与本地 Ollama 端点）无需改动任何 Go 代码即可注册出可用模型。
- 请求走的是 **`chat/completions`**，并非最古老的 OpenAI `completions` 端点。
- 真正缺失的是**协议/能力维度**、对已有 **`config`** 字段的消费，以及 **reasoning**
链路。另外有两个小问题建议在同一轮修掉，因为它们目前都是静默失败：`gemini`
被硬编码跳过（`ai/component/models/component.go` 中 `createModelPlugin` 对它直接返回
`nil`），但随仓库发布的 `models.yaml` 却完整配置了它；没有 `api_key` 的 provider 会被静默跳过，启动时没有任何诊断信息。
**为什么需要这个功能**：
1. **协议僵化确实存在，但本质是能力问题而非方言问题。** 由于 `provider`
目前同时混同了"命名空间""端点""传输协议"三种职责，模型层只能与 OpenAI 兼容端点通信。任何原生 API 不同的厂商——Anthropic 的
`messages`、或企业自定义的流式 schema——都需要改动模型层代码；而目前唯一现成的替代实现丢掉了工具调用能力，对 agent
而言不可用。显式声明 `protocol` 与 `capabilities`，可以把这件事变成"新增一个适配器，其余全部复用"。
2. **厂商差异化特性在模型边界处被丢弃。** 高级 provider 行为（流式输出原始推理过程、厂商特定的 system prompt
处理、厂商请求参数开关）目前无处表达：配置槽位存在却被忽略，数据在到达 UI
之前就已经丢失。对一个需要对接不同模型的治理工具链来说，这是明确的功能性限制，而非外观问题。
3. **企业部署需要在不改代码的前提下在厂商与私有端点之间切换。** 成本、合规、数据驻留与本地化部署（私有网关、Ollama、本地推理）都要求可切换
provider。端点层面的纯配置切换已是常态；缺的是把协议与能力差异也纳入同一机制，并在启动时就完成校验，而不是在请求期静默降级。
4. **静默错误配置是维护隐患。** 目前被跳过的 provider（不支持的名称、缺失的
key）既不报错也不告警，配置错误的部署看起来一切正常，直到用户反馈模型不可用。启动校验与清晰诊断应当是本项工作的一部分。
5. **与模块其余部分保持一致。** agent、hook、工具与追踪等层已经逐步组件化，具备 schema
校验的配置与显式契约。模型层是最后一处"行为由隐含假设决定"（即"一切皆 OpenAI 兼容"）的地方，应当一并纳入。
### 建议的验收标准
- [ ] provider 条目可声明 `protocol`；未知协议在启动时报错并给出明确提示。
- [ ] provider/model 可声明 `capabilities`；无法满足所配置 agent 所需能力（至少
`tools`）的协议适配器在启动校验阶段失败，而不是静默降级。
- [ ] provider、model、embedder 三个层级的 `config`
均能传递到对应的注册/请求参数，并有测试断言该值出现在实际发出的请求中。
- [ ] 未配置 key 的 provider、以及配置了不支持的 provider，都会产生启动告警或错误，绝不静默跳过。
- [ ] reasoning 内容可按 provider 开启，并可端到端观测（SSE 事件被发出、聊天 UI 中可渲染），至少用一个会返回推理内容的
provider 做集成测试验证。
- [ ] 至少提供一个非 OpenAI 协议的适配器，且**支持工具调用**，并有工具调用往返测试覆盖。
- [ ] 现有 `models.yaml` 配置保持可用（对当前 schema 无破坏性变更）。
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
