hengyuss opened a new issue, #7351:
URL: https://github.com/apache/shenyu/issues/7351
### Description
Description
## Background
The current AI Proxy implementation is tightly coupled to OpenAI Chat
Completions and Spring AI types:
- `AiProxyPlugin` directly handles OpenAI request/response DTOs and SSE
encoding.
- `AiProxyExecutorService` mixes upstream invocation, retry, and fallback
logic.
- Adding protocols such as OpenAI Responses or Anthropic Messages requires
modifying the main execution flow.
- Unknown provider-specific fields may be lost during DTO conversion.
This refactor refers to the layered design of [Apache APISIX AI
Proxy](https://github.com/apache/apisix/tree/master/
apisix/plugins).
## Proposal
Refactor AI Proxy into three extensible layers:
### Protocol
Responsible for:
- Detecting the client API format.
- Parsing and validating requests.
- Determining whether a request is streaming.
- Converting requests and responses between protocols.
- Handling protocol-specific streaming events.
Initial implementation:
- `openai-chat`
Future extensions may include:
- `openai-responses`
- `anthropic-messages`
- `openai-embeddings`
### Provider
Responsible for:
- Declaring supported protocols.
- Resolving upstream endpoints.
- Applying authentication, headers, and query parameters.
- Handling provider-specific request fields.
Initial implementations:
- `openai`
- `deepseek`
- `openai-compatible`
### Transport
Responsible for:
- Reactive HTTP and SSE communication.
- Timeouts, size limits, and connection cancellation.
- Preserving upstream response status, headers, and error bodies.
The Transport layer must not contain Provider- or Protocol-specific logic.
The request flow should be:
```text
AiProxyPlugin
-> Protocol
-> Provider
-> Transport
-> Upstream
```
Retry and fallback should be coordinated by AiProxyEngine instead of being
implemented inside the Transport layer.
## Spring AI Dependency
The refactored AI Proxy data path should no longer depend on Spring AI
clients or DTOs. Other AI plugins may continue
using Spring AI where its model abstraction is useful.
## Compatibility
The refactor should preserve:
- Existing AiProxyHandle configuration.
- OpenAI Chat Completions API behavior.
- Streaming and non-streaming requests.
- Proxy API key validation.
- Retry and fallback behavior.
- Existing Selector and Rule data.
Protocol payloads should retain raw JSON or JsonNode data instead of using
Spring AI DTOs as the internal model,
preventing unknown fields from being lost.
## Acceptance Criteria
- [ ] AiProxyPlugin no longer depends on OpenAI-specific DTOs, OpenAiApi,
or other Spring AI clients.
- [ ] Protocol, Provider, and Transport have independent interfaces and
registries.
- [ ] Adding a new Protocol does not require changing the Transport layer.
- [ ] Adding a new Provider does not require changing the plugin execution
flow.
- [ ] Streaming responses are forwarded without full buffering.
- [ ] Upstream response status, headers, and error bodies are preserved.
- [ ] Existing OpenAI-compatible behavior and tests continue to work.
- [ ] Other AI plugins may continue using Spring AI.
- [ ] Extension documentation and unit tests are added.
### Task List
## Task List
1. Define protocol-neutral request and response models.
2. Introduce Protocol, Provider, and Transport interfaces and registries.
3. Migrate OpenAI Chat Completions to the new architecture.
4. Implement OpenAI, DeepSeek, and OpenAI-compatible Providers.
5. Implement reactive HTTP and SSE Transports.
6. Move retry and fallback logic into the orchestration layer.
7. Remove Spring AI clients and DTOs from the AI Proxy data path.
8. Add compatibility tests and extension documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
