janiussyafiq opened a new issue, #13351:
URL: https://github.com/apache/apisix/issues/13351
### Current Behavior
When a Route has `ai-proxy` (or `ai-proxy-multi`) plus any other plugin, a
`PATCH` against the Route — even one that doesn't touch `ai-proxy` — corrupts
`ai-proxy.auth.header` and subsequent requests to that Route return `401
Incorrect API key provided` from the upstream LLM.
The Admin GET on the Route still returns the correct plaintext
`auth.header.Authorization` because `apisix/admin/init.lua` decrypts responses
(`utils.decrypt_params(plugin.decrypt_conf, data)` for
non-consumer/non-metadata resources), so the breakage is invisible from the
Admin API surface but very real on the data plane.
**Root cause.** The Admin PATCH path (`apisix/admin/resource.lua:_M:patch`)
reads the encrypted node from etcd, merges/patches the request payload into it,
then unconditionally calls `self:check_conf` → `encrypt_conf` on the merged
value. `encrypt_conf` (`apisix/plugin.lua:1092`) walks each plugin's
`encrypt_fields` and re-encrypts with no idempotency / decrypt-first guard. So
fields that were already ciphertext get wrapped in a second AES-CBC layer.
Because AES-CBC in APISIX uses a *fixed IV* (`apisix/ssl.lua:95`),
encryption is deterministic — a same-plaintext stability check would normally
look stable, but the *length* of `auth.header.Authorization` grows on every
PATCH (≈256 → ≈380 → … base64 chars in my MRE), which is direct proof of nested
encryption.
At request time the data plane decrypts only once via `plugin_checker` →
`decrypt_conf`, so the still-ciphertext bearer token is sent verbatim to OpenAI
/ Anthropic / etc. and rejected.
**Affected versions.**
- Latent since **3.1.0** — `encrypt_fields` infrastructure landed in PR
#8487 (commit `152ea80e`).
- Reachable by default from **3.10.0** — `enable_encrypt_fields: true`
flipped to default in PR #11076 (commit `cca94f10`).
- Any plugin declaring `encrypt_fields` is in principle vulnerable on PATCH;
`ai-proxy` / `ai-proxy-multi` are the user-facing failure mode because the
leaked secret is the upstream API key in `auth.header` / `auth.query` /
`auth.aws.*` / `auth.gcp.service_account_json`.
### Expected Behavior
PATCHing a sibling plugin on a Route must not mutate `ai-proxy.auth.header`.
Equivalently: every Admin write path must invoke `encrypt_conf` on plaintext
only. The fix is to decrypt the fields listed in each plugin's `encrypt_fields`
(recursing into nested schemas like `ai-proxy-multi.instances[*].auth.*`) after
reading the node from etcd and before merge/patch.
### Error Logs
Upstream rejection (OpenAI shown; same shape for any provider whose key is
in `auth.header`):
```json
{
"error": {
"message": "Incorrect API key provided: 5UhVzbqv***. You can find your
API key at https://platform.openai.com/account/api-keys.";,
"type": "invalid_request_error",
"param": null,
"code": "invalid_api_key"
}
}
```
The `5UhVzbqv***` prefix is the base64 of the *first* encryption layer being
sent to OpenAI as the bearer token — i.e. the data plane only peeled off one
layer.
### Steps to Reproduce
Prereqs: APISIX ≥ 3.10.0 with the default `enable_encrypt_fields: true`,
Admin API on `127.0.0.1:9180` with the default key
`edd1c9f034335f136f87ad84b625c8f1`, and `OPENAI_API_KEY` exported.
1. Create a Route with `ai-proxy` + a second plugin (here
`ai-prompt-decorator`):
```bash
curl -i http://127.0.0.1:9180/apisix/admin/routes/1 \
-H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d '{
"uri": "/v1/chat/completions",
"plugins": {
"ai-proxy": {
"provider": "openai",
"auth": { "header": { "Authorization": "Bearer
'"$OPENAI_API_KEY"'" } },
"options": { "model": "gpt-4o-mini" }
},
"ai-prompt-decorator": {
"prepend": [{ "role": "system", "content": "You are helpful." }]
}
}
}'
```
2. Confirm the route works:
```bash
curl -sS http://127.0.0.1:9080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}'
```
→ 200 with a normal completion.
3. PATCH only the *non-*`ai-proxy` plugin:
```bash
curl -i
http://127.0.0.1:9180/apisix/admin/routes/1/plugins/ai-prompt-decorator \
-H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PATCH -d '{
"prepend": [{ "role": "system", "content": "You are concise." }]
}'
```
4. Repeat the chat call from step 2 → `401 Incorrect API key provided:
***`.
5. (Optional) Compare the *length* of
`plugins["ai-proxy"].auth.header.Authorization` in the raw PATCH response body
