janiussyafiq opened a new issue, #13351:
URL: https://github.com/apache/apisix/issues/13351
### Current Behavior
When a Route has `ai-proxy` (or `ai-proxy-multi`) plus any other plugin, a
`PATCH` against the Route — even one that doesn't touch `ai-proxy` — corrupts
`ai-proxy.auth.header` and subsequent requests to that Route return `401
Incorrect API key provided` from the upstream LLM.
The Admin GET on the Route still returns the correct plaintext
`auth.header.Authorization` because `apisix/admin/init.lua` decrypts responses
(`utils.decrypt_params(plugin.decrypt_conf, data)` for
non-consumer/non-metadata resources), so the breakage is invisible from the
Admin API surface but very real on the data plane.
**Root cause.** The Admin PATCH path (`apisix/admin/resource.lua:_M:patch`)
reads the encrypted node from etcd, merges/patches the request payload into it,
then unconditionally calls `self:check_conf` → `encrypt_conf` on the merged
value. `encrypt_conf` (`apisix/plugin.lua:1092`) walks each plugin's
`encrypt_fields` and re-encrypts with no idempotency / decrypt-first guard. So
fields that were already ciphertext get wrapped in a second AES-CBC layer.
Because AES-CBC in APISIX uses a *fixed IV* (`apisix/ssl.lua:95`),
encryption is deterministic — a same-plaintext stability check would normally
look stable, but the *length* of `auth.header.Authorization` grows on every
PATCH (≈256 → ≈380 → … base64 chars in my MRE), which is direct proof of nested
encryption.
At request time the data plane decrypts only once via `plugin_checker` →
`decrypt_conf`, so the still-ciphertext bearer token is sent verbatim to OpenAI
/ Anthropic / etc. and rejected.
**Affected versions.**
- Latent since **3.1.0** — `encrypt_fields` infrastructure landed in PR
#8487 (commit `152ea80e`).
- Reachable by default from **3.10.0** — `enable_encrypt_fields: true`
flipped to default in PR #11076 (commit `cca94f10`).
- Any plugin declaring `encrypt_fields` is in principle vulnerable on PATCH;
`ai-proxy` / `ai-proxy-multi` are the user-facing failure mode because the
leaked secret is the upstream API key in `auth.header` / `auth.query` /
`auth.aws.*` / `auth.gcp.service_account_json`.
### Expected Behavior
PATCHing a sibling plugin on a Route must not mutate `ai-proxy.auth.header`.
Equivalently: every Admin write path must invoke `encrypt_conf` on plaintext
only. The fix is to decrypt the fields listed in each plugin's `encrypt_fields`
(recursing into nested schemas like `ai-proxy-multi.instances[*].auth.*`) after
reading the node from etcd and before merge/patch.
### Error Logs
Upstream rejection (OpenAI shown; same shape for any provider whose key is
in `auth.header`):
```json
{
"error": {
"message": "Incorrect API key provided: 5UhVzbqv***. You can find your
API key at https://platform.openai.com/account/api-keys.";,
"type": "invalid_request_error",
"param": null,
"code": "invalid_api_key"
}
}
```
The `5UhVzbqv***` prefix is the base64 of the *first* encryption layer being
sent to OpenAI as the bearer token — i.e. the data plane only peeled off one
layer.
### Steps to Reproduce
Prereqs: APISIX ≥ 3.10.0 with the default `enable_encrypt_fields: true`,
Admin API on `127.0.0.1:9180` with the default key
`edd1c9f034335f136f87ad84b625c8f1`, and `OPENAI_API_KEY` exported.
1. Create a Route with `ai-proxy` + a second plugin (here
`ai-prompt-decorator`):
```bash
curl -i http://127.0.0.1:9180/apisix/admin/routes/1 \
-H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d '{
"uri": "/v1/chat/completions",
"plugins": {
"ai-proxy": {
"provider": "openai",
"auth": { "header": { "Authorization": "Bearer
'"$OPENAI_API_KEY"'" } },
"options": { "model": "gpt-4o-mini" }
},
"ai-prompt-decorator": {
"prepend": [{ "role": "system", "content": "You are helpful." }]
}
}
}'
```
2. Confirm the route works:
```bash
curl -sS http://127.0.0.1:9080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}'
```
→ 200 with a normal completion.
3. PATCH only the *non-*`ai-proxy` plugin:
```bash
curl -i
http://127.0.0.1:9180/apisix/admin/routes/1/plugins/ai-prompt-decorator \
-H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PATCH -d '{
"prepend": [{ "role": "system", "content": "You are concise." }]
}'
```
4. Repeat the chat call from step 2 → `401 Incorrect API key provided:
<ciphertext>***`.
5. (Optional) Compare the *length* of
`plugins["ai-proxy"].auth.header.Authorization` in the raw PATCH response body
before vs. after step 3 — it grows, confirming the nested-encryption layer.
Admin GET hides this by decrypting, so use the raw PATCH response or `etcdctl
get` on the underlying key.
### Environment
- APISIX version (`apisix version`): **3.16.0**
- Operating system (`uname -a`): **Linux 7.0.5 (aarch64, Ubuntu in
OrbStack)**
- OpenResty / Nginx version (`openresty -V`): **openresty/1.27.1.2**
(OpenSSL 3.4.1)
- etcd version: **3.5.11** (`etcdcluster: 3.5.0`)
- APISIX Dashboard version: n/a
- Plugin runner version: n/a
- LuaRocks version: n/a
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
