jens-skribble opened a new issue, #2821:
URL: https://github.com/apache/apisix-ingress-controller/issues/2821
### Current Behavior
We run APISIX in **API-driven standalone** mode driven by the ingress
controller
(ADC, `provider.type: apisix-standalone`). We wanted a route's plugin to
reference a
Vault secret via APISIX's native `$secret://vault/...`. There is no CRD or
`GatewayProxy` field we could find to declare the backing `secrets`
secret-manager, so
the only way to get one onto the data plane is to register it directly on
APISIX's
standalone Admin API (out of band).
Doing that **deadlocks the controller**. Once the out-of-band manager bumps
APISIX's
`secrets_conf_version` from `0` to `1`, every subsequent controller/ADC
full-config
push is rejected:
```
PUT /apisix/admin/configs → 400 Bad Request:
secrets_conf_version must be greater than or equal to (1)
```
ADC has no `secrets` resource type, so its push carries
`secrets_conf_version: 0`
while APISIX is now at `1`; APISIX's monotonic version check rejects the
**whole**
push. Because each standalone update replaces the full configuration
**atomically**,
nothing reconciles after that: the out-of-band manager is **not** wiped, the
controller
is simply **stuck**, and even unrelated routes stop applying (they 404 at
the gateway).
This looks like a robustness bug independent of the secret use case: once
`secrets_conf_version` is `> 0` by **any** means (an out-of-band manager, a
leftover
from a prior setup), ADC - which always sends `0` - can never sync again,
and it takes
all other resources down with it.
For reference, APISIX itself handles `$secret://` fine in this mode - we
confirmed it
end-to-end with two standalone docker/podman baselines (in the attached
bundle), so the
problem is on the controller/ADC side, not APISIX:
- `compose/api-driven/` - in API-driven standalone, APISIX resolves
`$secret://vault/...`
when a `secrets` manager is present in the `/configs` payload.
- `compose/file-driven/` - the documented `apisix.yaml` `secrets:` block
resolves, but
that mode disables the Admin API the controller drives (so it can't be
used together
with the controller).
### Expected Behavior
- The controller/ADC should not deadlock on a resource type it doesn't
manage: it
should send a valid (or otherwise tolerated) `secrets_conf_version` so a
stray or
out-of-band `secrets` resource can't block reconciliation of everything
else.
- Ideally there would be a supported way to declare an APISIX `secrets`
secret-manager
through the controller (e.g. a `secrets` field on `GatewayProxy`, or a
dedicated
CRD), so `$secret://vault/...` references can be used. Today we can't find
one.
(`secretRef` reads a Kubernetes Secret and injects plaintext - a different
mechanism,
not APISIX's native `$secret://` manager.)
### Error Logs
Controller `manager` container - this block repeats on **every** sync
attempt after the
out-of-band `secrets` resource exists:
```
2026-07-22T09:14:48.222Z INFO provider.client client/client.go:214
syncing all resources
2026-07-22T09:14:48.239Z ERROR provider.executor
client/executor.go:420 ADC Server sync failed {"result":
{"status":"all_failed","total_resources":0,"success_count":0,"failed_count":1,"success":[],"failed":[],"endpoint_status":[{"server":"http://10.244.0.3:9180","success":false,"reason":"PUT
http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad
Request, error_msg: secrets_conf_version must be greater than or equal to
(1)"}]}, "error": "ADC Server sync failed (standalone mode):
http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs,
responded with status 400 Bad Request, error_msg: secrets_conf_version must be
greater than or equal to (1)"}
2026-07-22T09:14:48.240Z ERROR provider.executor
client/executor.go:153 failed to run http sync for server {"server":
"http://10.244.0.3:9180";, "error": "ServerAddr: http://10.244.0.3:9180, Err:
http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs,
responded with status 400 Bad Request, error_msg: secrets_conf_version must be
greater than or equal to (1)"}
2026-07-22T09:14:48.240Z ERROR provider.client client/client.go:306
failed to execute adc command {"config":
{"name":"GatewayProxy/ingress-apisix/apisix-config","serverAddrs":["http://10.244.0.3:9180"],"tlsVerify":false},
"error": "ADC execution error for GatewayProxy/ingress-apisix/apisix-config:
[ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT
http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad
Request, error_msg: secrets_conf_version must be greater than or equal to (1)]"}
2026-07-22T09:14:48.240Z ERROR provider.client client/client.go:247
failed to sync resources {"name":
"GatewayProxy/ingress-apisix/apisix-config", "error": "ADC execution errors:
[ADC execution error for GatewayProxy/ingress-apisix/apisix-config:
[ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT
http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad
Request, error_msg: secrets_conf_version must be greater than or equal to
(1)]]"}
2026-07-22T09:14:48.240Z ERROR provider apisix/provider.go:282
failed to sync {"error": "failed to sync 1 configs:
GatewayProxy/ingress-apisix/apisix-config"}
2026-07-22T09:14:56.241Z INFO provider.client client/client.go:214
syncing all resources
2026-07-22T09:14:56.254Z ERROR provider.executor
client/executor.go:420 ADC Server sync failed {"result":
{"status":"all_failed","total_resources":0,"success_count":0,"failed_count":1,"success":[],"failed":[],"endpoint_status":[{"server":"http://10.244.0.3:9180","success":false,"reason":"PUT
http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad
Request, error_msg: secrets_conf_version must be greater than or equal to
(1)"}]}, "error": "ADC Server sync failed (standalone mode):
http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs,
responded with status 400 Bad Request, error_msg: secrets_conf_version must be
greater than or equal to (1)"}
2026-07-22T09:14:56.254Z ERROR provider.executor
client/executor.go:153 failed to run http sync for server {"server":
"http://10.244.0.3:9180";, "error": "ServerAddr: http://10.244.0.3:9180, Err:
http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs,
responded with status 400 Bad Request, error_msg: secrets_conf_version must be
greater than or equal to (1)"}
2026-07-22T09:14:56.254Z ERROR provider.client client/client.go:306
failed to execute adc command {"config":
{"name":"GatewayProxy/ingress-apisix/apisix-config","serverAddrs":["http://10.244.0.3:9180"],"tlsVerify":false},
"error": "ADC execution error for GatewayProxy/ingress-apisix/apisix-config:
[ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT
http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad
Request, error_msg: secrets_conf_version must be greater than or equal to (1)]"}
2026-07-22T09:14:56.255Z ERROR provider.client client/client.go:247
failed to sync resources {"name":
"GatewayProxy/ingress-apisix/apisix-config", "error": "ADC execution errors:
[ADC execution error for GatewayProxy/ingress-apisix/apisix-config:
[ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT
http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad
Request, error_msg: secrets_conf_version must be greater than or equal to
(1)]]"}
2026-07-22T09:14:56.255Z ERROR provider apisix/provider.go:282
failed to sync {"error": "failed to sync 1 configs:
GatewayProxy/ingress-apisix/apisix-config"}
```
### Steps to Reproduce
A self-contained reproduction is attached as
[apisix-secrets-repro.zip](https://github.com/user-attachments/files/30263545/apisix-secrets-repro.zip):
- `k8s/` - a kustomization that stands everything up + a Job that automates
steps 3–6
below and prints the controller's own rejection log.
- `compose/api-driven/` - docker/podman baseline: APISIX resolves
`$secret://vault/...`
in API-driven standalone (shows the gap is controller-side, not APISIX).
- `compose/file-driven/` - docker/podman baseline: the documented
`apisix.yaml`
`secrets:` block resolves, but that mode has no Admin API for the
controller to drive.
To reproduce the deadlock manually against the controller:
1. Deploy APISIX in API-driven standalone mode (`deployment.role:
traditional`,
`role_traditional.config_provider: yaml`) + APISIX Ingress Controller
2.1.0 — e.g.
the `apisix` Helm chart 2.16.0 with `ingress-controller.enabled: true`,
`ingress-controller.config.provider.type: apisix-standalone`, and
`ingress-controller.gatewayProxy.createDefault: true`.
2. Apply any `ApisixRoute`; confirm the controller syncs it —
`GET /apisix/admin/configs` shows the route and `.secrets` is empty
(`secrets = 0`).
3. Register a `secrets` manager **out of band** via the standalone Admin API
(echo the
existing `*_conf_version` values back, omit `secrets_conf_version`, add
the manager):
```
PUT /apisix/admin/configs
{ ...existing *_conf_version..., "secrets": [
{ "id": "vault/oidc", "uri": "http://vault.example:8200";, "prefix":
"kv/apisix", "token": "root" } ] }
```
Confirm `.secrets` now has 1 entry (`secrets = 1`); APISIX's
`secrets_conf_version`
is now `1`.
4. Trigger a controller re-sync (e.g. `kubectl annotate apisixroute <name>
poke=$(date +%s) --overwrite`).
5. Watch the controller logs: every sync now fails with
`400 ... secrets_conf_version must be greater than or equal to (1)`.
6. Observe the blast radius: the out-of-band manager is **not** removed, and
because the
push is atomic, no route reconciles — `curl` against the gateway returns
`404` even
for routes unrelated to secrets.
### Environment
APISIX Ingress controller version (`apisix-ingress-controller version
--long`): **2.1.0**
```Version: 2.1.0
Git SHA:
Go Version: go1.24.13
Building OS/Arch: /
Running OS/Arch: linux/amd64
```
Kubernetes cluster version (`kubectl version`):
```
Client Version: v1.33.13
Kustomize Version: v5.6.0
Server Version: v1.35.1
```
(reproduced on minikube, Podman driver)
APISIX: **3.17.0**; installed via the `apisix` Helm chart **2.16.0**;
`provider.type: apisix-standalone`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
