vinson-hkgi opened a new issue, #2908:
URL: https://github.com/apache/apisix-ingress-controller/issues/2908
## Summary
In GatewayProxy/ADC mode, the controller issues `DELETE` for global-rule IDs
it no longer manages. When those IDs do not exist, the Admin API returns `404
Key not found`, and ADC reports the **whole batch** as failed — so no new
routes or TLS objects are ever committed. The desired config in the same
payload contains **zero** global rules, so the deletes cannot be coming from
desired state.
## Environment
- `apisix-ingress-controller` **2.2.0** (also reproduced on 2.0.1)
- chart `apisix` 2.17.0 → APISIX 3.18.0
- Provider: `GatewayProxy` → ADC sidecar
(`ADC_SERVER_URL=unix:/sockets/adc.sock`), `provider.type: apisix`,
`sync_period: 1m`
## Symptom
Every sync cycle (60 s) produces:
```
ERROR provider.executor ADC Server sync failed
{"result":{"status":"all_failed","total_resources":4,"success_count":0,"failed_count":4,
"failed":[{"event":{"resourceType":"global_rule","type":"delete","resourceId":"limit-conn"},
...
"reason":"DELETE http://<admin>/apisix/admin/global_rules/limit-conn,
responded with status 404 Not Found, response body: {\"message\":\"Key not
found\"}"}]}}
```
The four IDs vary with configuration history but are consistently **plugin
names** from earlier configs, e.g. `prometheus`, `coraza-filter`, `limit-conn`,
`limit-req` — including one (`limit-req`) that is not present in any current CR.
## Evidence that these are not desired state
The controller logs the exact request body it POSTs to the ADC sidecar
(`client/executor.go:372 "prepared request body"`):
```json
{"backend":"apisix","cacheKey":"GatewayProxy/<ns>/<name>",
"config":{"consumerGroups":0,"consumers":0,"globalRules":0,
"pluginMetadata":0,"services":19,"ssls":17}}
```
- `globalRules: 0` — desired state carries no global rules at all.
- None of the four IDs appears anywhere in the payload.
So the delete set is ADC's own record of what it previously owned, not a
diff against desired state.
## What we ruled out
The record survives controller pod restarts (three restarts, same four IDs),
and it is visible in no place we can find:
- not in the APISIX etcd keyspace (`/apisix/global_rules/` contains only the
two live rules, under different IDs)
- not in a ConfigMap or Secret in the namespace
- not in `ApisixGlobalRule` status (empty) nor `GatewayProxy` status (`{}`)
- no finalizers, no annotations carrying it
APISIX meanwhile holds the same logical configuration under different IDs
(e.g. one rule named after the CR holding three plugins, and one legacy numeric
ID), so the delete targets never exist.
## Impact
Because a 404 on one delete aborts the batch, **new Ingress/ApisixRoute and
TLS changes never commit** while pre-existing routes keep working. Adding any
new ingress reproduces it.
## Relation to existing issues
#2778 requested exactly this behaviour change — "handle the 404 gracefully
(idempotent DELETE)" — and was closed with documentation stating the controller
is the single source of truth. That may be correct for *writes*, but it leaves
no way to retire a stale ownership record, and the resulting permanent sync
failure is silent apart from logs.
## Ask
1. Is the ADC ownership record persisted anywhere documented, and how is it
meant to be cleared when the underlying resources are recreated out of band?
2. Would you accept a change making `DELETE` of a non-existent resource
idempotent (treat 404 as success), so one stale entry cannot block the entire
batch?
3. Is there a supported way to make `ApisixGlobalRule` CRs feed the desired
`globalRules` in GatewayProxy mode? In this deployment `globalRules: 0` despite
two `ApisixGlobalRule` CRs existing, which we did not expect.
Happy to provide full logs or test a patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
