dovics opened a new issue, #2778:
URL: https://github.com/apache/apisix-ingress-controller/issues/2778
### Current Behavior
The apisix-ingress-controller continuously attempts to DELETE a global_rule
resource with ID `prometheus`, but this resource does not exist in APISIX. This
results in a 404 "Key not found" error on every sync cycle (approximately every
60 seconds), causing persistent error logs.
**This bug persists even after restarting the apisix-ingress-controller**,
indicating the stale reference is stored in persistent storage, not in-memory
state.
### Expected Behavior
The controller should either:
- Only manage resources created through ApisixGlobalRule CRD, ignoring
pre-existing resources in APISIX, OR
- Handle the 404 response gracefully when deleting a resource that no longer
exists (idempotent DELETE)
### Error Logs
```
2026-06-02T06:49:29.563Z INFO provider.client client/client.go:201
syncing resources for config {"service_number": 0}
2026-06-02T06:49:29.914Z ERROR provider.executor
client/executor.go:328 ADC Server sync failed {"result":
{"status":"all_failed","total_resources":1,"success_count":0,"failed_count":1,"success":[],"failed":[{"event":{"resourceType":"global_rule","type":"delete","resourceId":"prometheus","resourceName":"prometheus"},"failed_at":"2026-06-02T06:49:29Z","synced_at":"0001-01-01T00:00:00Z","reason":"DELETE
http://apisix-admin.infra.svc:9180/apisix/admin/global_rules/prometheus,
responded with status 404 Not Found, response body: {\"message\":\"Key not
found\"}","response":{"status":404,"headers":{"access-control-allow-credentials":"true","access-control-allow-origin":"*","access-control-expose-headers":"*","access-control-max-age":"3600","connection":"keep-alive","content-type":"application/json","date":"Tue,
02 Jun 2026 06:49:29
GMT","server":"APISIX/3.14.1","transfer-encoding":"chunked","x-api-version":"v3"}}}]},
"error": "ADC Server sync failed: DELETE http://apisix-
admin.infra.svc:9180/apisix/admin/global_rules/prometheus, responded with
status 404 Not Found, response body: {\"message\":\"Key not found\"}"}
2026-06-02T06:49:29.914Z ERROR provider.executor
client/executor.go:142 failed to run http sync for server {"server":
"http://apisix-admin.infra.svc:9180";, "error": "ServerAddr:
http://apisix-admin.infra.svc:9180, Err: DELETE
http://apisix-admin.infra.svc:9180/apisix/admin/global_rules/prometheus,
responded with status 404 Not Found, response body: {\"message\":\"Key not
found\"}"}
2026-06-02T06:49:29.914Z ERROR provider.client client/client.go:269
failed to execute adc command {"config":
{"name":"GatewayProxy/infra/apisix-ingress-controller-config","serverAddrs":["http://apisix-admin.infra.svc:9180"],"tlsVerify":false},
"error": "ADC execution error for
GatewayProxy/infra/apisix-ingress-controller-config: [ServerAddr:
http://apisix-admin.infra.svc:9180, Err: DELETE
http://apisix-admin.infra.svc:9180/apisix/admin/global_rules/prometheus,
responded with status 404 Not Found, response body: {\"message\":\"Key not
found\"}]"}
2026-06-02T06:49:29.914Z ERROR provider.client client/client.go:210
failed to sync resources {"name":
"GatewayProxy/infra/apisix-ingress-controller-config", "error": "ADC execution
errors: [ADC execution error for
GatewayProxy/infra/apisix-ingress-controller-config: [ServerAddr:
http://apisix-admin.infra.svc:9180, Err: DELETE
http://apisix-admin.infra.svc:9180/apisix/admin/global_rules/prometheus,
responded with status 404 Not Found, response body: {\"message\":\"Key not
found\"}]]"}
2026-06-02T06:49:29.914Z ERROR provider apisix/status.go:321
failed to get resource label {"configName":
"GatewayProxy/infra/apisix-ingress-controller-config", "resourceType":
"global_rule", "id": "prometheus", "error": "not found"}
2026-06-02T06:49:29.914Z ERROR provider
apisix/provider.go:282 failed to sync {"error": "failed to sync 1 configs:
GatewayProxy/infra/apisix-ingress-controller-config"}
2026-06-02T06:50:29.562Z INFO provider.client client/client.go:177
syncing all resources
2026-06-02T06:50:29.562Z INFO provider.client client/client.go:201
syncing resources for config {"service_number": 0}
2026-06-02T06:50:29.914Z ERROR provider.executor
client/executor.go:328 ADC Server sync failed {"result":
{"status":"all_failed","total_resources":1,"success_count":0,"failed_count":1,"success":[],"failed":[{"event":{"resourceType":"global_rule","type":"delete","resourceId":"prometheus","resourceName":"prometheus"},"failed_at":"2026-06-02T06:50:29Z","synced_at":"0001-01-01T00:00:00Z","reason":"DELETE
http://apisix-admin.infra.svc:9180/apisix/admin/global_rules/prometheus,
responded with status 404 Not Found, response body: {\"message\":\"Key not
found\"}","response":{"status":404,"headers":{"access-control-allow-credentials":"true","access-control-allow-origin":"*","access-control-expose-headers":"*","access-control-max-age":"3600","connection":"keep-alive","content-type":"application/json","date":"Tue,
02 Jun 2026 06:50:29
GMT","server":"APISIX/3.14.1","transfer-encoding":"chunked","x-api-version":"v3"}}}]},
"error": "ADC Server sync failed: DELETE http://apisix-
admin.infra.svc:9180/apisix/admin/global_rules/prometheus, responded with
status 404 Not Found, response body: {\"message\":\"Key not found\"}"}
2026-06-02T06:50:29.914Z ERROR provider.executor
client/executor.go:142 failed to run http sync for server {"server":
"http://apisix-admin.infra.svc:9180";, "error": "ServerAddr:
http://apisix-admin.infra.svc:9180, Err: DELETE
http://apisix-admin.infra.svc:9180/apisix/admin/global_rules/prometheus,
responded with status 404 Not Found, response body: {\"message\":\"Key not
found\"}"}
2026-06-02T06:50:29.914Z ERROR provider.client client/client.go:269
failed to execute adc command {"config":
{"name":"GatewayProxy/infra/apisix-ingress-controller-config","serverAddrs":["http://apisix-admin.infra.svc:9180"],"tlsVerify":false},
"error": "ADC execution error for
GatewayProxy/infra/apisix-ingress-controller-config: [ServerAddr:
http://apisix-admin.infra.svc:9180, Err: DELETE
http://apisix-admin.infra.svc:9180/apisix/admin/global_rules/prometheus,
responded with status 404 Not Found, response body: {\"message\":\"Key not
found\"}]"}
2026-06-02T06:50:29.914Z ERROR provider.client client/client.go:210
failed to sync resources {"name":
"GatewayProxy/infra/apisix-ingress-controller-config", "error": "ADC execution
errors: [ADC execution error for
GatewayProxy/infra/apisix-ingress-controller-config: [ServerAddr:
http://apisix-admin.infra.svc:9180, Err: DELETE
http://apisix-admin.infra.svc:9180/apisix/admin/global_rules/prometheus,
responded with status 404 Not Found, response body: {\"message\":\"Key not
found\"}]]"}
2026-06-02T06:50:29.914Z ERROR provider apisix/status.go:321
failed to get resource label {"configName":
"GatewayProxy/infra/apisix-ingress-controller-config", "resourceType":
"global_rule", "id": "prometheus", "error": "not found"}
2026-06-02T06:50:29.914Z ERROR provider
apisix/provider.go:282 failed to sync {"error": "failed to sync 1 configs:
GatewayProxy/infra/apisix-ingress-controller-config"}
```
```
curl 172.20.11.247:9180/apisix/admin/global_rules -H "X-API-KEY: xxxxx" | jq
.
{
"total": 1,
"list": [
{
"createdIndex": 2809,
"modifiedIndex": 2809,
"key": "/apisix/global_rules/mjVW1NUzLqszfw9kLma6w",
"value": {
"update_time": 1775723725,
"create_time": 1775723725,
"id": "mjVW1NUzLqszfw9kLma6w",
"plugins": {
"prometheus": {}
}
}
}
]
}
```
### Steps to Reproduce
1. Create a global_rule directly via APISIX Admin API **before** starting
the apisix-ingress-controller
2. Start the apisix-ingress-controller
3. Observe that the controller continuously attempts to DELETE a global_rule
named `prometheus` which does not exist
4. **Restart the controller** - observe the same error persists
### Environment
- **apisix-ingress-controller version**: 2.0.1
- **APISIX version**: 3.14.1
- **ADC version**: 0.26.0 (also reproducible in ADC 0.21.2)
- **kubenetes version**: 1.33.4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
