circleyu opened a new issue, #2849:
URL: https://github.com/apache/apisix-ingress-controller/issues/2849
## Description
Deleting an `ApisixConsumer` is processed correctly by the provider (the
consumer is removed from APISIX), but `ApisixConsumerReconciler` still returns
the original `NotFound` error. controller-runtime then rate-limits and retries
the same request indefinitely, flooding the logs with:
```text
ERROR controllers.ApisixConsumer failed to get ApisixConsumer {"error":
"ApisixConsumer.apisix.apache.org \"<name>\" not found"}
ERROR controller-runtime Reconciler error
INFO provider.client syncing all resources
```
This is still present on `master`:
https://github.com/apache/apisix-ingress-controller/blob/master/internal/controller/apisixconsumer_controller.go
## Steps to reproduce
1. Create a namespaced `ApisixConsumer` that AIC watches (any namespace the
controller is configured for), for example key-auth with a `secretRef`.
2. Wait until it is accepted / synced (`service_number` increases, ingest
with the key succeeds).
3. Delete the CR:
```bash
kubectl -n <ns> delete apisixconsumer <name>
```
4. Watch AIC logs.
## Expected
After `Get` returns `NotFound` and `Provider.Delete` succeeds, reconcile
should return `ctrl.Result{}, nil` and stop. No further retries for that object.
This is already how `ApisixTlsReconciler` behaves:
```go
if err := r.Get(ctx, req.NamespacedName, &tls); err != nil {
if client.IgnoreNotFound(err) == nil {
// ...
if err := r.Provider.Delete(ctx, &tls); err != nil {
return ctrl.Result{}, err
}
r.Log.Info("deleted apisix tls", "tls", tls.Name)
return ctrl.Result{}, nil
}
return ctrl.Result{}, err
}
```
https://github.com/apache/apisix-ingress-controller/blob/master/internal/controller/apisixtls_controller.go
## Actual
`ApisixConsumerReconciler.Reconcile` (current `master`):
```go
ac := &apiv2.ApisixConsumer{}
if err := r.Get(ctx, req.NamespacedName, ac); err != nil {
if k8serrors.IsNotFound(err) {
ac.Namespace = req.Namespace
ac.Name = req.Name
ac.TypeMeta = metav1.TypeMeta{
Kind: KindApisixConsumer,
APIVersion: apiv2.GroupVersion.String(),
}
if err := r.Provider.Delete(ctx, ac); err != nil {
r.Log.Error(err, "failed to delete provider", "ApisixConsumer",
utils.NamespacedName(ac))
return ctrl.Result{}, err
}
// missing: return ctrl.Result{}, nil
}
r.Log.Error(err, "failed to get ApisixConsumer", "request",
req.NamespacedName)
return ctrl.Result{}, err
}
```
When `Get` is `NotFound` and `Provider.Delete` succeeds, execution falls
through and still returns the `NotFound` error. controller-runtime treats that
as a failed reconcile and retries with exponential backoff (observed ~20s,
~41s, then continuing until max delay).
The provider path itself appears fine: each retry is followed by `syncing
all resources`, so the delete is applied (and then re-applied). The object is
gone from the API server.
## Sample logs
```text
INFO controllers.ApisixConsumer reconcile
{"request":
{"name":"cust-8fa274728c24","namespace":"signalgate-consumers"}}
ERROR controllers.ApisixConsumer failed to get ApisixConsumer
{"request":
{"name":"cust-8fa274728c24","namespace":"signalgate-consumers"},
"error": "ApisixConsumer.apisix.apache.org \"cust-8fa274728c24\" not
found"}
ERROR controller-runtime Reconciler error
{"controller": "apisixconsumer", "controllerKind": "ApisixConsumer",
"namespace": "signalgate-consumers", "name": "cust-8fa274728c24",
"error": "ApisixConsumer.apisix.apache.org \"cust-8fa274728c24\" not
found"}
INFO provider.client syncing all resources
INFO provider.client syncing resources for config
{"service_number": 27}
```
Line numbers in a 2.x build match
`controller/apisixconsumer_controller.go:61` (reconcile) and `:77` (`failed to
get ApisixConsumer`).
Restarting the AIC pod clears the workqueue and stops the loop until the
next delete.
## Suggested fix
Return success after a successful provider delete on `NotFound`, and only
log/return `Get` errors that are not `NotFound`:
```go
if err := r.Get(ctx, req.NamespacedName, ac); err != nil {
if k8serrors.IsNotFound(err) {
ac.Namespace = req.Namespace
ac.Name = req.Name
ac.TypeMeta = metav1.TypeMeta{
Kind: KindApisixConsumer,
APIVersion: apiv2.GroupVersion.String(),
}
if err := r.Provider.Delete(ctx, ac); err != nil {
r.Log.Error(err, "failed to delete provider", "ApisixConsumer",
utils.NamespacedName(ac))
return ctrl.Result{}, err
}
return ctrl.Result{}, nil
}
r.Log.Error(err, "failed to get ApisixConsumer", "request",
req.NamespacedName)
return ctrl.Result{}, err
}
```
Please also check other 2.x reconcilers for the same fall-through.
`ApisixTlsReconciler` already returns `nil` after delete; `ApisixConsumer` does
not.
## Environment
- APISIX Ingress Controller: 2.x (controller-runtime `ApisixConsumer`
reconciler; `controller/apisixconsumer_controller.go`)
- Trigger: `kubectl delete` of `ApisixConsumer` (or any client that deletes
the CR)
- Confirmed against current `master` source
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
