davidpoltorak-io opened a new issue, #1627: URL: https://github.com/apache/apisix-ingress-controller/issues/1627
### Issue description TL;DR When deleting Consumer manifests - and - if using an authentication scheme such as `KeyAuth` which references a Kubernetes secret - if the secret has already been deleted within the environment, the consumer cannot be removed [and retries infinitely] More Context: We use argocd and helm charts to deploy our services. We manage ingress access in these charts by creating secrets and consumers. The secrets created are the API tokens that consumers refer to for KeyAuth. The secrets are managed by external-secrets and thus pre-exist in a secret manager. Deploying consumers is no issue - however, when we delete everything in a particular release - not all consumers are deleted [non-deterministic] and those that cannot be deleted enter into an infinite loop of retries. They are still contained with etcd and therefore available on the APISIX api. From a quick glance at the code for 1.5 - I believe this is happening due to the fact that in the environment, argocd can delete secrets before the ingress-controller has processed the consumer deletion event [race condition] The ingress-controller tries to `translate` the consumer to be removed and fails to parse / translate due to the secret being missing. The only way to remove the consumer entry is to manually delete it or restore the secret. Useful links to functions https://github.com/apache/apisix-ingress-controller/blob/32be235aaf1f571a9f70f6375459429cd4ecc1af/pkg/kube/translation/apisix_consumer.go#L68 https://github.com/apache/apisix-ingress-controller/blob/32be235aaf1f571a9f70f6375459429cd4ecc1af/pkg/kube/translation/plugin.go#L125 ### Environment - your apisix-ingress-controller version (output of apisix-ingress-controller version --long): 1.5.0 - your Kubernetes cluster version (output of kubectl version): v1.23.13-eks - if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a): ### Minimal test code / Steps to reproduce 1. Create Kubernetes Secret with property `key=token` 2. Create Consumer with keyAuth reference to Kubernetes Secret 3. Ensure syncrhonised with APISIX [check admin API or etcd] 4. Delete Secret 5. Delete Consumer ### Actual result Consumer is not deleted ### Error log error ingress/apisix_consumer.go:166 failed to translate ApisixConsumer {"error": "invalid key auth config: secret \"dev-api-key-gn78j-15\" not found"... [Apisix Consumer Object JSON is printed to screen - removed for brevity] ingress/status.go:220 failed to record status change for ApisixConsumer {"error": "Operation cannot be fulfilled on apisixconsumers.apisix.apache.org \"t-gn78j-15\": StorageError: invalid object, Code: 4, Key: /registry/apisix.apache.org/apisixconsumers/t-gn78j/t-gn78j-15, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: e8fb79f7-e8e2-4e7c-9746-b17c8670f683, UID in object meta: ", "name": "t-gn78j-15", "namespace": "t-gn78j"} ### Expected result Consumer is deleted -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
