bzp2010 opened a new issue, #2076:
URL: https://github.com/apache/apisix-ingress-controller/issues/2076
### Current Behavior
When using composite mode and modifying CRD resources, Ingress Controller
will output an error, the modification will not take effect, and APISIX DP will
not receive the configuration change push.
Errors are like:
```
error adapter/etcd.go:166 object not found (during update event),
ignore it{revision 11 22 <nil>} {key 15 0 /apisix/ssls/e2134ade <nil>}
```
### Expected Behavior
Data can be saved into the memory cache normally, and APISIX DP can receive
configuration change event pushes.
### Error Logs
```
error adapter/etcd.go:166 object not found (during update event),
ignore it{revision 11 22 <nil>} {key 15 0 /apisix/ssls/e2134ade <nil>}
```
### Steps to Reproduce
The reproduction steps are exemplified by the ApisixTls resource. But this
could be the case with any other resource.
1. Create a secret object named `test-cert` containing a TLS certificate
(you can use cert-manager to issue self-signed certificates for testing)
2. Create an ApisixTls that references the `test-cert` you just created
**(but append some random characters to it, with the goal of making it
impossible for the ingress controller to find the resource)**
3. View the ingress controller log and wait for the synchronization to
complete. You will see:
```
error apisix/apisix_tls.go:165 failed to translate ApisixTls
{"error": "secret \"example-com-tls-d\" not found", "ApisixTls":
{"metadata":{"name":"sample-tls-test","namespace":"ingress-apisix-composite","uid":"cd39cf52-6d40-4b28-a633-643ef6fff98b","resourceVersion":"14723127","generation":2,"creationTimestamp":"2023-11-30T08:09:45Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apisix.apache.org/v2\",\"kind\":\"ApisixTls\",\"metadata\":{\"annotations\":{},\"name\":\"sample-tls-test\",\"namespace\":\"ingress-apisix-composite\"},\"spec\":{\"hosts\":[\"example.org\"],\"secret\":{\"name\":\"example-com-tls-d\",\"namespace\":\"ingress-apisix\"}}}\n"},"managedFields":[{"manager":"apisix-ingress-controller","operation":"Update","apiVersion":"apisix.apache.org/v2","time":"2023-11-30T08:09:45Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:conditions":{}}},"subresource":"status"},{"manager":"kubectl-client-side-apply","operation":"Update",
"apiVersion":"apisix.apache.org/v2","time":"2023-11-30T08:10:26Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:hosts":{},"f:secret":{".":{},"f:name":{},"f:namespace":{}}}}}]},"spec":{"hosts":["example.org"],"secret":{"name":"example-com-tls-d","namespace":"ingress-apisix"}},"status":{"conditions":[{"type":"ResourcesAvailable","status":"True","observedGeneration":1,"lastTransitionTime":"2023-11-30T08:09:45Z","reason":"ResourcesSynced","message":"Sync
Successfully"}]}}}
info apisix/apisix_tls.go:270 sync ApisixTls but not found, ignore
{"event_type": "update", "ApisixTls":
"ingress-apisix-composite/sample-tls-test", "version": "apisix.apache.org/v2"}
```
4. Change the secret name in the ApisixTls resource to the correct value
(test-cert). You will see:
```
error adapter/etcd.go:166 object not found (during update event),
ignore it{revision 11 22 <nil>} {key 15 0 /apisix/ssls/538378bb <nil>}
```
5. Delete ApisixTls CRD. You will see:
```
error adapter/etcd.go:211 object not found (during delete event),
ignore it{revision 11 22 <nil>} {key 15 0 /apisix/ssls/538378bb <nil>}
```
6. The error has been reproduced.
### Description
Let me describe why the error occurred:
- Ingress internally now uses etcd-adapter and its btree cache backend to
implement an etcd mock server.
- In the btree cache implementation, when a caller tries to update or delete
a value, it first performs a fetch operation to confirm that the key actually
exists in the cache.
- When we create a resource, if the resource itself does not pass the
checksum, the call does not actually reach the write function of the btree
cache, so the data is not written to the store.
- When we try to correct the CRD resource, the K8s SDK informer will trigger
the update event, and internally Ingress also sends an update event to the etcd
adapter, and this operation will eventually be converted to an Update operation
on the btree cache backend, which triggers a bug in the read-modify-write
chain. Since no data can be read, the modification behavior will be interrupted
and the configuration changes will not be pushed to the APISIX DP through the
watch channel.
- The same behavior occurs when deleting.
The result of this is that if a user first creates a CRD resource containing
an error and wants to fix it, it cannot be done. It must first be deleted and
then a new one created (Informer will tell Ingress that a new CRD was created,
not updated).
### Environment
- APISIX Ingress controller version (run `apisix-ingress-controller version
--long`) 1.7.0
- Kubernetes cluster version (run `kubectl version`) 1.28.2
- OS version if running APISIX Ingress controller in a bare-metal
environment (run `uname -a`) Linux masked-hostname 6.1.0-12-amd64 # 1 SMP
PREEMPT_DYNAMIC Debian 6.1.52-1 (2023-09-07) x86_64 GNU/Linux
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
