Vacant2333 commented on issue #10500: URL: https://github.com/apache/apisix/issues/10500#issuecomment-1898288646
I think we have found the reason and how to reproduce this issue: # Reason: In the code shown in the following figure, when `Etcd` undergoes a compaction action, it will result in the `need_ reload` is marked as `true`. https://github.com/Vacant2333/apisix/blob/a18573a3b8775f5a92ba92b948e8212affbf7057/apisix/core/config_etcd.lua#L394 <img width="694" alt="image" src="https://github.com/apache/apisix/assets/19872346/bdf9426f-e0d3-46df-b99c-87a9362b83bd";> Then it will flush all cache related to this piece of content (including healthcheck of the ***nodes***). https://github.com/Vacant2333/apisix/blob/a18573a3b8775f5a92ba92b948e8212affbf7057/apisix/core/config_etcd.lua#L362 <img width="554" alt="image" src="https://github.com/apache/apisix/assets/19872346/551a37a6-a57a-4821-899a-3ff849aafbb2";> Q: Why it will happen(etcd compaction)? A: Perhaps the user has set `auto-compaction` on `ETCD` or run `etcdctl compaction X`. Q: Why `APISIX` needs reload when compaction happend? A: Let me give an example: 1. Start `APISIX`: The revisions for `APISIX` and `Etcd` are both 60, which means they are in a synchronized state. 2. Added a `Route` through `APISIX`: At this time, the revision for `APISIX` is still 60, but the latest revision for `Etcd` is actually 61. 3. Before `APISIX` sync the new revision, manually/automatically execute `etcdctl compaction 61`: At this point, `Etcd` cannot synchronize the content of revision 61 to `APISIX` because revisions 1-61 have already been Compact (i.e. snapshot in `Raft`), and `APISIX` can only choose full synchronization (which is usually done incrementally by adding revision). Therefore, full synchronization must be done. # Reporduce steps: 1. Start `APISIX` and `Etcd`: `APISIX` 3.0 and below are easier to reproduce. 2. Create a `Route`: ```shell curl "http://127.0.0.1:9180/apisix/admin/routes/6"; -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d ' { "methods": ["GET"], "host": "example.com", "uri": "/anything/*", "upstream": { "type": "roundrobin", "nodes": { "httpbin.org:80": 1 } } }' ``` 3. Get the newly revision and run compaction: ``` # Get the newly revision id etcdctl endpoint status --write-out=json | jq .[0].Status.header.revision # Run compaction etcdctl compaction {Latest-revision} ``` 4. Wait 2 minutes:  # How to resolve this problem: 1. Update `APISIX` to version 3.2 LTS at least. 2. Avoid clearing the latest revisions (such as the latest 1000). For now, i think the issue could close now, cc @shreemaan-abhishek -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
