aron-intframe commented on issue #2908: URL: https://github.com/apache/apisix-ingress-controller/issues/2908#issuecomment-6487178934
The delete set isn't a stored ownership record. ADC rebuilds it from APISIX on every sync, and the way it reads global rules throws the real IDs away. `listGlobalRules()` in ADC's APISIX backend flattens every live global rule into one map keyed by plugin name ([fetcher.ts#L135-L147](https://github.com/api7/adc/blob/1da3c4c342c75a0ccdb142f53f3e35cc78e7597c/libs/backend-apisix/src/fetcher.ts#L135-L147)): ```ts // [{ 'key-auth': {}, 'basic-auth': {} }, { 'real-ip': {} }] => // { 'key-auth': {}, 'basic-auth': {}, 'real-ip': {} } Object.fromEntries(list.flatMap((item) => Object.entries(item.value?.plugins ?? []))) ``` The differ then uses each plugin name as the resource ID (`generateId: (key) => key` for `GLOBAL_RULE` in `libs/sdk/src/core/differ.ts`). Your desired state has `globalRules: 0`, so ADC sends `DELETE /apisix/admin/global_rules/<plugin-name>` for every plugin found in any global rule. Your two live rules (the CR-named one with three plugins and the legacy numeric one) hold those plugins under other IDs, so every delete is a 404. That's why it survives restarts and you can't find it in etcd or a ConfigMap: it is re-derived from the live rules each cycle. The four IDs should be exactly the union of the `plugins` keys across `GET /apisix/admin/global_rules`. Reproduced with the ADC image the chart runs (`ghcr.io/api7/adc:0.27.1`) against APISIX 3.18.0, plain `adc sync`, no controller involved: ``` ## 1. two global rules created out of band, ids are NOT plugin names id=1 plugins=['limit-req'] id=ingress-apisix_global-plugins plugins=['limit-conn', 'prometheus', 'request-id'] ## 2. what ADC reads back (adc dump): original ids are gone, keys are plugin names global_rules: limit-conn: limit-req: prometheus: {} request-id: {} ## 3.1 sync a config with 1 service and 0 global_rules (cycle 1) [ADC] › ✖ fatal Error: DELETE http://adcr-apisix:9180/apisix/admin/global_rules/request-id, responded with status 404 Not Found, response body: {"message":"Key not found"} ## 3.2 sync a config with 1 service and 0 global_rules (cycle 2) [ADC] › ✖ fatal Error: DELETE http://adcr-apisix:9180/apisix/admin/global_rules/request-id, responded with status 404 Not Found, response body: {"message":"Key not found"} ``` The same flattening is still on ADC `main` as of today, so bumping the sidecar image alone won't change it. **Unblocking it.** The loop stops once APISIX no longer holds global rules whose ID differs from the plugin name. Same repro, continued: ``` ## 4. fix: put the plugins in desired state, delete the old rules by their REAL ids, sync DELETE /global_rules/1 -> 200 DELETE /global_rules/ingress-apisix_global-plugins -> 200 [APISIX] › ✔ success Create global_rule: "prometheus" [APISIX] › ✔ success Create global_rule: "limit-req" [APISIX] › ✔ success Create global_rule: "limit-conn" [APISIX] › ✔ success Create global_rule: "request-id" [ADC] › ★ star All is well, see you next time! id=limit-conn plugins=['limit-conn'] id=limit-req plugins=['limit-req'] id=prometheus plugins=['prometheus'] id=request-id plugins=['request-id'] ## 5. steady state, then removing a plugin from desired state now deletes cleanly [ADC] › ★ star All is well, see you next time! ``` In your cluster that means: 1. Get the `ApisixGlobalRule` CRs into desired state first (next section). If you skip this, step 2 removes those plugins from traffic. 2. Right after, delete the two legacy rules by their real IDs: `curl -X DELETE -H "X-API-KEY: $KEY" http://<admin>:9180/apisix/admin/global_rules/<id>`, using the IDs from `GET /apisix/admin/global_rules`. Don't leave both in place for long, or the same plugin sits in two global rules. 3. The next sync creates one rule per plugin with ID = plugin name. From then on deletes resolve and the batch commits. If you don't need those plugins at all, step 2 alone is enough. **Question 3 (`globalRules: 0` with two CRs).** An empty `status` on both CRs means the reconciler never reached the point where it writes a condition. In v2.2.0 (`internal/controller/apisixglobalrule_controller.go`, `FindMatchingIngressClass` in `utils.go`) a CR is only translated when `spec.ingressClassName` names an IngressClass whose `spec.controller` is this controller, or is empty and that IngressClass is marked default. Anything else is skipped with only a V(1) log line, `no matching IngressClass available`. Compare: ``` kubectl get apisixglobalrule -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.ingressClassName kubectl get ingressclass -o custom-columns=NAME:.metadata.name,CONTROLLER:.spec.controller,PARAMS:.spec.parameters.name ``` Once they match and the IngressClass points at your GatewayProxy, the "prepared request body" log should show `globalRules` > 0. I didn't run the controller itself, so this part is from the v2.2.0 source, not a repro. On question 2: a 404-tolerant DELETE would hide this, but the actual defect is that the dump drops the real global rule IDs. That is worth a separate issue on api7/adc. The worst part here, a sync that fails every 60 s for weeks with only log lines to show for it, is a "last success went stale" problem. The watcher we use for that is in our repo (ours): https://github.com/aron-intframe/agent-guards/blob/main/guards/heartbeat-watch.py -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
