manueljishi commented on issue #2708: URL: https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-5265451571
We hit this and tracked it to a reproducible root cause. Posting in full since the issue was just marked stale, and because I think #2689 and #2699 are the same bug. **TL;DR:** it is not the controller and not APISIX. It is the **adc sidecar version boundary at v0.22.0**, combined with adc's *update* path being unable to migrate a service from the old inline-upstream format to the new split format. ## Environment | | before | after | |---|---|---| | apisix helm chart | 2.12.4 | 2.16.0 | | APISIX | 3.14.1 | 3.17.0 | | apisix-ingress-controller | 2.0.0-rc5 | 2.1.0 | | **adc** | **0.21.2** | **0.26.0** | Provider is etcd/traditional (`deployment.role: traditional`, `config_provider: etcd`), external etcd, ~218 services, Kubernetes 1.33 on EKS. ## Symptom After the chart upgrade, services whose pods were replaced started returning 502 permanently. Everything else looked healthy: - controller logs `"status":"success"` - Argo shows Synced / Healthy - `ApisixRoute` status is `Accepted=True` - the controller sends the **correct** node IPs Only services whose pods happened to cycle after the upgrade were affected, so it degrades gradually rather than failing loudly. ## Root cause adc changed how service config is stored: - **[api7/adc#354](https://github.com/api7/adc/pull/354)** (v0.22.0) `feat(apisix): separate inline upstream` - **[api7/adc#384](https://github.com/api7/adc/pull/384)** (v0.23.0) `fix(apisix): both inline and referenced upstreams coexist within the service` — *"This PR will commit the removal of the inline upstream upon service updates."* So: - adc **≤ 0.21.x** → service holds an embedded `"upstream": {nodes: [...]}` - adc **≥ 0.22.0** → service holds `"upstream_id"`, nodes live in a separate `/apisix/upstreams/<id>` The apisix helm chart defaults the adc sidecar to **0.21.2 in chart 2.12.x** and **0.26.0 in chart 2.16.0**, so a routine chart upgrade crosses this boundary. Neither the APISIX nor the apisix-ingress-controller release notes mention it. **Existing services are not migrated.** adc's update path reads a service that still has an inline upstream, assumes the separate upstream object already exists, and emits only the service write. APISIX rejects it: ``` PUT /apisix/admin/services/<id> -> 400 Bad Request error_msg: failed to fetch upstream info by upstream id [<id>], response code: 404 ``` One rejection fails the whole ADC batch, so unrelated services in the same batch also stop converging. In our cluster this was 28 failures per sync. ## Evidence chain Names below are redacted as `<ns>` / `<route>`; IDs are the real deterministic hashes for a single service. Controller is sending the right thing — from the `prepared request body` log line: ```json "upstream":{"labels":{"managed-by":"apisix-ingress-controller"}, "nodes":[{"host":"10.0.17.10","port":3000,"weight":100}]} ``` etcd still holds the old value: ```console $ etcdctl get /apisix/services/<svc-id> --print-value-only {"upstream":{"nodes":[{"weight":100,"port":3000,"host":"10.0.30.68"}],...}} ``` Real endpoint at the time was `10.0.17.10`; `10.0.30.68` belonged to no running pod. Data plane confirms: ``` balancer.lua:376: run(): proxy request to 10.0.30.68:3000 while connecting to upstream connect() failed (113: No route to host) while connecting to upstream, upstream: "http://10.0.30.68:3000/"; ``` State was split roughly in half — **218 services, only 107 upstream objects** — so over 100 services referenced an `upstream_id` that was never created. Notably, the controller payload contains `services`, `routes` and `consumers` but **no `upstreams` array** — the inline→split conversion happens entirely inside adc. ## What did not work - **Restarting the controller / APISIX.** No effect — the stale values are in etcd. - **`ingressClassName: apisix` on the CRs** (the resolution on #2699). Tested directly; the service stayed on the inline format. Those routes were already `Accepted=True`, so that fix addresses a different failure. - **Forcing a service rewrite** by appending a throwaway host to the route. This *does* migrate a service — but only where the upstream object already exists. It migrated 45 of 215 and then stalled with no error. - **Upgrading adc forward.** Checked every release 0.26.0 → 0.29.0. The differ v4 work in 0.27.1/0.28.0 is explicitly *"only a code refactoring; you should expect no difference in behavior"*. Nothing addresses upstream creation ordering. ## What did work **adc's create path is fine — only the update path is broken.** Deleting the service object makes adc treat it as new, and it creates both objects correctly. Single-service proof (note the service and its upstream share the same id): ```console $ etcdctl del /apisix/services/<svc-id> 1 $ etcdctl get /apisix/upstreams/<svc-id> --print-value-only {"labels":{"managed-by":"apisix-ingress-controller"},"retries":0, "name":"<ns>_<route>_0", "scheme":"http","nodes":[{"host":"10.0.47.0","port":3000,"weight":100}], "type":"roundrobin","timeout":{"read":10,"send":5,"connect":2},"id":"<svc-id>"} ``` The service came back with `upstream_id`, the upstream object was created with the correct nodes, and the `ApisixUpstream` timeout/retries settings were preserved. We scripted this in small batches, verifying after each that the service returned **and** its upstream object exists. Final state across 218 services: - 218/218 on `upstream_id`, 0 inline - 0 sync failures (from 28 per sync) - 0 referenced upstreams pointing at dead pods - pod restart → APISIX picks up the new IP within one `syncPeriod` Cost is a short unroutable window per service, between the delete and the controller recreating it — about one `syncPeriod`. Only the current batch is affected. ## Suggestions 1. **adc should migrate inline services on update** — when it reads a service with an inline `upstream` and intends to write `upstream_id`, it needs to create the upstream object first. Right now it emits a service referencing an object that does not exist. 2. **Ordering**: if the upstream create and service update are in the same batch, the upstream must be applied first, otherwise APISIX's reference validation rejects the service. 3. **Document the v0.22.0 format change as an upgrade note in the chart**, since the sidecar version is a chart default that most users never set explicitly. A chart bump silently crosses it. 4. Worth checking whether a partial failure should really poison an entire ADC batch — one invalid resource stopping unrelated services from converging is what turned a single bad route into a wider outage for us. Happy to provide full controller/adc logs or test anything against a patch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
