vortegatorres commented on issue #2708:
URL:
https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-6196371603
@AlinsRan thanks for the follow-up. Answers and new evidence below (IPs, pod
names, namespaces and hostnames redacted).
### Deployment
- `apisix-standalone` provider, no etcd. APISIX 3.15.0 (chart 2.13.0),
ingress controller 2.0.1, ADC sidecar 0.23.1.
- **2 ingress controller replicas** (leader election on), preStop delay on
the controller.
- APISIX gateway: autoscaled by CPU (KEDA/HPA), min 4 / max 75, usually 4–6
pods.
- The affected upstream does **not** get its nodes from the controller. It
uses APISIX's built-in Kubernetes discovery:
```yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
spec:
discovery:
type: kubernetes
serviceName: <ns>/<service>:8000
```
So the controller/ADC only pushes `discovery_type: kubernetes` +
`service_name`; each APISIX pod resolves the endpoints itself through its own
Endpoints informer.
### Two recent incidents
| | Incident A | Incident B |
|---|---|---|
| Duration of stale node | ~45h | ~40h |
| Gateway pods affected | 3 of 4 | 1 of 5 |
| Leader election change during the incident | none | one, ~21h after it
started; the stale node stayed ~19h after the new leader synced |
| `*_conf_version` errors (controller, ADC, APISIX) | none | none |
| `failed to sync` on the controller | none | none |
| `PUT /apisix/admin/configs` status on every pod, incl. affected ones | 202
only | 202 only |
On the affected pods the dead pod IP stayed in the balancer and every
request first tried it (`connect() failed (111: Connection refused) while
connecting to upstream`), then retried another node. The unaffected pods in the
same deployment, receiving the same config pushes, had already dropped that IP.
In the last 14 days of logs there isn't a single `conf_version` line. There
is one 1h burst of `failed to sync` (`Put "http://unix/sync": context
canceled`), and it isn't related to either incident.
### What the APISIX pods do log
The gateway pods periodically log Endpoints informer watch failures:
```
[error] [lua] informer_factory.lua:310: watch failed, kind: Endpoints,
reason: RequestError, message : closed, context: ngx.timer
```
- Incident A: the two worst-affected pods logged about twice as many of
these as the other pods.
- Incident B: the affected pod logged one about 25h before the incident
started. Two other pods also logged one at the same time and recovered.
This is a correlation, not proof.
### Interpretation
I don't think the multi-replica / #2785 theory explains these incidents:
there were no `conf_version` rejections, every config push was accepted, and in
incident B the stale node started well before the leader change and survived
it. It also lines up with what we saw from the start: restarting APISIX fixes
it, restarting the controller doesn't. That would fit if the stale data lives
in each APISIX pod's Kubernetes discovery state, which never comes from the
controller.
My current suspicion is that, after some watch failures, an APISIX pod's
Kubernetes discovery stops applying Endpoints updates for that service. If you
agree, this probably belongs in `apache/apisix` (kubernetes discovery) rather
than here. Happy to open it there and link it.
### Things we've tried
- **Active + passive upstream health checks.** They reduced the impact, but
active probes left orphan targets in the healthcheck shared dict
(apache/apisix#13282) and caused recurring 5xx bursts that needed restarts, so
we removed them. I see #13627 and #13629 fixed that in 3.18.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
