AlinsRan commented on issue #2708:
URL:
https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-5866738339
@vortegatorres Sorry for the slow follow-up. The information available so
far is not enough to narrow this down: there is no controller or APISIX log
from the time of an incident, and nothing about how the deployment is set up.
### `socket hang up`
This one has been fixed in ADC (`fix(server): free socket timeout caused
socket hang up`, released in ADC 0.24.1). The ingress controller ships a fixed
ADC starting from 2.2.0. It is recoverable, though: a sync that fails this way
is retried on the next round, so it delays an update by about one sync period
but does not leave APISIX stale for hours. It is unlikely to be the cause of
what you are seeing.
### Current suspicion: multiple controller replicas
Based on the current information, the most likely cause is a multi-replica
ingress controller deployment, the same problem as #2774:
1. Only the leader syncs. A standby replica's ADC sidecar keeps the
configuration it last pushed, along with the `*_conf_version` values it
generated.
2. When leadership moves back to that replica, it syncs from that old
baseline and sends older `*_conf_version` values than APISIX currently holds.
3. APISIX rejects the whole configuration with `<resource>_conf_version must
be greater than or equal to (...)`. Every instance that rejects it keeps
serving its old configuration, including removed pod IPs, until an APISIX
restart or until those versions catch up.
This matches most of what you described: the failures are intermittent, the
stale IP stays together with valid ones, a restart of APISIX fixes it, the
problem sometimes clears on its own after hours, and the health checker keeps
probing a pod that no longer exists. It was fixed in 2.2.0 by #2785, which
resets the ADC baseline whenever a replica becomes leader.
If you run a single replica, this does not apply, and more specific
information is needed to investigate.
### What would help (please redact anything sensitive)
- **Deployment:**
- Helm values for the ingress controller and APISIX, in particular the
number of controller replicas and the ADC sidecar image/version.
- The number of APISIX replicas, and whether they are autoscaled.
- **GatewayProxy:** the GatewayProxy resource (`kubectl get gatewayproxy -A
-o yaml`).
- **Ingress controller logs** covering the incident window, from both the
`manager` and `adc` containers, and from every replica if you run more than
one. In particular:
- any lines mentioning `conf_version`;
- leader election messages, such as lease acquired or lost;
- restarts of the `manager` container (`kubectl get pod` restart count,
`kubectl describe pod`).
- **APISIX logs:** the `error.log` from an instance that is routing to a
stale IP, covering the same window.
- **Data plane state:** while the problem is happening, the output of `GET
/apisix/admin/configs` from an affected APISIX instance and from an unaffected
one, so the upstream nodes and `*_conf_version` values can be compared.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
