janiussyafiq commented on issue #2756:
URL:
https://github.com/apache/apisix-ingress-controller/issues/2756#issuecomment-5290372276
I looked into the `socket hang up` part and managed to reproduce it locally.
Sharing the root cause since it explains all the reports in this thread and
#2704.
**Root cause:** it is a keep-alive idle-timeout race between the ADC sidecar
and the APISIX Admin API. ADC (<= 0.24.0) pools admin connections with a
hardcoded 60s `freeSocketTimeout`, and the Admin API's nginx also uses
`keepalive_timeout 60s` (APISIX and Helm chart default). With `syncPeriod: 1m`,
each sync reuses a pooled connection right at the 60s idle boundary, exactly
when nginx closes it, so the request lands on a closed socket and fails with
`socket hang up`. Whether it appears as `HTTP 500 {"message":"Error: socket
hang up"}` (dump stage, #2704) or a `partial_failure` with `reason: "socket
hang up"` (apply stage, the log above) is just where in the sync it strikes.
**Why the workaround works:** raising `nginx.keepaliveTimeout` to 75s (or
120s) makes the server-side timeout longer than ADC's 60s, so ADC closes idle
connections first and the race disappears.
**This is already fixed:** api7/adc#407 + api7/adc#409 lowered ADC's default
to 50s, effective from **adc 0.24.2**. However:
- controller install manifests 2.0.0 / 2.0.1 / 2.1.0 pin adc **0.23.1**
(affected)
- Helm chart `apisix-ingress-controller-1.1.1` (pulled by the
`apisix-2.14.0` umbrella chart) also pins **0.23.1** (affected)
- controller 2.2.0 and chart 1.1.2+ ship adc >= 0.24.2 (fixed)
**Remediation:** upgrade to controller 2.2.0 / a chart with adc >= 0.24.2,
or simply override the adc sidecar image tag (`adcContainer.image.tag` in Helm,
the `sidecar` image in kustomize) to >= 0.24.2. No nginx tuning needed after
that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
