Kukunin opened a new pull request, #14700:
URL: https://github.com/apache/cloudstack/pull/14700

   ### Description
   
   `HAProxyConfigurator` emits only two session timeouts, `timeout client` and 
`timeout server`, and `network.loadbalancer.haproxy.idle.timeout=0` blanks both 
of them. That zero is a legitimate setting — it is what you use when 
load-balanced connections must survive an idle period, a database reached 
through a TCP rule being the obvious case — so the timeouts cannot simply be 
put back.
   
   With no session timeout at all, a client that half-closes leaves its session 
held indefinitely. haproxy keeps the fd, so the socket stays in 
`FIN_WAIT2`/`CLOSE_WAIT`, and the kernel cannot reap it: `tcp_fin_timeout` only 
applies to *orphaned* sockets, and these are not orphaned.
   
   Enough of them reach `maxconn`, which is a **per-process** budget shared by 
every VIP the router serves. Past that, the kernel still completes TCP 
handshakes from the backlog for sessions haproxy never services — connections 
establish, no data is ever returned — and every service behind that router 
stops answering while its backends stay healthy. Seamless reloads compound it, 
since an old worker is only retired once its sessions close, so each reload 
leaves another worker resident holding its own stuck sessions.
   
   This was observed in production: a VR reached 4096/4096 with ~4,100 
half-closed sessions from one client, taking down two unrelated applications 
simultaneously for ~25 minutes. Backend health checks stayed `UP` throughout. 
The same router had accumulated six haproxy workers, the oldest a week old, one 
holding 254 fds.
   
   `timeout client-fin` and `timeout server-fin` arm **only once a FIN has been 
seen**, so they bound the half-closed case and leave fully open idle 
connections untouched. They therefore hold for `idleTimeout=0` and for any 
configured value alike, which is why they are added unconditionally rather than 
being made configurable.
   
   The entries are appended rather than inserted because the defaults list is 
fixed-size and indices 7, 9 and 10 are addressed positionally further down the 
method.
   
   The existing `idleTimeout=0` assertions tested for the absence of 
`"\ttimeout server"` / `"\ttimeout client"` — prefixes rather than whole 
directives — so the new entries tripped a test whose subject is still true. 
They now match one character further, up to the space separating directive from 
value: the session timeouts render space-padded, while the fin timeouts have a 
hyphen immediately after the name. A positive assertion was added that the fin 
timeouts survive `idleTimeout=0`, since that is the point of them.
   
   ### Types of changes
   
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to change)
   - [ ] New feature (non-breaking change which adds functionality)
   - [x] Bug fix (non-breaking change which fixes an issue)
   - [ ] Enhancement (improves an existing feature and functionality)
   - [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
   - [ ] Build/CI
   - [ ] Test (unit or integration test code)
   
   ### Feature/Enhancement Scale or Bug Severity
   
   #### Bug Severity
   
   - [ ] BLOCKER
   - [x] Critical
   - [ ] Major
   - [ ] Minor
   - [ ] Trivial
   
   ### How Has This Been Tested?
   
   `HAProxyConfiguratorTest` passes against `main` (5 tests, 0 failures).
   
   Deployed to two production zones. The regenerated `haproxy.cfg` carries both 
fin timeouts with `timeout client`/`timeout server` still absent under 
`idleTimeout=0`, confirming the zero is still honoured. On a busy router, 
half-closed sockets now turn over instead of accumulating — sampling the same 
socket identities 75s apart showed **0 of 332** still present, against the 
unbounded growth to ~4,100 seen before. A quieter router went from a steady 13 
`FIN_WAIT2` / 13 `CLOSE_WAIT` to 0/0.
   
   Note that an existing router needs its haproxy restarted, not just reloaded, 
to pick this up: workers started before the change keep running their original 
config and never drain.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to