Kukunin opened a new pull request, #14700: URL: https://github.com/apache/cloudstack/pull/14700
### Description `HAProxyConfigurator` emits only two session timeouts, `timeout client` and `timeout server`, and `network.loadbalancer.haproxy.idle.timeout=0` blanks both of them. That zero is a legitimate setting — it is what you use when load-balanced connections must survive an idle period, a database reached through a TCP rule being the obvious case — so the timeouts cannot simply be put back. With no session timeout at all, a client that half-closes leaves its session held indefinitely. haproxy keeps the fd, so the socket stays in `FIN_WAIT2`/`CLOSE_WAIT`, and the kernel cannot reap it: `tcp_fin_timeout` only applies to *orphaned* sockets, and these are not orphaned. Enough of them reach `maxconn`, which is a **per-process** budget shared by every VIP the router serves. Past that, the kernel still completes TCP handshakes from the backlog for sessions haproxy never services — connections establish, no data is ever returned — and every service behind that router stops answering while its backends stay healthy. Seamless reloads compound it, since an old worker is only retired once its sessions close, so each reload leaves another worker resident holding its own stuck sessions. This was observed in production: a VR reached 4096/4096 with ~4,100 half-closed sessions from one client, taking down two unrelated applications simultaneously for ~25 minutes. Backend health checks stayed `UP` throughout. The same router had accumulated six haproxy workers, the oldest a week old, one holding 254 fds. `timeout client-fin` and `timeout server-fin` arm **only once a FIN has been seen**, so they bound the half-closed case and leave fully open idle connections untouched. They therefore hold for `idleTimeout=0` and for any configured value alike, which is why they are added unconditionally rather than being made configurable. The entries are appended rather than inserted because the defaults list is fixed-size and indices 7, 9 and 10 are addressed positionally further down the method. The existing `idleTimeout=0` assertions tested for the absence of `"\ttimeout server"` / `"\ttimeout client"` — prefixes rather than whole directives — so the new entries tripped a test whose subject is still true. They now match one character further, up to the space separating directive from value: the session timeouts render space-padded, while the fin timeouts have a hyphen immediately after the name. A positive assertion was added that the fin timeouts survive `idleTimeout=0`, since that is the point of them. ### Types of changes - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] New feature (non-breaking change which adds functionality) - [x] Bug fix (non-breaking change which fixes an issue) - [ ] Enhancement (improves an existing feature and functionality) - [ ] Cleanup (Code refactoring and cleanup, that may add test cases) - [ ] Build/CI - [ ] Test (unit or integration test code) ### Feature/Enhancement Scale or Bug Severity #### Bug Severity - [ ] BLOCKER - [x] Critical - [ ] Major - [ ] Minor - [ ] Trivial ### How Has This Been Tested? `HAProxyConfiguratorTest` passes against `main` (5 tests, 0 failures). Deployed to two production zones. The regenerated `haproxy.cfg` carries both fin timeouts with `timeout client`/`timeout server` still absent under `idleTimeout=0`, confirming the zero is still honoured. On a busy router, half-closed sockets now turn over instead of accumulating — sampling the same socket identities 75s apart showed **0 of 332** still present, against the unbounded growth to ~4,100 seen before. A quieter router went from a steady 13 `FIN_WAIT2` / 13 `CLOSE_WAIT` to 0/0. Note that an existing router needs its haproxy restarted, not just reloaded, to pick this up: workers started before the change keep running their original config and never drain. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
