A stopping process keeps the idle connections of its servers until it exits, and puts connections back into the idle lists as their streams end, although only the sessions still attached to it can use them. For most servers this only delays their release, but it breaks reverse HTTP. A connection attached with attach-srv is removed from the stopping list when it is reversed, and the soft-stop GOAWAY is only sent on frontend connections, so the peer is never told to leave. The connection is not a job either, so the old process keeps it open until its last client stream ends. The peer has no reason to redial, while the new process has no connection to it: requests to that server get a 503 for as long as the old process lingers, which with long-running requests and no hard-stop-after can be minutes.
Once the backends are stopped in do_soft_stop_now(), release the idle connections of every server, as is done for a server put in maintenance, and refuse connections back into the idle list of a stopped backend, as is already done for a server in maintenance, so a connection still carrying streams is closed as soon as its last stream ends. Each closure prompts a reverse HTTP peer to redial into the new process. Gating on the stopped backend rather than on "stopping" keeps idle connections usable during a "grace" period, while the process still serves. Sessions still attached to the stopping process may open new streams after the listeners are closed. Those streams no longer find an idle connection: a normal server opens a new one, but a reverse HTTP server cannot, so such a stream fails unless a busy connection to that server still has a free stream. The new process is the one receiving new traffic, so this seems the better trade. This only helps connections that are idle at the reload. One that carries a stream is still kept until its streams end, and a reverse HTTP peer is not told to replace it meanwhile, so a pool whose connections are all busy is not improved. Measured with a reverse HTTP peer behind pool-purge-delay 3600s and an 8s or 20s request in flight across a reload: requests to the server are served again after the peer's reconnect delay (1.2-1.3s for a peer with a 1.0-1.5s backoff), against the whole remaining life of the in-flight request without this patch. The in-flight request still completes. With 16 concurrent requests over 4 connections, nothing is idle and the result is unchanged. This partially addresses GitHub issue #3556. Co-Authored-By: Claude Opus 5.5 <[email protected]> --- v2: release the idle connections of every server on soft-stop, not only reverse HTTP ones, and refuse idle-list entry for any stopped backend, as suggested by Amaury in the v1 thread ("[PATCH] BUG/MEDIUM: rhttp: close idle reverse connections on soft-stop"). Commit message updated accordingly, including the case of streams opened on the old process after the sweep. include/haproxy/server.h | 1 + src/proxy.c | 14 ++++++++++++++ src/server.c | 4 ++-- 3 files changed, 17 insertions(+), 2 deletions(-) diff --git a/include/haproxy/server.h b/include/haproxy/server.h index 23d0c0d72..b299cc1bf 100644 --- a/include/haproxy/server.h +++ b/include/haproxy/server.h @@ -112,6 +112,7 @@ int snr_resolution_error_cb(struct resolv_requester *requester, int error_code); struct server *snr_check_ip_callback(struct server *srv, void *ip, unsigned char *ip_family); struct task *srv_cleanup_idle_conns(struct task *task, void *ctx, unsigned int state); void srv_release_conn(struct server *srv, struct connection *conn); +void srv_cleanup_connections(struct server *srv); struct connection *srv_lookup_conn(struct ceb_root **tree, uint64_t hash); struct connection *srv_lookup_conn_next(struct ceb_root **tree, struct connection *conn); diff --git a/src/proxy.c b/src/proxy.c index 2601d8b29..725921b59 100644 --- a/src/proxy.c +++ b/src/proxy.c @@ -58,6 +58,7 @@ #include <haproxy/quic_tp.h> #include <haproxy/quic_tune.h> #include <haproxy/server-t.h> +#include <haproxy/server.h> #include <haproxy/signal.h> #include <haproxy/stats.h> #include <haproxy/stconn.h> @@ -4154,6 +4155,19 @@ static void do_soft_stop_now() HA_RWLOCK_WRUNLOCK(PROXY_LOCK, &p->lock); } + /* A stopped backend will not reuse its idle connections, so release + * them now rather than at exit, as for a server put in maintenance. + * Busy ones are closed on release (see srv_add_to_idle_list()). Reverse + * HTTP peers depend on it: they only redial once their connection to + * this process is closed. + */ + list_for_each_entry(p, &main_proxies, el) { + struct server *srv; + + list_for_each_entry(srv, &p->servers, el_px) + srv_cleanup_connections(srv); + } + /* signal zero is used to broadcast the "stopping" event */ signal_handler(0); } diff --git a/src/server.c b/src/server.c index 50c7771ce..ec144f937 100644 --- a/src/server.c +++ b/src/server.c @@ -61,7 +61,6 @@ static void srv_update_status(struct server *s, int type, int cause); static int srv_apply_lastaddr(struct server *srv, int *err_code); -static void srv_cleanup_connections(struct server *srv); /* extra keywords used as value for other arguments. They are used as * suggestions for mistyped words. @@ -8082,7 +8081,7 @@ static int srv_migrate_conns_to_remove(struct server *srv, int thr, int toremove /* cleanup connections for a given server * might be useful when going on forced maintenance or live changing ip/port */ -static void srv_cleanup_connections(struct server *srv) +void srv_cleanup_connections(struct server *srv) { struct sess_priv_conns *sess_conns; int did_remove; @@ -8222,6 +8221,7 @@ int srv_add_to_idle_list(struct server *srv, struct connection *conn, int is_saf if (!(conn->flags & CO_FL_PRIVATE) && srv && srv->pool_purge_delay > 0 && !(srv->cur_admin & SRV_ADMF_MAINT) && + !(srv->proxy->flags & PR_FL_STOPPED) && ((srv->proxy->options & PR_O_REUSE_MASK) != PR_O_REUSE_NEVR) && ha_used_fds < global.tune.pool_high_count && (srv->max_idle_conns == -1 || srv->max_idle_conns > srv->curr_idle_conns) && -- 2.54.0 (Apple Git-157)
