Hi Luke, On Tue, Jan 29, 2019 at 10:31:00AM +0000, Luke Seelenbinder wrote: > Hi all, > > I'm observing some odd behavior with seamless reloads and processes hanging > around. It appears when a reload is triggered with any active client > connections, the new process comes up with the proper -sf <pid list>, but the > old process(es) is only terminated after the last client disconnects > (definitely longer than timeout client) and lives for a very long time in our > experience or until a restart of the master process is triggered. I'm using > h2 on the front-end with HTX. >
The normal behavior of HAProxy during a soft stop (-sf) is to wait for all jobs to stop before leaving. This behavior is the same with a reload in master worker mode. The timeout client applies on inactivity, are you sure those connections are inactives? Try to do a "show sess" on the old process so you can see the remaining sessions. Are you using the seamless reload feature or just the master worker? Looks like there is some confusion about what is the seamless reload in HAProxy, It's not the kill -USR2 in master-worker mode. It's the hability to transfer the bind socket to a new process over the unix socket using the -x parameter. This can be configurated with "expose-fd listeners" on the stats socket configuration. But this won't change the behavior you are observing. > Would hard-stop-after apply in this case since the reload signal is USR2, but > hard-stop-after is documented to apply to USR1? Once the -USR2 signal is received by the master it will reexec itself with -sf, which will send -USR1 signals to the workers. So yes, it applies on the reload in the master worker mode. > This is somewhat disconcerting, since it results in an effective memory leak > on every reload, which can happen pretty frequently in our setup. If > hard-stop-after applies here, then it's not a bug, but perhaps the > documentation should clarify its meaning. > The reload of HAProxy in master worker mode or in daemon mode implies that you will still have old instances of HAProxy running until all there jobs are done. If you don't want this behavior, you should do a restart of the service and not a reload. > I first observed this with 1.9.2, it continues in 1.9.3, but the behavior may > exist before that. > > I've included my systemd configuration (from systemctl cat haproxy), in case > that's the culprit. > > # Add extra flags here, see haproxy(1) for a few options > EXTRAOPTS="-x /run/haproxy/admin.sock" This option isn't useful in master worker mode, you just have to specify "expose-fd listeners" in the configuration, and the master will had this option for you. The systemd configuration looks good to me. Cheers, -- William Lallemand
