Hi William, > The timeout client applies on inactivity, are you sure those connections are > inactives? Try to do a "show sess" on the old process so you can see the > remaining sessions.
I suspected the same. This confirms what I observed. > Are you using the seamless reload feature or just the master worker? > Looks like there is some confusion about what is the seamless reload in > HAProxy, It's not the kill -USR2 in master-worker mode. It's the hability to > transfer the bind socket to a new process over the unix socket using the -x > parameter. This can be configurated with "expose-fd listeners" on the stats > socket configuration. But this won't change the behavior you are observing. I'm using both, so I was conflating the two in my email. Thanks for clarifying this! > Once the -USR2 signal is received by the master it will reexec itself with > -sf, > which will send -USR1 signals to the workers. So yes, it applies on the reload > in the master worker mode. Ah—I missed that string of actions in the docs. Makes sense. > The reload of HAProxy in master worker mode or in daemon mode implies that you > will still have old instances of HAProxy running until all there jobs are > done. > If you don't want this behavior, you should do a restart of the service and > not a reload. Again, makes sense. Thanks for the clarification. > This option isn't useful in master worker mode, you just have to specify > "expose-fd listeners" in the configuration, and the master will had this > option > for you. Cool! So I think the answer to this all is to add hard-stop-after, and it will fix the issue. H2 likely has longer-lived sessions on the client-side, so I only noticed the hole in my configuration after enabling h2. Thanks for the help! Best, Luke — Luke Seelenbinder Stadia Maps | Founder stadiamaps.com ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, January 29, 2019 12:23 PM, William Lallemand <[email protected]> wrote: > Hi Luke, > > On Tue, Jan 29, 2019 at 10:31:00AM +0000, Luke Seelenbinder wrote: > > > Hi all, > > I'm observing some odd behavior with seamless reloads and processes hanging > > around. It appears when a reload is triggered with any active client > > connections, the new process comes up with the proper -sf <pid list>, but > > the > > old process(es) is only terminated after the last client disconnects > > (definitely longer than timeout client) and lives for a very long time in > > our > > experience or until a restart of the master process is triggered. I'm using > > h2 on the front-end with HTX. > > The normal behavior of HAProxy during a soft stop (-sf) is to wait for all > jobs to stop before leaving. This behavior is the same with a reload in > master worker mode. > > The timeout client applies on inactivity, are you sure those connections are > inactives? Try to do a "show sess" on the old process so you can see the > remaining sessions. > > Are you using the seamless reload feature or just the master worker? > Looks like there is some confusion about what is the seamless reload in > HAProxy, It's not the kill -USR2 in master-worker mode. It's the hability to > transfer the bind socket to a new process over the unix socket using the -x > parameter. This can be configurated with "expose-fd listeners" on the stats > socket configuration. But this won't change the behavior you are observing. > > > Would hard-stop-after apply in this case since the reload signal is USR2, > > but > > hard-stop-after is documented to apply to USR1? > > Once the -USR2 signal is received by the master it will reexec itself with > -sf, > which will send -USR1 signals to the workers. So yes, it applies on the reload > in the master worker mode. > > > This is somewhat disconcerting, since it results in an effective memory leak > > on every reload, which can happen pretty frequently in our setup. If > > hard-stop-after applies here, then it's not a bug, but perhaps the > > documentation should clarify its meaning. > > The reload of HAProxy in master worker mode or in daemon mode implies that you > will still have old instances of HAProxy running until all there jobs are > done. > If you don't want this behavior, you should do a restart of the service and > not a reload. > > > I first observed this with 1.9.2, it continues in 1.9.3, but the behavior > > may > > exist before that. > > > I've included my systemd configuration (from systemctl cat haproxy), in case > > that's the culprit. > > > Add extra flags here, see haproxy(1) for a few options > > > > ======================================================= > > > > EXTRAOPTS="-x /run/haproxy/admin.sock" > > This option isn't useful in master worker mode, you just have to specify > "expose-fd listeners" in the configuration, and the master will had this > option > for you. > > The systemd configuration looks good to me. > > Cheers, > > -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > William Lallemand
publickey - [email protected] - 0xB23C1E8A.asc
Description: application/pgp-keys
signature.asc
Description: OpenPGP digital signature
